This was recently installed with a P2V'd Windows 2000 server and a fresh install of SBS 2011. The 2000 server was already so slow that I didn't really notice the issue until we loaded the SBS2011 guest, which is extremely slow.
The hardware is a dual processor Intel board with an Intel RMS25KB080 RAID controller (which I believe is an LSI 2308 HBA) with 4 drives in a RAID 10. Firmware is current.
I've done some poking around and I believe the issue is with storage latency. I've been watching esxtop for a while and DAVG/cmd ranges from around 35-90, but occasionally spikes over 110. I've attached the full vmkernel.log but here are a few lines I'm seeing:
2013-07-21T11:07:27.963Z cpu3:8297)WARNING: ScsiScan: 1276: Failed to add path vmhba2:C0:T4:L0 : Failure
2013-07-21T11:12:28.024Z cpu2:1837209)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412402282880, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2013-07-21T11:12:28.024Z cpu2:1837209)ScsiDeviceIO: 2331: Cmd(0x412402282880) 0x1a, CmdSN 0xb875 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2013-07-22T13:43:56.325Z cpu2:9997)VmkAccess: 637: sfcb-LSIESG_SMI: access denied:: dom:pluginDom(3), obj:forkExecSys(88), mode:syscall_allow(2)
2013-07-22T13:43:56.325Z cpu2:9997)VmkAccess: 922: VMkernel syscall invalid (1025)
I believe I have the correct VIBs installed because I am able to connect to the controller to monitor the RAID via the Megaraid Storage Manager installed on a guest OS. I have not been able to verify with 100% certainty which LSI drivers I should use with the RMS25KB080, but here is what I have installed:
~ # esxcli software vib list | grep sas
scsi-mpt2sas 15.00.01.00.1vmw-1OEM.500.0.0.472560 LSI VMwareCertified 2013-06-20
scsi-megaraid-sas 5.34-4vmw.510.1.12.1065491 VMware VMwareCertified 2013-06-18
scsi-mptsas 4.23.01.00-6vmw.510.0.0.799733 VMware VMwareCertified 2013-06-18
~ # esxcli software vib list | grep lsi
lsiprovider 500.04.V0.34-0012 LSI VMwareAccepted 2013-06-20
Can anyone make any suggestions to help me troubleshoot the issue?