(Discovered) Best Practice Config for OpenFiler iSCSI Storage for ESXi4

Update: OpenFiler performance and reliability remained low. I've since moved to another platform, and no, this isn't an ad so I'm not saying where I ended up.

That title sucks, but I'm hoping Google will like it.

The OpenFiler iSCSI box at the Eastern site bailed, due largely to congestion on the still-not-redundant switch we have in place (new stuff still in shipping, argh) and after the fallout I found some config tips for the OF 2.3 server.  This thing's a physical box with nothing but iSCSI on it, so it may be different if you're simulating some ESXi-ESXi virtual worlds in your mini-lab at work.

1. Disable Delayed ACK in the iSCSI storage adapter advanced properties on the ESXi host.  (scroll down)(via)
2. Reduce the VMs-per-LUN count to reduce the reservation count and conflict  Smaller LUNs, kids, despite what you hear.(via)
3. Change the default iSCSI timeout on your ESX server to 14 seconds: esxcfg-advcfg -s 14000 /VMFS3/HBTokenTimeout (via, via, via)
4. Keep your snapshots down:  they increase load, with the obvious impact.(via)
5. Consider upgrading your intel e1000 or exgb NICs, code here, intel instructions here, openFiler-derived procedure here.
6. Don't use more than one NIC (bonding) or more than one interface (different subnets) at a time with Openfiler. (via, via, instructions for OF)
7. Target tuning (via):
   1. DefaultTime2Wait should be changed to "10"
   2. ImmediateData should be changed to "Yes"
   3. MaxRecvDataSegmentLength should be changed to "262144"
   4. MaxXmitDataSegmentLength should be changed to "262144"
8. LUN tuning
   1. R/W mode should be changed to "write-back"
      (it should go without saying that a BBU is essential for your RAID controller)
   2. Transfer mode should be changed to "fileio"
      (this is controversial)
9. NEVER, EVER share LUNs between ESX hosts using Openfiler and IET.
   (Not sure how doable this one is in an ESXi cluster, but that's the suggestion.) (via)

So in my case, we're looking to implement a few of those.  Hopefully we'll not see this kind of issue any time soon, but watch this space in case we do see something after the upgrade/tuning.

The good news is that OF is considering another iSCSI target, and the changes are supposed to be very beneficial for performance and for us.

Disabling LACP with iSCSI in ESXi4 to Enable MPIO

My mini Atlantis cluster is working well, but I found a document suggesting it can be better -- or another way that it can be better, at least.

It's using iSCSI for the storage, right now, and I have it configured with LACP for redundancy off one bonded channel.  I'd hoped to also get some load-balancing from it, but I was young and naive (last week) when I set it up.  I've since learned of a better way to set it up.  It's all in A Multivendor Post on using iSCSI with VMware vSphere at Virtual Geek .

Here's the money-shot, as I told DD in email:

1. don't have your iSCSI set up with LACP
2. set up one vmkX for each NIC you're going to dedicate for iSCSI
3. have them not team
4. subscribe to each effective target on each interface on the remote side separately
5. drop your IOps per path factor to a lower number to reduce stickiness on a particular path and improve the MP part of the IO on lower-speed iSCSI configs.

Not sure about the last part, but it's what passes for the sense it's making right now.  I'm off to read it more fully and plan a migration from one method to another in Atlantis.

I'm actually excited that I can improve performance on this little cluster without requiring more capital.  Let you know how it goes.