(Discovered) Best Practice Config for OpenFiler iSCSI Storage for ESXi4

Update: OpenFiler performance and reliability remained low. I've since moved to another platform, and no, this isn't an ad so I'm not saying where I ended up.

That title sucks, but I'm hoping Google will like it.

The OpenFiler iSCSI box at the Eastern site bailed, due largely to congestion on the still-not-redundant switch we have in place (new stuff still in shipping, argh) and after the fallout I found some config tips for the OF 2.3 server.  This thing's a physical box with nothing but iSCSI on it, so it may be different if you're simulating some ESXi-ESXi virtual worlds in your mini-lab at work.

1. Disable Delayed ACK in the iSCSI storage adapter advanced properties on the ESXi host.  (scroll down)(via)
2. Reduce the VMs-per-LUN count to reduce the reservation count and conflict  Smaller LUNs, kids, despite what you hear.(via)
3. Change the default iSCSI timeout on your ESX server to 14 seconds: esxcfg-advcfg -s 14000 /VMFS3/HBTokenTimeout (via, via, via)
4. Keep your snapshots down:  they increase load, with the obvious impact.(via)
5. Consider upgrading your intel e1000 or exgb NICs, code here, intel instructions here, openFiler-derived procedure here.
6. Don't use more than one NIC (bonding) or more than one interface (different subnets) at a time with Openfiler. (via, via, instructions for OF)
7. Target tuning (via):
   1. DefaultTime2Wait should be changed to "10"
   2. ImmediateData should be changed to "Yes"
   3. MaxRecvDataSegmentLength should be changed to "262144"
   4. MaxXmitDataSegmentLength should be changed to "262144"
8. LUN tuning
   1. R/W mode should be changed to "write-back"
      (it should go without saying that a BBU is essential for your RAID controller)
   2. Transfer mode should be changed to "fileio"
      (this is controversial)
9. NEVER, EVER share LUNs between ESX hosts using Openfiler and IET.
   (Not sure how doable this one is in an ESXi cluster, but that's the suggestion.) (via)

So in my case, we're looking to implement a few of those.  Hopefully we'll not see this kind of issue any time soon, but watch this space in case we do see something after the upgrade/tuning.

The good news is that OF is considering another iSCSI target, and the changes are supposed to be very beneficial for performance and for us.

Kickstarting ESX VMs and Physical Hosts -- Knowing Which is Which

UPDATE: This method also does not work. Sorry.

Cobbler and kickstarting is my new cool toy.  I tinker with it FAR too much. I had a problem, though, that I need to install the vmware tools only on the VMs, and install smartmontools only on the physical hosts.  What's a guy to do?

After some digging, coding, hacking, testing, cursing, I finally discovered a decent switch I can use to identify a box by its mac.

Then I lost that code.

So I found another method.  This one's ugly as sin, but it may actually work.  Check this nasty-ass kung-fu:

#if ":".join($interfaces.eth0.mac_address.split(':')[0:3]) in "00:50:56 00:0C:29"

[code]

#end if

Yeah.  That's one ugly baby.  If Blogspot again truncates that line all to hell, remember the #if statement is all on the first line of 3.

Now to see how well it works.

Installing VMware Tools in Linux after ESX 4.1

Tried to install VMware-Tools lately?  Not sure how it goes on Windows, but on Linux it's a bit more challenging than it used to be.  Used to be there was an RPM on the ISO that popped into the Linux machine's vCDROM drive when the option to install VMware Tools was chosen.

No more.  Like ESXi4.0 before it, ESX4.1 seems to ship with an ISO containing, for Linux, a little tarball with the stuff.

Those who've worked for a distro or packaged anything will know that shipping and installing bare tarballs is really not the best way to go.  Sure, it's fine for the cowboys who maintain 1-2 machines at the home or home-office, but for anyone who has to maintain more than one, has to roll back or verify a package level, or who just wants a clean system, a good package is the only way to go.

Note that I said good package.  If you hear anyone mention "dependency hell" or anything like that, they weren't paying attention.  Stop using bad packages!  Stop mixing repositories.  The pathetic blunt object that pass for packages is the topic of many blogwails past and many blogwails to come.

The great news?  VMware seem to have a few packages for us.  In fact, they come in 4 flavours of RedHat and some even for the you-bunt-you crowd.  Seems all you have to do is mark up the apt config ...

cat <<EOF> /etc/apt/sources.list.d/esx.list

repomd  http://packages.vmware.com tools/esx/4.1/rhel$(VERSION)/$(ARCH)
EOF
apt-get update

... or yum config if you don't build your own packages and know where apt excels ...

cat <<EOF> /etc/yum.repos.d/esx.repo

[esx]
name=VMware-tools for RHEL\$releasever - \$basearch

baseurl=http://packages.vmware.com/tools/esx/4.1/rhel\$releasever/\$basearch
enabled=1
gpgcheck=1
gpgkey=http://packages.vmware.com/tools/VMWARE-PACKAGING-GPG-KEY.pub

EOF

yum update

and after refreshing, just install the package:

apt-get install vmware-tools
Reading Package Lists... Done
Building Dependency Tree... Done
The following extra packages will be installed:
   vmware-open-vm-tools (8.3.2-257589.el5)
   vmware-open-vm-tools-common (8.3.2-257589.el5)
   vmware-open-vm-tools-kmod (8.3.2-257589.el5)
   vmware-open-vm-tools-nox (8.3.2-257589.el5)
   vmware-open-vm-tools-xorg-drv-display (10.16.7.0-0.257589.el5)
   vmware-open-vm-tools-xorg-drv-mouse (12.6.4.0-0.257589.el5)
   vmware-open-vm-tools-xorg-utilities (8.3.2-257589.el5)
   vmware-tools-common (8.3.2-257589.el5)
   vmware-tools-nox (8.3.2-257589.el5)
The following packages will be REPLACED:
   xorg-x11-drv-vmmouse (12.4.0-2.1)
   (by (10.13.0-2.1)
   vmware-open-vm-tools-xorg-drv-mouse) ()
    (12.4.0-2.1)
   xorg-x11-drv-vmware (10.13.0-2.1)
   (by ()
   vmware-open-vm-tools-xorg-drv-display) (12.4.0-2.1)
The following NEW packages will be installed:
   vmware-open-vm-tools (8.3.2-257589.el5)
   vmware-open-vm-tools-common (8.3.2-257589.el5)
   vmware-open-vm-tools-kmod (8.3.2-257589.el5)
   vmware-open-vm-tools-nox (8.3.2-257589.el5)
   vmware-open-vm-tools-xorg-drv-display (10.16.7.0-0.257589.el5)
   vmware-open-vm-tools-xorg-drv-mouse (12.6.4.0-0.257589.el5)
   vmware-open-vm-tools-xorg-utilities (8.3.2-257589.el5)
   vmware-tools (8.3.2-257589.el5)
   vmware-tools-common (8.3.2-257589.el5)
   vmware-tools-nox (8.3.2-257589.el5)
0 upgraded, 10 newly installed, 2 replaced, 0 removed and 0 not upgraded.
Need to get 44.8kB/13.7MB of archives.
After unpacking 41.1MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-kmod 8.3.2-257589.el5 [524kB]
Get:2 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-common 8.3.2-257589.el5 [5283kB]
Get:3 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-nox 8.3.2-257589.el5 [2631B]
Get:4 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-xorg-drv-mouse 12.6.4.0-0.257589.el5 [17.5kB]
Get:5 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-xorg-drv-display 10.16.7.0-0.257589.el5 [33.4kB]
Get:6 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-xorg-utilities 8.3.2-257589.el5 [7828kB]
Get:7 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools 8.3.2-257589.el5 [2811B]
Get:8 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-tools-common 8.3.2-257589.el5 [39.4kB]
Get:9 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-tools-nox 8.3.2-257589.el5 [2635B]
Get:10 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-tools 8.3.2-257589.el5 [2763B]
Fetched 44.8kB in 0s (46.5kB/s)
Committing changes...
Preparing                                ############################## [100%]
Updating / installing
  vmware-open-vm-tools-xorg-drv-display- ############################## [100%]
  vmware-open-vm-tools-xorg-drv-mouse-12 ############################## [100%]
  vmware-open-vm-tools-kmod-8.3.2-257589 ############################## [100%]
  vmware-open-vm-tools-common-8.3.2-2575 ############################## [100%]
  vmware-open-vm-tools-nox-8.3.2-257589. ############################## [100%]
  vmware-open-vm-tools-xorg-utilities-8. ############################## [100%]
  vmware-open-vm-tools-8.3.2-257589.el5. ############################## [100%]
  vmware-tools-common-8.3.2-257589.el5.x ############################## [100%]
  vmware-tools-nox-8.3.2-257589.el5.x86_ ############################## [100%]
  vmware-tools-8.3.2-257589.el5.x86_64   ############################## [100%]
Cleaning up / removing
  xorg-x11-drv-vmmouse-12.4.0-2.1.x86_64 ############################## [100%]
  xorg-x11-drv-vmware-10.13.0-2.1.x86_64 ############################## [100%]
Done.

While I avoid Yum, it should look like this:

[root@0-50-56-bc-0-7 ~]# yum install vmware-tools
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * addons: mirror.raystedman.net
 * base: mirror.skiplink.com
 * extras: mirror.skiplink.com
 * updates: styx.biochem.wfubmc.edu
Setting up Install Process
Resolving Dependencies
:
[blah blah]
:
Dependencies Resolved

==================================================================================================================================
 Package                                           Arch               Version                               Repository       Size
==================================================================================================================================
Installing:
 vmware-open-vm-tools-xorg-drv-display             x86_64             10.16.7.0-0.257589.el5                esx              33 k
     replacing  xorg-x11-drv-vmware.x86_64 10.13.0-2.1

 vmware-open-vm-tools-xorg-drv-mouse               x86_64             12.6.4.0-0.257589.el5                 esx              17 k
     replacing  xorg-x11-drv-vmmouse.x86_64 12.4.0-2.1

 vmware-tools                                      x86_64             8.3.2-257589.el5                      esx             2.7 k
Installing for dependencies:
 vmware-open-vm-tools                              x86_64             8.3.2-257589.el5                      esx             2.7 k
 vmware-open-vm-tools-common                       x86_64             8.3.2-257589.el5                      esx             5.0 M
 vmware-open-vm-tools-kmod                         x86_64             8.3.2-257589.el5                      esx             512 k
 vmware-open-vm-tools-nox                          x86_64             8.3.2-257589.el5                      esx             2.6 k
 vmware-open-vm-tools-xorg-utilities               x86_64             8.3.2-257589.el5                      esx             7.5 M
 vmware-tools-common                               x86_64             8.3.2-257589.el5                      esx              39 k
 vmware-tools-nox                                  x86_64             8.3.2-257589.el5                      esx             2.6 k

Transaction Summary
==================================================================================================================================
Install      10 Package(s)
Upgrade       0 Package(s)

Total download size: 13 M
Is this ok [y/N]: y
Downloading Packages:
:
[ more blah blah ]
:
Running Transaction
  Installing     : vmware-open-vm-tools-xorg-drv-display                                                                     1/12
  Installing     : vmware-open-vm-tools-kmod                                                                                 2/12
  Installing     : vmware-open-vm-tools-common                                                                               3/12
  Installing     : vmware-open-vm-tools-nox                                                                                  4/12
  Installing     : vmware-tools-common                                                                                       5/12
  Installing     : vmware-tools-nox                                                                                          6/12
  Installing     : vmware-open-vm-tools-xorg-drv-mouse                                                                       7/12
  Installing     : vmware-open-vm-tools-xorg-utilities                                                                       8/12
  Installing     : vmware-open-vm-tools                                                                                      9/12
  Installing     : vmware-tools                                                                                             10/12
  Erasing        : xorg-x11-drv-vmmouse                                                                                     11/12
  Erasing        : xorg-x11-drv-vmware                                                                                      12/12

Installed:
  vmware-open-vm-tools-xorg-drv-display.x86_64 0:10.16.7.0-0.257589.el5
  vmware-open-vm-tools-xorg-drv-mouse.x86_64 0:12.6.4.0-0.257589.el5
  vmware-tools.x86_64 0:8.3.2-257589.el5

Dependency Installed:
  vmware-open-vm-tools.x86_64 0:8.3.2-257589.el5                      vmware-open-vm-tools-common.x86_64 0:8.3.2-257589.el5
  vmware-open-vm-tools-kmod.x86_64 0:8.3.2-257589.el5                 vmware-open-vm-tools-nox.x86_64 0:8.3.2-257589.el5
  vmware-open-vm-tools-xorg-utilities.x86_64 0:8.3.2-257589.el5       vmware-tools-common.x86_64 0:8.3.2-257589.el5
  vmware-tools-nox.x86_64 0:8.3.2-257589.el5

Replaced:
  xorg-x11-drv-vmmouse.x86_64 0:12.4.0-2.1                        xorg-x11-drv-vmware.x86_64 0:10.13.0-2.1

Complete!

 The experience will be very similar -- if a bit more verbose.


That, my friends, may be it.  It seems convoluted but, really, once you ship a metapackage out with a proper apt+yum config, and your Linux VMs upgrade themselves overnight (well, maybe the test ones get the new metaRPMs first) then you'll be so much more pleased at the ability to do all that automatically.  Add in Spacewalk and/or cobbler and stir for best results.

Disabling LACP with iSCSI in ESXi4 to Enable MPIO

My mini Atlantis cluster is working well, but I found a document suggesting it can be better -- or another way that it can be better, at least.

It's using iSCSI for the storage, right now, and I have it configured with LACP for redundancy off one bonded channel.  I'd hoped to also get some load-balancing from it, but I was young and naive (last week) when I set it up.  I've since learned of a better way to set it up.  It's all in A Multivendor Post on using iSCSI with VMware vSphere at Virtual Geek .

Here's the money-shot, as I told DD in email:

1. don't have your iSCSI set up with LACP
2. set up one vmkX for each NIC you're going to dedicate for iSCSI
3. have them not team
4. subscribe to each effective target on each interface on the remote side separately
5. drop your IOps per path factor to a lower number to reduce stickiness on a particular path and improve the MP part of the IO on lower-speed iSCSI configs.

Not sure about the last part, but it's what passes for the sense it's making right now.  I'm off to read it more fully and plan a migration from one method to another in Atlantis.

I'm actually excited that I can improve performance on this little cluster without requiring more capital.  Let you know how it goes.