Updating Cobbler on RHEL5 to Import ESXi5 for PXE Install

So you want to play with ESXi5, but on RHEL5, today, the latest version of cobbler (2.2.1?) doesn't yet support it.  Cobbler 2.2.3 does, though, but it's not available at all yet.  Nooooooo.

Here's what you do:

1. install an RPMbuild of cobbler 2.2.3 .  Normally this is laughable-stupid, but we know the official one's coming eventually and we'll get the update from the official channels automatically.
   1. From the cobbler release_22 branch, go get the TGZ file for the latest 2.2 release:
      https://github.com/cobbler/cobbler/tarball/release22
   2. unpack that
      tar -C /tmp -xf cobbler-cobbler-cobbler-2.2.3-2-0-g80d646a.tar.gz(or just do it all in one go:
        wget -qO-  https://github.com/cobbler/cobbler/tarball/release22 | tar -C /tmp -xzf - )
   3. change and build it
      make -C /tmp/cobbler-cobbler-80d646a  rpms
   4. if you're me, you'll then remake it just to get the right apt repo, but hey.  People like to make things inside-out, and forget that the packaging is on the outside of the build.  Be nice to them, though, as they may be amateurs, and it's easy like rolling a stop-sign.
   5. install cobbler over what you have:
      rpm -Uvh  /tmp/cobbler-cobbler-80d646a/rpm-build/cobbler-2.2.3-2.noarch.rpm
   6. watch it fail when you do a cobbler check due to a missing and unreported dependency:
      cobbler check
   7. install said dependency, re-run cobbler check, think disapproving thoughts of the packagers:
      yum -y install python-ctypes
      cobbler check
2. grab your ESXi5 ISO
3. import that
   mount -o loop VMware-VMvisor-Installer-5.0.0.update01-623860.x86_64.iso /mnt/cdrom
   cobbler import -name esxi5 --path /mnt/cdrom/
4. rejoice
   more beer

It really is that easy, once you have a cheat-sheet.  Now go hack up a decent esxi5 KS and start installing like a boss.

(Discovered) Best Practice Config for OpenFiler iSCSI Storage for ESXi4

Update: OpenFiler performance and reliability remained low. I've since moved to another platform, and no, this isn't an ad so I'm not saying where I ended up.

That title sucks, but I'm hoping Google will like it.

The OpenFiler iSCSI box at the Eastern site bailed, due largely to congestion on the still-not-redundant switch we have in place (new stuff still in shipping, argh) and after the fallout I found some config tips for the OF 2.3 server.  This thing's a physical box with nothing but iSCSI on it, so it may be different if you're simulating some ESXi-ESXi virtual worlds in your mini-lab at work.

1. Disable Delayed ACK in the iSCSI storage adapter advanced properties on the ESXi host.  (scroll down)(via)
2. Reduce the VMs-per-LUN count to reduce the reservation count and conflict  Smaller LUNs, kids, despite what you hear.(via)
3. Change the default iSCSI timeout on your ESX server to 14 seconds: esxcfg-advcfg -s 14000 /VMFS3/HBTokenTimeout (via, via, via)
4. Keep your snapshots down:  they increase load, with the obvious impact.(via)
5. Consider upgrading your intel e1000 or exgb NICs, code here, intel instructions here, openFiler-derived procedure here.
6. Don't use more than one NIC (bonding) or more than one interface (different subnets) at a time with Openfiler. (via, via, instructions for OF)
7. Target tuning (via):
   1. DefaultTime2Wait should be changed to "10"
   2. ImmediateData should be changed to "Yes"
   3. MaxRecvDataSegmentLength should be changed to "262144"
   4. MaxXmitDataSegmentLength should be changed to "262144"
8. LUN tuning
   1. R/W mode should be changed to "write-back"
      (it should go without saying that a BBU is essential for your RAID controller)
   2. Transfer mode should be changed to "fileio"
      (this is controversial)
9. NEVER, EVER share LUNs between ESX hosts using Openfiler and IET.
   (Not sure how doable this one is in an ESXi cluster, but that's the suggestion.) (via)

So in my case, we're looking to implement a few of those.  Hopefully we'll not see this kind of issue any time soon, but watch this space in case we do see something after the upgrade/tuning.

The good news is that OF is considering another iSCSI target, and the changes are supposed to be very beneficial for performance and for us.

Hot Remove VMDKs in Linux VMs

We glue vHDDs onto VMs like crazy. It's like a sport or something. The awesome part is that I just found an article on Xtravirt showing you how to remove one.

Awesome.  So here's the process:

1. umount the drive.
2. pull the drive from /etc/fstab, if necessary.
3. here's the magic.  Unlink the drive from the HBA:
   echo 1 > /sys/block/sdb/device/delete
   oh yeah. That hits the spot.
4. remove the VMDK from the VM via the VIC.

And that's it.  Holy crap.


I'm not including this to take credit for it.  Personally, I never knew we had the power in Linux, yet, to yank a drive off the bus like that.  It's awesome, and it shows what kind of awesome power we have here.

Go see the article at its source:

vSphere: Hot Add or Remove a VMDK with a Linux VM | Xtravirt

Installing VMware Tools in Linux after ESX 4.1

Tried to install VMware-Tools lately?  Not sure how it goes on Windows, but on Linux it's a bit more challenging than it used to be.  Used to be there was an RPM on the ISO that popped into the Linux machine's vCDROM drive when the option to install VMware Tools was chosen.

No more.  Like ESXi4.0 before it, ESX4.1 seems to ship with an ISO containing, for Linux, a little tarball with the stuff.

Those who've worked for a distro or packaged anything will know that shipping and installing bare tarballs is really not the best way to go.  Sure, it's fine for the cowboys who maintain 1-2 machines at the home or home-office, but for anyone who has to maintain more than one, has to roll back or verify a package level, or who just wants a clean system, a good package is the only way to go.

Note that I said good package.  If you hear anyone mention "dependency hell" or anything like that, they weren't paying attention.  Stop using bad packages!  Stop mixing repositories.  The pathetic blunt object that pass for packages is the topic of many blogwails past and many blogwails to come.

The great news?  VMware seem to have a few packages for us.  In fact, they come in 4 flavours of RedHat and some even for the you-bunt-you crowd.  Seems all you have to do is mark up the apt config ...

cat <<EOF> /etc/apt/sources.list.d/esx.list

repomd  http://packages.vmware.com tools/esx/4.1/rhel$(VERSION)/$(ARCH)
EOF
apt-get update

... or yum config if you don't build your own packages and know where apt excels ...

cat <<EOF> /etc/yum.repos.d/esx.repo

[esx]
name=VMware-tools for RHEL\$releasever - \$basearch

baseurl=http://packages.vmware.com/tools/esx/4.1/rhel\$releasever/\$basearch
enabled=1
gpgcheck=1
gpgkey=http://packages.vmware.com/tools/VMWARE-PACKAGING-GPG-KEY.pub

EOF

yum update

and after refreshing, just install the package:

apt-get install vmware-tools
Reading Package Lists... Done
Building Dependency Tree... Done
The following extra packages will be installed:
   vmware-open-vm-tools (8.3.2-257589.el5)
   vmware-open-vm-tools-common (8.3.2-257589.el5)
   vmware-open-vm-tools-kmod (8.3.2-257589.el5)
   vmware-open-vm-tools-nox (8.3.2-257589.el5)
   vmware-open-vm-tools-xorg-drv-display (10.16.7.0-0.257589.el5)
   vmware-open-vm-tools-xorg-drv-mouse (12.6.4.0-0.257589.el5)
   vmware-open-vm-tools-xorg-utilities (8.3.2-257589.el5)
   vmware-tools-common (8.3.2-257589.el5)
   vmware-tools-nox (8.3.2-257589.el5)
The following packages will be REPLACED:
   xorg-x11-drv-vmmouse (12.4.0-2.1)
   (by (10.13.0-2.1)
   vmware-open-vm-tools-xorg-drv-mouse) ()
    (12.4.0-2.1)
   xorg-x11-drv-vmware (10.13.0-2.1)
   (by ()
   vmware-open-vm-tools-xorg-drv-display) (12.4.0-2.1)
The following NEW packages will be installed:
   vmware-open-vm-tools (8.3.2-257589.el5)
   vmware-open-vm-tools-common (8.3.2-257589.el5)
   vmware-open-vm-tools-kmod (8.3.2-257589.el5)
   vmware-open-vm-tools-nox (8.3.2-257589.el5)
   vmware-open-vm-tools-xorg-drv-display (10.16.7.0-0.257589.el5)
   vmware-open-vm-tools-xorg-drv-mouse (12.6.4.0-0.257589.el5)
   vmware-open-vm-tools-xorg-utilities (8.3.2-257589.el5)
   vmware-tools (8.3.2-257589.el5)
   vmware-tools-common (8.3.2-257589.el5)
   vmware-tools-nox (8.3.2-257589.el5)
0 upgraded, 10 newly installed, 2 replaced, 0 removed and 0 not upgraded.
Need to get 44.8kB/13.7MB of archives.
After unpacking 41.1MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-kmod 8.3.2-257589.el5 [524kB]
Get:2 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-common 8.3.2-257589.el5 [5283kB]
Get:3 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-nox 8.3.2-257589.el5 [2631B]
Get:4 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-xorg-drv-mouse 12.6.4.0-0.257589.el5 [17.5kB]
Get:5 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-xorg-drv-display 10.16.7.0-0.257589.el5 [33.4kB]
Get:6 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-xorg-utilities 8.3.2-257589.el5 [7828kB]
Get:7 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools 8.3.2-257589.el5 [2811B]
Get:8 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-tools-common 8.3.2-257589.el5 [39.4kB]
Get:9 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-tools-nox 8.3.2-257589.el5 [2635B]
Get:10 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-tools 8.3.2-257589.el5 [2763B]
Fetched 44.8kB in 0s (46.5kB/s)
Committing changes...
Preparing                                ############################## [100%]
Updating / installing
  vmware-open-vm-tools-xorg-drv-display- ############################## [100%]
  vmware-open-vm-tools-xorg-drv-mouse-12 ############################## [100%]
  vmware-open-vm-tools-kmod-8.3.2-257589 ############################## [100%]
  vmware-open-vm-tools-common-8.3.2-2575 ############################## [100%]
  vmware-open-vm-tools-nox-8.3.2-257589. ############################## [100%]
  vmware-open-vm-tools-xorg-utilities-8. ############################## [100%]
  vmware-open-vm-tools-8.3.2-257589.el5. ############################## [100%]
  vmware-tools-common-8.3.2-257589.el5.x ############################## [100%]
  vmware-tools-nox-8.3.2-257589.el5.x86_ ############################## [100%]
  vmware-tools-8.3.2-257589.el5.x86_64   ############################## [100%]
Cleaning up / removing
  xorg-x11-drv-vmmouse-12.4.0-2.1.x86_64 ############################## [100%]
  xorg-x11-drv-vmware-10.13.0-2.1.x86_64 ############################## [100%]
Done.

While I avoid Yum, it should look like this:

[root@0-50-56-bc-0-7 ~]# yum install vmware-tools
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * addons: mirror.raystedman.net
 * base: mirror.skiplink.com
 * extras: mirror.skiplink.com
 * updates: styx.biochem.wfubmc.edu
Setting up Install Process
Resolving Dependencies
:
[blah blah]
:
Dependencies Resolved

==================================================================================================================================
 Package                                           Arch               Version                               Repository       Size
==================================================================================================================================
Installing:
 vmware-open-vm-tools-xorg-drv-display             x86_64             10.16.7.0-0.257589.el5                esx              33 k
     replacing  xorg-x11-drv-vmware.x86_64 10.13.0-2.1

 vmware-open-vm-tools-xorg-drv-mouse               x86_64             12.6.4.0-0.257589.el5                 esx              17 k
     replacing  xorg-x11-drv-vmmouse.x86_64 12.4.0-2.1

 vmware-tools                                      x86_64             8.3.2-257589.el5                      esx             2.7 k
Installing for dependencies:
 vmware-open-vm-tools                              x86_64             8.3.2-257589.el5                      esx             2.7 k
 vmware-open-vm-tools-common                       x86_64             8.3.2-257589.el5                      esx             5.0 M
 vmware-open-vm-tools-kmod                         x86_64             8.3.2-257589.el5                      esx             512 k
 vmware-open-vm-tools-nox                          x86_64             8.3.2-257589.el5                      esx             2.6 k
 vmware-open-vm-tools-xorg-utilities               x86_64             8.3.2-257589.el5                      esx             7.5 M
 vmware-tools-common                               x86_64             8.3.2-257589.el5                      esx              39 k
 vmware-tools-nox                                  x86_64             8.3.2-257589.el5                      esx             2.6 k

Transaction Summary
==================================================================================================================================
Install      10 Package(s)
Upgrade       0 Package(s)

Total download size: 13 M
Is this ok [y/N]: y
Downloading Packages:
:
[ more blah blah ]
:
Running Transaction
  Installing     : vmware-open-vm-tools-xorg-drv-display                                                                     1/12
  Installing     : vmware-open-vm-tools-kmod                                                                                 2/12
  Installing     : vmware-open-vm-tools-common                                                                               3/12
  Installing     : vmware-open-vm-tools-nox                                                                                  4/12
  Installing     : vmware-tools-common                                                                                       5/12
  Installing     : vmware-tools-nox                                                                                          6/12
  Installing     : vmware-open-vm-tools-xorg-drv-mouse                                                                       7/12
  Installing     : vmware-open-vm-tools-xorg-utilities                                                                       8/12
  Installing     : vmware-open-vm-tools                                                                                      9/12
  Installing     : vmware-tools                                                                                             10/12
  Erasing        : xorg-x11-drv-vmmouse                                                                                     11/12
  Erasing        : xorg-x11-drv-vmware                                                                                      12/12

Installed:
  vmware-open-vm-tools-xorg-drv-display.x86_64 0:10.16.7.0-0.257589.el5
  vmware-open-vm-tools-xorg-drv-mouse.x86_64 0:12.6.4.0-0.257589.el5
  vmware-tools.x86_64 0:8.3.2-257589.el5

Dependency Installed:
  vmware-open-vm-tools.x86_64 0:8.3.2-257589.el5                      vmware-open-vm-tools-common.x86_64 0:8.3.2-257589.el5
  vmware-open-vm-tools-kmod.x86_64 0:8.3.2-257589.el5                 vmware-open-vm-tools-nox.x86_64 0:8.3.2-257589.el5
  vmware-open-vm-tools-xorg-utilities.x86_64 0:8.3.2-257589.el5       vmware-tools-common.x86_64 0:8.3.2-257589.el5
  vmware-tools-nox.x86_64 0:8.3.2-257589.el5

Replaced:
  xorg-x11-drv-vmmouse.x86_64 0:12.4.0-2.1                        xorg-x11-drv-vmware.x86_64 0:10.13.0-2.1

Complete!

 The experience will be very similar -- if a bit more verbose.


That, my friends, may be it.  It seems convoluted but, really, once you ship a metapackage out with a proper apt+yum config, and your Linux VMs upgrade themselves overnight (well, maybe the test ones get the new metaRPMs first) then you'll be so much more pleased at the ability to do all that automatically.  Add in Spacewalk and/or cobbler and stir for best results.

Disabling LACP with iSCSI in ESXi4 to Enable MPIO

My mini Atlantis cluster is working well, but I found a document suggesting it can be better -- or another way that it can be better, at least.

It's using iSCSI for the storage, right now, and I have it configured with LACP for redundancy off one bonded channel.  I'd hoped to also get some load-balancing from it, but I was young and naive (last week) when I set it up.  I've since learned of a better way to set it up.  It's all in A Multivendor Post on using iSCSI with VMware vSphere at Virtual Geek .

Here's the money-shot, as I told DD in email:

1. don't have your iSCSI set up with LACP
2. set up one vmkX for each NIC you're going to dedicate for iSCSI
3. have them not team
4. subscribe to each effective target on each interface on the remote side separately
5. drop your IOps per path factor to a lower number to reduce stickiness on a particular path and improve the MP part of the IO on lower-speed iSCSI configs.

Not sure about the last part, but it's what passes for the sense it's making right now.  I'm off to read it more fully and plan a migration from one method to another in Atlantis.

I'm actually excited that I can improve performance on this little cluster without requiring more capital.  Let you know how it goes.