Updating Cobbler on RHEL5 to Import ESXi5 for PXE Install

So you want to play with ESXi5, but on RHEL5, today, the latest version of cobbler (2.2.1?) doesn't yet support it.  Cobbler 2.2.3 does, though, but it's not available at all yet.  Nooooooo.

Here's what you do:

1. install an RPMbuild of cobbler 2.2.3 .  Normally this is laughable-stupid, but we know the official one's coming eventually and we'll get the update from the official channels automatically.
   1. From the cobbler release_22 branch, go get the TGZ file for the latest 2.2 release:
      https://github.com/cobbler/cobbler/tarball/release22
   2. unpack that
      tar -C /tmp -xf cobbler-cobbler-cobbler-2.2.3-2-0-g80d646a.tar.gz(or just do it all in one go:
        wget -qO-  https://github.com/cobbler/cobbler/tarball/release22 | tar -C /tmp -xzf - )
   3. change and build it
      make -C /tmp/cobbler-cobbler-80d646a  rpms
   4. if you're me, you'll then remake it just to get the right apt repo, but hey.  People like to make things inside-out, and forget that the packaging is on the outside of the build.  Be nice to them, though, as they may be amateurs, and it's easy like rolling a stop-sign.
   5. install cobbler over what you have:
      rpm -Uvh  /tmp/cobbler-cobbler-80d646a/rpm-build/cobbler-2.2.3-2.noarch.rpm
   6. watch it fail when you do a cobbler check due to a missing and unreported dependency:
      cobbler check
   7. install said dependency, re-run cobbler check, think disapproving thoughts of the packagers:
      yum -y install python-ctypes
      cobbler check
2. grab your ESXi5 ISO
3. import that
   mount -o loop VMware-VMvisor-Installer-5.0.0.update01-623860.x86_64.iso /mnt/cdrom
   cobbler import -name esxi5 --path /mnt/cdrom/
4. rejoice
   more beer

It really is that easy, once you have a cheat-sheet.  Now go hack up a decent esxi5 KS and start installing like a boss.

Snarling little Upstart

The ubuntification of RedHat continues.  Upstart comes to RHEL6 and derivatives (including derivatives of derivatives, like OEL).

While perusing the net and stalling for clue on upstart, I ran across one brochure piece:  Why Upstart Is Good For Your Distro, in which the author says:

• Helen is an experienced UNIX user, with multiple years of experience. She does not wish to have to relearn that which she has learned already, and would rather continue using the tools that she is used to and only learn the newer ones when necessary.
• Matthieu is a distribution developer who maintains several packages that provide services or perform tasks. He does not want to have to update his packages until he is ready to take advantage of new features or abilities, his existing scripts should continue to work unmodified in their original locations.

The thing is, he says it in his brochure piece as if Upstart provides those features.  Did he see /etc/inittab?

# inittab is only used by upstart for the default runlevel.
#
# ADDING OTHER CONFIGURATION HERE WILL HAVE NO EFFECT ON YOUR SYSTEM.

Apparently retaining compatibility with the industry standard is the first casualty of the Shiny Things Gold Rush infection that has now spread to RedHat.  Leave that half-baked new-is-instantly-better crap for the Fedora derivatives that don't make it to the enterprise product, if you want to preserve compatibility.

Cobbler 2.0.11 Hack to Limit Rsync Bandwidth

I was having a problem with one of my cobbler app installs, where the rsync run would overload the already-loaded network signal. This is a pipe where we're already doing a lot of management, because the physical premises are not conducive to an upgrade (and we're not doing QoS yet; I know, I know).

Lanton Vhengani was also having the same problem back in 2008, and although Mr DeHaan was considering adding a switch to the rsync invocation from within the cobbler app, it's not yet appeared.  Lanton found the section where the rsync invocation is literally called out, and provided a great patch hint for the source.  It still works under 2.0.11 .

diff -uBb /usr/lib/python2.4/site-packages/cobbler/action_reposync.py\~ /usr/lib/python2.4/site-packages/cobbler/action_reposync.py
--- /usr/lib/python2.4/site-packages/cobbler/action_reposync.py~        2011-04-20 08:40:48.000000000 -0400
+++ /usr/lib/python2.4/site-packages/cobbler/action_reposync.py 2011-07-16 22:00:22.000000000 -0400
@@ -220,6 +220,7 @@
         if not repo.mirror.endswith("/"):
             repo.mirror = "%s/" % repo.mirror

+        spacer = " --bwlimit=50"
         # FIXME: wrapper for subprocess that logs to logger
         cmd = "rsync -rltDv %s --delete --exclude-from=/etc/cobbler/rsync.exclude %s %s" % (spacer, repo.mirror, dest_path)
         rc = utils.subprocess_call(self.logger, cmd)

restart cobblerd before testing, and there you go:  Bandwidth isn't pinned and the boss is happier.  That's a pretty small setting there, but I'll open it up after Centos6 comes down completely -- the periodic kicks the pipe will take when 1-2 RPMs are updated will be so small I can push it up a bit higher.

Automatically Changing Sound Preferences in OSX with USB insertion

Seems simple enough:  when we insert a USB device, or ideally when a Bluetooth device is paired and lit up, I'd like the customer's Mac OS/X box to jigger the sound prefs to fit the pre-configured settings for the new device, and set it as the output device automatically.  So, when we light up a headset, skype and wow and all that will just work the way it should.

I'm thinking there's gotta be a HAL-like dealy for Macs, and that we can hook in a script in there.  That's what I'm thinking.  There's a potential script we can run here, but the trick is finding out how/where the subsystem lives.  Until then, we could be stuck with cruder, user-diddled tools like

• SoundSource at Rogue Amoeba
• Change audio devices via shell script at who's hacks 

Here's hoping we can make it more automated.  It seems Apple machines should do this, if you believe the hype.  Me, I don't have nearly enough/any Mac OS/X experience.

Maintaining Repos in Kickstarted Machines After Install

After you've installed a machine, its install-time repository config in /etc/yum.repos.d is pretty much set.

Bah, I say! Bah! Just keep it updated.

Kickstart (cobbler):

#set yumconfcronfilename = "/etc/cron.daily/50-yum-config-stanza"
cat << EOECYCS > $yumconfcronfilename
#!/bin/sh
$yum_config_stanza

sed -ne '
        /^baseurl=/{
                s/baseurl=/repomd /
                s://:__:
                s:/: :
                s:__://:
                p
        }
        ' /etc/yum.repos.d/cobbler-config.repo \
          > /etc/apt/sources.list.d/cobbler-config.list
EOECYCS
chmod a+x $yumconfcronfilename

If you're not running cobbler, set it into place by hand:

cat << EOECYCS > /etc/cron.daily/50-yum-config-stanza
#!/bin/sh
wget "http://archive/cblr/svc/op/yum/profile/centos5-i386-minimal" --output-document=/etc/yum.repos.d/cobbler-config.repo

sed -ne '
 /^baseurl=/{
  s/baseurl=/repomd /
  s://:__:
  s:/: :
  s:__://:
  p
 }
 ' /etc/yum.repos.d/cobbler-config.repo \
   > /etc/apt/sources.list.d/cobbler-config.list
EOECYCS

chmod a+x /etc/cron.daily/50-yum-config-stanza

That's dereferenced for you. The actual profile's going to be way off, though, so don't use that one verbatim. Find your own:

awk -F/ '/^url/{print $NF}' anaconda-ks.cfg

As usual, watch carefully for the way in which the 'new', 'better' blogspot editor makes an artistic puree of the quoted stuff;  grain of salt, kids.

Whither be Withered Stateless Linux?

I've been pocking at Stateless Linux for a while.

Probably since it was called Diskless Linux.  Names change.

Anyway, I stumbled over a stateless linux page referring to a tech preview in RHEL5, which sounds awesome.

Sadly it doesn't appear to be available.  "Install FC7," it says, not knowing that FC7's stereotypical 3-week support window closed at least a week ago, and thus any projects using it will simply not work.

So where did the Tech Preview go?  If it was a tech preview way back in 1992 or whenever RHEL5 was released, should it not be at least a tech preview now?

I worry that the Open Source Community has an attention span even shorter than mine.

Kickstarting ESX VMs and Physical Hosts -- Knowing Which is Which

UPDATE: This method also does not work. Sorry.

Cobbler and kickstarting is my new cool toy.  I tinker with it FAR too much. I had a problem, though, that I need to install the vmware tools only on the VMs, and install smartmontools only on the physical hosts.  What's a guy to do?

After some digging, coding, hacking, testing, cursing, I finally discovered a decent switch I can use to identify a box by its mac.

Then I lost that code.

So I found another method.  This one's ugly as sin, but it may actually work.  Check this nasty-ass kung-fu:

#if ":".join($interfaces.eth0.mac_address.split(':')[0:3]) in "00:50:56 00:0C:29"

[code]

#end if

Yeah.  That's one ugly baby.  If Blogspot again truncates that line all to hell, remember the #if statement is all on the first line of 3.

Now to see how well it works.

Pruning Bogus Value Spikes from MRTG Gauge Values after a Reboot

MRTG is awesome and simple.  I use it in a bunch of places to quickly figure out roughly what's going on over my networks and servers.  Traffic numbers, disk space, the usual stuff.  My usage of it is strictly ghetto, but it works really well for me even when I misuse it.

When I reboot a machine, though, I find my gauges all spike.  Traffic numbers, CPU load, etc, there's a spike right at the point where I rebooted the machine.  It seems to be that MRTG sees the gauge is reset, but it treats it as an overflow, as if the gauge wrapped around, and not as if it's been zeroed out.  So the spike may be its attempt to account for the massive jump in data that would cause a wraparound.

the only solution I have just yet - because I can't use ABSMax or MaxBytes parameters - is to prune the ugly data points:

val=500000; \
for F in /var/www/html/mrtg/{cpu,dev-*}/*.{log,old};\
  do awk -vX=$val '
    NF < 4 || ($2 < X && $3 < X && $4 < X && $5 < X)
  ' $F | diff -u $F - | patch $F;\
done

Really, though, you should use the ABSMax and MaxBytes parameters whenever and wherever you can.  It'll prevent this spike when you reset your machine.

Finally, I'm sorry if the above code snippet looks like absolute ass.  There is some formatting for reading ease, while I usually do it all on one line, but also what is with the pathetic format munging in this 'new, better' editor?  It's horrid!  Can we roll it back, please?

Hot Remove VMDKs in Linux VMs

We glue vHDDs onto VMs like crazy. It's like a sport or something. The awesome part is that I just found an article on Xtravirt showing you how to remove one.

Awesome.  So here's the process:

1. umount the drive.
2. pull the drive from /etc/fstab, if necessary.
3. here's the magic.  Unlink the drive from the HBA:
   echo 1 > /sys/block/sdb/device/delete
   oh yeah. That hits the spot.
4. remove the VMDK from the VM via the VIC.

And that's it.  Holy crap.


I'm not including this to take credit for it.  Personally, I never knew we had the power in Linux, yet, to yank a drive off the bus like that.  It's awesome, and it shows what kind of awesome power we have here.

Go see the article at its source:

vSphere: Hot Add or Remove a VMDK with a Linux VM | Xtravirt

vLANs on Linux (physical) Hosts

vLANs on VMware virtual linux guests is easy.  Set it and you're done, right?

On physical hosts, though, it can be a bit more tricky.  It seems the format's changed a bit, too, between RHEL4 and RHEL5 -- for the better!

By example, though, to add vLan 101 to your eth2, it's like this:

(emacs /etc/sysconfig/network-scripts/ifcfg-eth2.101)VLAN=yes
DEVICE=eth2.101ONBOOT=yes

BOOTPROTO=none
IPADDR=10.101.2.17NETMASK=255.255.255.0

 Yeah, it's that easy.  The scripts extract the device name (eth2) and vLan (101) from the DEVICE tag, if it sees the VLAN=yes setting.  When you're done:

ifup eth2.101

Your machine should (wait an agonizing 2 seconds and) display something like this:

Added VLAN with VID == 101 to IF -:eth2:-

And you know you're done.  You can use all the tools you would expect, just like with regular devices and/or ipaliased devices:

ifconfig eth2.101
tshark -i eth2.101 host sniffme.mynet.com
ping -I eth2.101 sniffme

One bit of caution, though:  vLANs are privacy; not security.  What this means is that you should never have vLANs carrying data where anything-but-completely trustable machines can hear it, even if it's in a vLAN.  It's trivial to peel the leading vLAN tag off a packet and read it from the untagged network (eg eth2).

Get your stuff onto a managed switch!  Segregate your traffic!  If you aren't 100% sure, the best security between two networks is an air gap.

Pra Pra Pra

I've lost a machine.

In the monster setup I'm doing in the 404, I blew a static DHCP setup and directed an IPMI NIC over to an address it couldn't have -- it was taken.  After the dust cleared from the resulting fight between it and a photocopier that was squatting on that IP - right in the middle of the range I'd pre-allocated - the IPMI NIC seems to have gone away.  It's not at the IP it should be, it's not at the IP I set aside for it during the squabble and to which it never went, and it's not sleuthable at a 169.254.x.y IP as it also could be.  I'm all but sunk on that one machine until I can get a human to dump the power on the box (that's why we have IPMI) and bring it back up.  It should kick the IPMI back onto the state table and get it networking again.  Ugh.

I've since learned two things:  arping and inarp.

Arping, here, helps me ensure an IP is really vacant before I allocate it.  DHCP should, too, but it wasn't doing so well in this case since I hard-coded it into the config.  This, folks, is why we define pools and stick to them.  All the problems with "dhcp causing IP conflicts", as some hicks in the 561 claimed, were all through mis-use like I did here.  So, do this next time:

# arping -D 192.168.1.6 -w 5
ARPING 192.168.1.6 from 0.0.0.0 eth0
Unicast reply from 192.168.1.6 [00:04:01:EF:D8:4C] for 192.168.1.6 [00:04:01:EF:D8:4C] 0.650ms
Sent 1 probes (1 broadcast(s))
Received 1 response(s)

arping -D 192.168.1.112 -w 5
ARPING 192.168.1.112 from 0.0.0.0 eth0
Sent 6 probes (6 broadcast(s))
Received 0 response(s)

And we can be sure that something's there if it replied within 5 seconds.

The next thing, which I dearly need now, is a linux implementation of Inverse ARP, or inarp.  It's different from RARP ("who am I on this MAC") because it solicits for a third party, essentially asking "What IP is on that MAC?"

No inarp.c though.  I found one for Windows, but I don't have control of a winbox on that network, where I can inject arps.  Argh.

Anyone found a decent Inverse ARP tool for linux?  I'll even take one in perl.

Busy

I'm nearing the end of my project.  That's it, mostly, depicted on the left.

I use a bugzilla to keep my sanity, and I break jobs down into discrete chunks.  I keep it mostly macro-atomic, when possible, and group whole tasks together, but I often need to break things up into separate bits so that I can finish the job in a set period of time and wait some unknown period before the next chunk begins.

I also use dependencies in BZ a lot.  I use them for not only a job that follows another, as you'd expect, but also to indicate a common component or the previous version of a repeated task or issue.  this drives everyone else nuts.

The one above is the report of a drive failure killing a RAID and tanking a VM - no really - but also deals with the purchase of some larger hard drives, their install, the migration onto the new drives from the old and it will show the removal of the old hardware as well.

In this way, BZ shows me a project at a glance, and it's easy to see my place in that project.  For a guy who constantly needs reminding where he is in projects, be they an hour or a year old, or is coming back to something with all but completely fresh eyes, the ability to quickly see my place in the grand scheme of things is invaluable.

Go get Bugzilla.  Or something.  And use it.