ReCollecting the FileProvides

I use APT for RPMs, as it's the most versatile tool out there.  Some folks don't seem to understand that APT was ported for RPMs about a decade ago, and those using Yum for their RPM management are encouraged to do so, lest their horizons grow too broad.  The truly genius folks at Conectiva used APT to great benefit, and allowed massive flexibility;  upgrading in-place over a major release was a possible, easy and well-tested procedure, as I recall.  But this isn't for the Apt-RPM genius so recently localized at Conectiva and now lost to the rest of the world.

Lately there's been another change to the repository format and layout as used by YUM;  this is nothing new, as many open-source de-facto standards are developed and grow in the same kind of seeming vacuum, and the gaijin must adapt.  If you're suffering as I was, your apt-get update invocations will look like this:

E: Error occured while processing sec (CollectFileProvides)
E: Problem with MergeFileProvides /var/lib/apt/lists/archive_cobbler_repo%5fmirror_epel5-i386_repodata_277d21a2341fe766d0daff22b2846905517bcd71-primary.sqlite

I'm using Cobbler to mirror many, many repos locally; almost anywhere I have more than one machine.  The update to the createrepo invocation will either come from /etc/cobbler/settings or the createrepo flags in a particular repository item.

Here's my change:

--- /etc/cobbler/settings~      2011-09-23 07:03:38.000000000 -0700
+++ /etc/cobbler/settings       2011-09-26 14:24:32.000000000 -0700
@@ -58,7 +58,7 @@
 # enables working with Fedora repos from F11/F12 from EL-4 or
 # EL-5 without python-hashlib installed (which is not available
 # on EL-4)
-createrepo_flags: "-c cache -s sha -C --update"
+createrepo_flags: "-q -c cache -s sha --update -d"

 # if no kickstart is specified to profile add, use this template
 default_kickstart: /var/lib/cobbler/kickstarts/default.ks

You will need to adjust to suit, as the patch will not apply cleanly unless you've been mucking about there already, but take the patch and make yours match the result.  Restart cobbler when you're done and your repos will be updating madly as expected.

As always, sorry if the Blogspot editor butchers my blockquotes.

Cobbler 2.0.11 Hack to Limit Rsync Bandwidth

I was having a problem with one of my cobbler app installs, where the rsync run would overload the already-loaded network signal. This is a pipe where we're already doing a lot of management, because the physical premises are not conducive to an upgrade (and we're not doing QoS yet; I know, I know).

Lanton Vhengani was also having the same problem back in 2008, and although Mr DeHaan was considering adding a switch to the rsync invocation from within the cobbler app, it's not yet appeared.  Lanton found the section where the rsync invocation is literally called out, and provided a great patch hint for the source.  It still works under 2.0.11 .

diff -uBb /usr/lib/python2.4/site-packages/cobbler/action_reposync.py\~ /usr/lib/python2.4/site-packages/cobbler/action_reposync.py
--- /usr/lib/python2.4/site-packages/cobbler/action_reposync.py~        2011-04-20 08:40:48.000000000 -0400
+++ /usr/lib/python2.4/site-packages/cobbler/action_reposync.py 2011-07-16 22:00:22.000000000 -0400
@@ -220,6 +220,7 @@
         if not repo.mirror.endswith("/"):
             repo.mirror = "%s/" % repo.mirror

+        spacer = " --bwlimit=50"
         # FIXME: wrapper for subprocess that logs to logger
         cmd = "rsync -rltDv %s --delete --exclude-from=/etc/cobbler/rsync.exclude %s %s" % (spacer, repo.mirror, dest_path)
         rc = utils.subprocess_call(self.logger, cmd)

restart cobblerd before testing, and there you go:  Bandwidth isn't pinned and the boss is happier.  That's a pretty small setting there, but I'll open it up after Centos6 comes down completely -- the periodic kicks the pipe will take when 1-2 RPMs are updated will be so small I can push it up a bit higher.

Cheap Mutex with Fuser

On one box, I have a cronjob which runs every minute.  The cron task grabs a snapshot from a web cam, saves it in a monster tree, based on date and time, and then pushes it up to a larger 'mother ship' computer for it to process.  Don't get creeped out -- it's a time-laps movie of a building being built.

Anyway, it's that last rsync that sucks.  It's pushing stuff about 500mi away, and it's over SBC lines, so it's crunchy as hell.  Often, the VPN will die and take out not one but four sockets at once.  Of course, with cron, that means 4 email messages and one more every minute until that VPN comes back up.

So here's a cheap-ass mutex thing:

#!/bin/sh
lckfile=/tmp/foo
exec 2>${lckfile}
wget -O- http://www.cnn.com/
fuser ${lckfile}
echo $?

if ! fuser ${lckfile}
then
    fuser ${lckfile} > ${lckfile}
else
    touch /tmp/foot
fi
rm -f \
     /tmp/foop \
     /tmp/foot \
     ${lckfile}

fuser ${lckfile} ; echo $?

There.  This is a working demo I used to build the eventual thing, so the implementation varied as much as YMMV, but that's the guts of it.

Enjoy, kids. And yes, I enjoy foo.

NoStorage and Kickstart - How to Specify Multiple HBA Modules

When kickstarting, you have the option of using 'nostorage' on the PXE command line to prevent storage HBA drivers from loading -- you can do the same to NICs, but it's not as interesting, not as common and the command line is dumber.

default linux
prompt 0
timeout 1
label linux
     kernel /images/centos55-x86_64/vmlinuz
     ipappend 2
     append initrd=/images/centos55-x86_64/initrd.img ksdevice=eth0 lang= kssendmac nostorage text ks=http://archive/cblr/svc/op/ks/system/Bish-PXETest

See that?  NoStorage.  Okay.

So what if you want to use the same kickstart for different machines?  For different HBAs ?  Normally you're screwed.  This won't work, either:

device scsi ahci
device scsi mptspi
device scsi cciss

Specifying the HBA drivers on multiple lines should work, but it's not that simple -- doing so makes it choose the first one and ignore every other invocation of the device line.  This does work, though:

device scsi ahci:mptspi:cciss

See the colons? There you go.  It allows/forces you to choose the order, so plan carefully.

Maintaining Repos in Kickstarted Machines After Install

After you've installed a machine, its install-time repository config in /etc/yum.repos.d is pretty much set.

Bah, I say! Bah! Just keep it updated.

Kickstart (cobbler):

#set yumconfcronfilename = "/etc/cron.daily/50-yum-config-stanza"
cat << EOECYCS > $yumconfcronfilename
#!/bin/sh
$yum_config_stanza

sed -ne '
        /^baseurl=/{
                s/baseurl=/repomd /
                s://:__:
                s:/: :
                s:__://:
                p
        }
        ' /etc/yum.repos.d/cobbler-config.repo \
          > /etc/apt/sources.list.d/cobbler-config.list
EOECYCS
chmod a+x $yumconfcronfilename

If you're not running cobbler, set it into place by hand:

cat << EOECYCS > /etc/cron.daily/50-yum-config-stanza
#!/bin/sh
wget "http://archive/cblr/svc/op/yum/profile/centos5-i386-minimal" --output-document=/etc/yum.repos.d/cobbler-config.repo

sed -ne '
 /^baseurl=/{
  s/baseurl=/repomd /
  s://:__:
  s:/: :
  s:__://:
  p
 }
 ' /etc/yum.repos.d/cobbler-config.repo \
   > /etc/apt/sources.list.d/cobbler-config.list
EOECYCS

chmod a+x /etc/cron.daily/50-yum-config-stanza

That's dereferenced for you. The actual profile's going to be way off, though, so don't use that one verbatim. Find your own:

awk -F/ '/^url/{print $NF}' anaconda-ks.cfg

As usual, watch carefully for the way in which the 'new', 'better' blogspot editor makes an artistic puree of the quoted stuff;  grain of salt, kids.

Kickstarting ESX VMs and Physical Hosts -- Knowing Which is Which

UPDATE: This method also does not work. Sorry.

Cobbler and kickstarting is my new cool toy.  I tinker with it FAR too much. I had a problem, though, that I need to install the vmware tools only on the VMs, and install smartmontools only on the physical hosts.  What's a guy to do?

After some digging, coding, hacking, testing, cursing, I finally discovered a decent switch I can use to identify a box by its mac.

Then I lost that code.

So I found another method.  This one's ugly as sin, but it may actually work.  Check this nasty-ass kung-fu:

#if ":".join($interfaces.eth0.mac_address.split(':')[0:3]) in "00:50:56 00:0C:29"

[code]

#end if

Yeah.  That's one ugly baby.  If Blogspot again truncates that line all to hell, remember the #if statement is all on the first line of 3.

Now to see how well it works.

Auto-Updating Distro profile in Cobbler

As part of a very short, unsuccessful Fedora 13 test, I found it advantageous to set up an auto-updating distro.  Okay, given the release behaviour of Fedora vs, say, RHEL, an auto-updating distro profile for a distro which doesn't actually update is kinda pointless.  Work with me here, though.

How to do this in Cobbler?  Create a repo, which we can update automatically, and link the distro to that:

cobbler repo add --name fedora13-os-x86_64 --arch x86_64 \
  --mirror rsync://mirrors.kernel.org/fedora/releases/13/Fedora/x86_64/os

cobbler import --name fedora13 --arch x86_64 \
  --path /var/www/cobbler/repo_mirror/fedora13-os-x86_64 \
  --available-as http://10.10.4.1/cblr/repo_mirror/fedora13-os-x86_64 \
  --breed redhat --kickstart /etc/cobbler/centos-X.ks

Too easy.  Blah blah, test often, blah blah, change the IP to suit, etc.

Check the CommandLine Options in your PXE Booting Linux

I'm not sure if the grammar in the subject is the best it can be.

A question came up on a mailing list about command line options which can be passed to the linux installer at boot time.  The user in question is rolling out a series of new hosts with RocketRaid HBAs in them - for which I admire his courage where I gave up - and needs to push the driver disk to them at install time or they'll never see the root disks.  No, kickstart is not a viable option yet.

After some groveling through the options - and I suspect a RH employee who I fear had  to hold back for fear of breaching a Centos/RHEL condition - I think we got him exactly what he needs, which is awesome.

Want to look at those command line options in one go?  Try this:

zcat /tftpboot/images/centos5-x86_64/initrd.img | \
     cpio -iv --to-stdout sbin/loader|strings|less

Do that - adjusting for location - and start vgrepping.

For what it's worth, after jamming a fan onto my Adaptec 2405 (a Scythe Mini Kaze HTPC Silent Mini Fan 50MM 4500RPM 9.42CFM 26.09DBA 2 Pin W/ 3 Pin Adaptor, to be OCD about it) works like a charm.  Go get one of those and discard the HotPoint stuff, imho.

Installing VMware Tools in Linux after ESX 4.1

Tried to install VMware-Tools lately?  Not sure how it goes on Windows, but on Linux it's a bit more challenging than it used to be.  Used to be there was an RPM on the ISO that popped into the Linux machine's vCDROM drive when the option to install VMware Tools was chosen.

No more.  Like ESXi4.0 before it, ESX4.1 seems to ship with an ISO containing, for Linux, a little tarball with the stuff.

Those who've worked for a distro or packaged anything will know that shipping and installing bare tarballs is really not the best way to go.  Sure, it's fine for the cowboys who maintain 1-2 machines at the home or home-office, but for anyone who has to maintain more than one, has to roll back or verify a package level, or who just wants a clean system, a good package is the only way to go.

Note that I said good package.  If you hear anyone mention "dependency hell" or anything like that, they weren't paying attention.  Stop using bad packages!  Stop mixing repositories.  The pathetic blunt object that pass for packages is the topic of many blogwails past and many blogwails to come.

The great news?  VMware seem to have a few packages for us.  In fact, they come in 4 flavours of RedHat and some even for the you-bunt-you crowd.  Seems all you have to do is mark up the apt config ...

cat <<EOF> /etc/apt/sources.list.d/esx.list

repomd  http://packages.vmware.com tools/esx/4.1/rhel$(VERSION)/$(ARCH)
EOF
apt-get update

... or yum config if you don't build your own packages and know where apt excels ...

cat <<EOF> /etc/yum.repos.d/esx.repo

[esx]
name=VMware-tools for RHEL\$releasever - \$basearch

baseurl=http://packages.vmware.com/tools/esx/4.1/rhel\$releasever/\$basearch
enabled=1
gpgcheck=1
gpgkey=http://packages.vmware.com/tools/VMWARE-PACKAGING-GPG-KEY.pub

EOF

yum update

and after refreshing, just install the package:

apt-get install vmware-tools
Reading Package Lists... Done
Building Dependency Tree... Done
The following extra packages will be installed:
   vmware-open-vm-tools (8.3.2-257589.el5)
   vmware-open-vm-tools-common (8.3.2-257589.el5)
   vmware-open-vm-tools-kmod (8.3.2-257589.el5)
   vmware-open-vm-tools-nox (8.3.2-257589.el5)
   vmware-open-vm-tools-xorg-drv-display (10.16.7.0-0.257589.el5)
   vmware-open-vm-tools-xorg-drv-mouse (12.6.4.0-0.257589.el5)
   vmware-open-vm-tools-xorg-utilities (8.3.2-257589.el5)
   vmware-tools-common (8.3.2-257589.el5)
   vmware-tools-nox (8.3.2-257589.el5)
The following packages will be REPLACED:
   xorg-x11-drv-vmmouse (12.4.0-2.1)
   (by (10.13.0-2.1)
   vmware-open-vm-tools-xorg-drv-mouse) ()
    (12.4.0-2.1)
   xorg-x11-drv-vmware (10.13.0-2.1)
   (by ()
   vmware-open-vm-tools-xorg-drv-display) (12.4.0-2.1)
The following NEW packages will be installed:
   vmware-open-vm-tools (8.3.2-257589.el5)
   vmware-open-vm-tools-common (8.3.2-257589.el5)
   vmware-open-vm-tools-kmod (8.3.2-257589.el5)
   vmware-open-vm-tools-nox (8.3.2-257589.el5)
   vmware-open-vm-tools-xorg-drv-display (10.16.7.0-0.257589.el5)
   vmware-open-vm-tools-xorg-drv-mouse (12.6.4.0-0.257589.el5)
   vmware-open-vm-tools-xorg-utilities (8.3.2-257589.el5)
   vmware-tools (8.3.2-257589.el5)
   vmware-tools-common (8.3.2-257589.el5)
   vmware-tools-nox (8.3.2-257589.el5)
0 upgraded, 10 newly installed, 2 replaced, 0 removed and 0 not upgraded.
Need to get 44.8kB/13.7MB of archives.
After unpacking 41.1MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-kmod 8.3.2-257589.el5 [524kB]
Get:2 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-common 8.3.2-257589.el5 [5283kB]
Get:3 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-nox 8.3.2-257589.el5 [2631B]
Get:4 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-xorg-drv-mouse 12.6.4.0-0.257589.el5 [17.5kB]
Get:5 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-xorg-drv-display 10.16.7.0-0.257589.el5 [33.4kB]
Get:6 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-xorg-utilities 8.3.2-257589.el5 [7828kB]
Get:7 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools 8.3.2-257589.el5 [2811B]
Get:8 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-tools-common 8.3.2-257589.el5 [39.4kB]
Get:9 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-tools-nox 8.3.2-257589.el5 [2635B]
Get:10 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-tools 8.3.2-257589.el5 [2763B]
Fetched 44.8kB in 0s (46.5kB/s)
Committing changes...
Preparing                                ############################## [100%]
Updating / installing
  vmware-open-vm-tools-xorg-drv-display- ############################## [100%]
  vmware-open-vm-tools-xorg-drv-mouse-12 ############################## [100%]
  vmware-open-vm-tools-kmod-8.3.2-257589 ############################## [100%]
  vmware-open-vm-tools-common-8.3.2-2575 ############################## [100%]
  vmware-open-vm-tools-nox-8.3.2-257589. ############################## [100%]
  vmware-open-vm-tools-xorg-utilities-8. ############################## [100%]
  vmware-open-vm-tools-8.3.2-257589.el5. ############################## [100%]
  vmware-tools-common-8.3.2-257589.el5.x ############################## [100%]
  vmware-tools-nox-8.3.2-257589.el5.x86_ ############################## [100%]
  vmware-tools-8.3.2-257589.el5.x86_64   ############################## [100%]
Cleaning up / removing
  xorg-x11-drv-vmmouse-12.4.0-2.1.x86_64 ############################## [100%]
  xorg-x11-drv-vmware-10.13.0-2.1.x86_64 ############################## [100%]
Done.

While I avoid Yum, it should look like this:

[root@0-50-56-bc-0-7 ~]# yum install vmware-tools
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * addons: mirror.raystedman.net
 * base: mirror.skiplink.com
 * extras: mirror.skiplink.com
 * updates: styx.biochem.wfubmc.edu
Setting up Install Process
Resolving Dependencies
:
[blah blah]
:
Dependencies Resolved

==================================================================================================================================
 Package                                           Arch               Version                               Repository       Size
==================================================================================================================================
Installing:
 vmware-open-vm-tools-xorg-drv-display             x86_64             10.16.7.0-0.257589.el5                esx              33 k
     replacing  xorg-x11-drv-vmware.x86_64 10.13.0-2.1

 vmware-open-vm-tools-xorg-drv-mouse               x86_64             12.6.4.0-0.257589.el5                 esx              17 k
     replacing  xorg-x11-drv-vmmouse.x86_64 12.4.0-2.1

 vmware-tools                                      x86_64             8.3.2-257589.el5                      esx             2.7 k
Installing for dependencies:
 vmware-open-vm-tools                              x86_64             8.3.2-257589.el5                      esx             2.7 k
 vmware-open-vm-tools-common                       x86_64             8.3.2-257589.el5                      esx             5.0 M
 vmware-open-vm-tools-kmod                         x86_64             8.3.2-257589.el5                      esx             512 k
 vmware-open-vm-tools-nox                          x86_64             8.3.2-257589.el5                      esx             2.6 k
 vmware-open-vm-tools-xorg-utilities               x86_64             8.3.2-257589.el5                      esx             7.5 M
 vmware-tools-common                               x86_64             8.3.2-257589.el5                      esx              39 k
 vmware-tools-nox                                  x86_64             8.3.2-257589.el5                      esx             2.6 k

Transaction Summary
==================================================================================================================================
Install      10 Package(s)
Upgrade       0 Package(s)

Total download size: 13 M
Is this ok [y/N]: y
Downloading Packages:
:
[ more blah blah ]
:
Running Transaction
  Installing     : vmware-open-vm-tools-xorg-drv-display                                                                     1/12
  Installing     : vmware-open-vm-tools-kmod                                                                                 2/12
  Installing     : vmware-open-vm-tools-common                                                                               3/12
  Installing     : vmware-open-vm-tools-nox                                                                                  4/12
  Installing     : vmware-tools-common                                                                                       5/12
  Installing     : vmware-tools-nox                                                                                          6/12
  Installing     : vmware-open-vm-tools-xorg-drv-mouse                                                                       7/12
  Installing     : vmware-open-vm-tools-xorg-utilities                                                                       8/12
  Installing     : vmware-open-vm-tools                                                                                      9/12
  Installing     : vmware-tools                                                                                             10/12
  Erasing        : xorg-x11-drv-vmmouse                                                                                     11/12
  Erasing        : xorg-x11-drv-vmware                                                                                      12/12

Installed:
  vmware-open-vm-tools-xorg-drv-display.x86_64 0:10.16.7.0-0.257589.el5
  vmware-open-vm-tools-xorg-drv-mouse.x86_64 0:12.6.4.0-0.257589.el5
  vmware-tools.x86_64 0:8.3.2-257589.el5

Dependency Installed:
  vmware-open-vm-tools.x86_64 0:8.3.2-257589.el5                      vmware-open-vm-tools-common.x86_64 0:8.3.2-257589.el5
  vmware-open-vm-tools-kmod.x86_64 0:8.3.2-257589.el5                 vmware-open-vm-tools-nox.x86_64 0:8.3.2-257589.el5
  vmware-open-vm-tools-xorg-utilities.x86_64 0:8.3.2-257589.el5       vmware-tools-common.x86_64 0:8.3.2-257589.el5
  vmware-tools-nox.x86_64 0:8.3.2-257589.el5

Replaced:
  xorg-x11-drv-vmmouse.x86_64 0:12.4.0-2.1                        xorg-x11-drv-vmware.x86_64 0:10.13.0-2.1

Complete!

 The experience will be very similar -- if a bit more verbose.


That, my friends, may be it.  It seems convoluted but, really, once you ship a metapackage out with a proper apt+yum config, and your Linux VMs upgrade themselves overnight (well, maybe the test ones get the new metaRPMs first) then you'll be so much more pleased at the ability to do all that automatically.  Add in Spacewalk and/or cobbler and stir for best results.