Upgrading Cobbler from 2.0 to 2.2 -- and overcoming WSGI woes

If you've used Cobbler for a while, you'll want to upgrade it.  Naturally!

There's a problem with the upgrade, though:  the 2.0 version seems to use mod_python, and the new one uses mod_wsgi.  No problem, right?  So you install mod_wsgi as part of the process:

# service httpd start
Starting httpd: Syntax error on line 10 of /etc/httpd/conf.d/cobbler.conf:
Invalid command 'WSGIScriptAliasMatch', perhaps misspelled or defined by a
module not included in the server configuration
                                                           [FAILED]

So your server won't start.  Yay!

What's really going on is

1. mod_python and mod_wsgi don't play well together
2. mod_wsgi is impotent on install and needs activation
3. mod_python is still the go-to for rendering the configs, which now use syntax it can't handle

The remedy is simple:

1. remove mod_python.  It can't be used, so let's get it out to avoid dep- and other issues.
   
   

   rpm -e mod_python

2. create a mod_wsgi config
   
   

   cat > /etc/httpd/conf.d/05-load-wsgi.conf
   LoadModule wsgi_module modules/mod_wsgi.so
   

3. restart httpd
   
   

   service httpd restart

And that's it:

Stopping httpd:                                            [FAILED]
Starting httpd:                                            [  OK  ]

And you're back up and running.

Is it disappointing that it doesn't Just Work?  For sure.  Could you figure it out if you were a mod_python user or an expert, and knew the hell WSGi was?  Maybe.  But I'm not, and I think that as an app user it's not really on me to be an expert.  You may argue how proficient one needs to be to use any device, but I'm thinking it's not ready for prime-time yet.  Boo!

ReCollecting the FileProvides

I use APT for RPMs, as it's the most versatile tool out there.  Some folks don't seem to understand that APT was ported for RPMs about a decade ago, and those using Yum for their RPM management are encouraged to do so, lest their horizons grow too broad.  The truly genius folks at Conectiva used APT to great benefit, and allowed massive flexibility;  upgrading in-place over a major release was a possible, easy and well-tested procedure, as I recall.  But this isn't for the Apt-RPM genius so recently localized at Conectiva and now lost to the rest of the world.

Lately there's been another change to the repository format and layout as used by YUM;  this is nothing new, as many open-source de-facto standards are developed and grow in the same kind of seeming vacuum, and the gaijin must adapt.  If you're suffering as I was, your apt-get update invocations will look like this:

E: Error occured while processing sec (CollectFileProvides)
E: Problem with MergeFileProvides /var/lib/apt/lists/archive_cobbler_repo%5fmirror_epel5-i386_repodata_277d21a2341fe766d0daff22b2846905517bcd71-primary.sqlite

I'm using Cobbler to mirror many, many repos locally; almost anywhere I have more than one machine.  The update to the createrepo invocation will either come from /etc/cobbler/settings or the createrepo flags in a particular repository item.

Here's my change:

--- /etc/cobbler/settings~      2011-09-23 07:03:38.000000000 -0700
+++ /etc/cobbler/settings       2011-09-26 14:24:32.000000000 -0700
@@ -58,7 +58,7 @@
 # enables working with Fedora repos from F11/F12 from EL-4 or
 # EL-5 without python-hashlib installed (which is not available
 # on EL-4)
-createrepo_flags: "-c cache -s sha -C --update"
+createrepo_flags: "-q -c cache -s sha --update -d"

 # if no kickstart is specified to profile add, use this template
 default_kickstart: /var/lib/cobbler/kickstarts/default.ks

You will need to adjust to suit, as the patch will not apply cleanly unless you've been mucking about there already, but take the patch and make yours match the result.  Restart cobbler when you're done and your repos will be updating madly as expected.

As always, sorry if the Blogspot editor butchers my blockquotes.

Cobbler 2.0.11 Hack to Limit Rsync Bandwidth

I was having a problem with one of my cobbler app installs, where the rsync run would overload the already-loaded network signal. This is a pipe where we're already doing a lot of management, because the physical premises are not conducive to an upgrade (and we're not doing QoS yet; I know, I know).

Lanton Vhengani was also having the same problem back in 2008, and although Mr DeHaan was considering adding a switch to the rsync invocation from within the cobbler app, it's not yet appeared.  Lanton found the section where the rsync invocation is literally called out, and provided a great patch hint for the source.  It still works under 2.0.11 .

diff -uBb /usr/lib/python2.4/site-packages/cobbler/action_reposync.py\~ /usr/lib/python2.4/site-packages/cobbler/action_reposync.py
--- /usr/lib/python2.4/site-packages/cobbler/action_reposync.py~        2011-04-20 08:40:48.000000000 -0400
+++ /usr/lib/python2.4/site-packages/cobbler/action_reposync.py 2011-07-16 22:00:22.000000000 -0400
@@ -220,6 +220,7 @@
         if not repo.mirror.endswith("/"):
             repo.mirror = "%s/" % repo.mirror

+        spacer = " --bwlimit=50"
         # FIXME: wrapper for subprocess that logs to logger
         cmd = "rsync -rltDv %s --delete --exclude-from=/etc/cobbler/rsync.exclude %s %s" % (spacer, repo.mirror, dest_path)
         rc = utils.subprocess_call(self.logger, cmd)

restart cobblerd before testing, and there you go:  Bandwidth isn't pinned and the boss is happier.  That's a pretty small setting there, but I'll open it up after Centos6 comes down completely -- the periodic kicks the pipe will take when 1-2 RPMs are updated will be so small I can push it up a bit higher.

NoStorage and Kickstart - How to Specify Multiple HBA Modules

When kickstarting, you have the option of using 'nostorage' on the PXE command line to prevent storage HBA drivers from loading -- you can do the same to NICs, but it's not as interesting, not as common and the command line is dumber.

default linux
prompt 0
timeout 1
label linux
     kernel /images/centos55-x86_64/vmlinuz
     ipappend 2
     append initrd=/images/centos55-x86_64/initrd.img ksdevice=eth0 lang= kssendmac nostorage text ks=http://archive/cblr/svc/op/ks/system/Bish-PXETest

See that?  NoStorage.  Okay.

So what if you want to use the same kickstart for different machines?  For different HBAs ?  Normally you're screwed.  This won't work, either:

device scsi ahci
device scsi mptspi
device scsi cciss

Specifying the HBA drivers on multiple lines should work, but it's not that simple -- doing so makes it choose the first one and ignore every other invocation of the device line.  This does work, though:

device scsi ahci:mptspi:cciss

See the colons? There you go.  It allows/forces you to choose the order, so plan carefully.

Maintaining Repos in Kickstarted Machines After Install

After you've installed a machine, its install-time repository config in /etc/yum.repos.d is pretty much set.

Bah, I say! Bah! Just keep it updated.

Kickstart (cobbler):

#set yumconfcronfilename = "/etc/cron.daily/50-yum-config-stanza"
cat << EOECYCS > $yumconfcronfilename
#!/bin/sh
$yum_config_stanza

sed -ne '
        /^baseurl=/{
                s/baseurl=/repomd /
                s://:__:
                s:/: :
                s:__://:
                p
        }
        ' /etc/yum.repos.d/cobbler-config.repo \
          > /etc/apt/sources.list.d/cobbler-config.list
EOECYCS
chmod a+x $yumconfcronfilename

If you're not running cobbler, set it into place by hand:

cat << EOECYCS > /etc/cron.daily/50-yum-config-stanza
#!/bin/sh
wget "http://archive/cblr/svc/op/yum/profile/centos5-i386-minimal" --output-document=/etc/yum.repos.d/cobbler-config.repo

sed -ne '
 /^baseurl=/{
  s/baseurl=/repomd /
  s://:__:
  s:/: :
  s:__://:
  p
 }
 ' /etc/yum.repos.d/cobbler-config.repo \
   > /etc/apt/sources.list.d/cobbler-config.list
EOECYCS

chmod a+x /etc/cron.daily/50-yum-config-stanza

That's dereferenced for you. The actual profile's going to be way off, though, so don't use that one verbatim. Find your own:

awk -F/ '/^url/{print $NF}' anaconda-ks.cfg

As usual, watch carefully for the way in which the 'new', 'better' blogspot editor makes an artistic puree of the quoted stuff;  grain of salt, kids.

False Start on Cobbler Repos -> Apt RepoMD statements

Man, I know so little python.  And while I hate it, I see that it's a deficiency I need to fix.  That sucks, but I may just find it useful eventually.

So I was poking around, and started to extend cobbler to feed an apt sources.list files the same as it does for a yum repo file when a given profile or system has supplemental repos attached. 

But all I did was start.  The curve's a bit high since I know so little, and I have so many other things I have to do -- like, for money.

Here's a tiny patch so far.  I'll find the rest eventually:

--- kickgen.py  2010/09/29 03:40:45     1.1
+++ kickgen.py  2010/09/29 07:06:23
@@ -175,10 +175,15 @@
         blended = utils.blender(self.api, False, obj)
         if is_profile:
            url = "http://%s/cblr/svc/op/yum/profile/%s" % (blended["http_server"], obj.name)
+           lru = "http://%s/cblr/svc/op/apt/profile/%s" % (blended["http_server"], obj.name)
         else:
            url = "http://%s/cblr/svc/op/yum/system/%s" % (blended["http_server"], obj.name)
+           lru = "http://%s/cblr/svc/op/apt/system/%s" % (blended["http_server"], obj.name)

-        return "wget \"%s\" --output-document=/etc/yum.repos.d/cobbler-config.repo\n" % (url)
+        ulines = "wget \"%s\" --output-document=/etc/yum.repos.d/cobbler-config.repo\n" % (url)
+#        ulines += "wget \"%s\" --output-document=/etc/apt/sources.list.d/cobbler-config.list\n" % (lru)
+
+        return ulines

     def generate_kickstart_for_system(self, sys_name):

Yeah, that's all I've got.  Lose the comment on the ulines+= line, and then find out the part where /op/apt/system/ actually rolls the template snippet for the apt sources.list file.  I see it here and there but can't nail down the bit in the code.  Frustrating, almost as much as running out of frivolity time.


Learning is fun, when I have time.

Kickstarting ESX VMs and Physical Hosts -- Knowing Which is Which

UPDATE: This method also does not work. Sorry.

Cobbler and kickstarting is my new cool toy.  I tinker with it FAR too much. I had a problem, though, that I need to install the vmware tools only on the VMs, and install smartmontools only on the physical hosts.  What's a guy to do?

After some digging, coding, hacking, testing, cursing, I finally discovered a decent switch I can use to identify a box by its mac.

Then I lost that code.

So I found another method.  This one's ugly as sin, but it may actually work.  Check this nasty-ass kung-fu:

#if ":".join($interfaces.eth0.mac_address.split(':')[0:3]) in "00:50:56 00:0C:29"

[code]

#end if

Yeah.  That's one ugly baby.  If Blogspot again truncates that line all to hell, remember the #if statement is all on the first line of 3.

Now to see how well it works.

Auto-Updating Distro profile in Cobbler

As part of a very short, unsuccessful Fedora 13 test, I found it advantageous to set up an auto-updating distro.  Okay, given the release behaviour of Fedora vs, say, RHEL, an auto-updating distro profile for a distro which doesn't actually update is kinda pointless.  Work with me here, though.

How to do this in Cobbler?  Create a repo, which we can update automatically, and link the distro to that:

cobbler repo add --name fedora13-os-x86_64 --arch x86_64 \
  --mirror rsync://mirrors.kernel.org/fedora/releases/13/Fedora/x86_64/os

cobbler import --name fedora13 --arch x86_64 \
  --path /var/www/cobbler/repo_mirror/fedora13-os-x86_64 \
  --available-as http://10.10.4.1/cblr/repo_mirror/fedora13-os-x86_64 \
  --breed redhat --kickstart /etc/cobbler/centos-X.ks

Too easy.  Blah blah, test often, blah blah, change the IP to suit, etc.