Upgrading Cobbler from 2.0 to 2.2 -- and overcoming WSGI woes

If you've used Cobbler for a while, you'll want to upgrade it.  Naturally!

There's a problem with the upgrade, though:  the 2.0 version seems to use mod_python, and the new one uses mod_wsgi.  No problem, right?  So you install mod_wsgi as part of the process:

# service httpd start
Starting httpd: Syntax error on line 10 of /etc/httpd/conf.d/cobbler.conf:
Invalid command 'WSGIScriptAliasMatch', perhaps misspelled or defined by a
module not included in the server configuration
                                                           [FAILED]

So your server won't start.  Yay!

What's really going on is

1. mod_python and mod_wsgi don't play well together
2. mod_wsgi is impotent on install and needs activation
3. mod_python is still the go-to for rendering the configs, which now use syntax it can't handle

The remedy is simple:

1. remove mod_python.  It can't be used, so let's get it out to avoid dep- and other issues.
   
   

   rpm -e mod_python

2. create a mod_wsgi config
   
   

   cat > /etc/httpd/conf.d/05-load-wsgi.conf
   LoadModule wsgi_module modules/mod_wsgi.so
   

3. restart httpd
   
   

   service httpd restart

And that's it:

Stopping httpd:                                            [FAILED]
Starting httpd:                                            [  OK  ]

And you're back up and running.

Is it disappointing that it doesn't Just Work?  For sure.  Could you figure it out if you were a mod_python user or an expert, and knew the hell WSGi was?  Maybe.  But I'm not, and I think that as an app user it's not really on me to be an expert.  You may argue how proficient one needs to be to use any device, but I'm thinking it's not ready for prime-time yet.  Boo!

ReCollecting the FileProvides

I use APT for RPMs, as it's the most versatile tool out there.  Some folks don't seem to understand that APT was ported for RPMs about a decade ago, and those using Yum for their RPM management are encouraged to do so, lest their horizons grow too broad.  The truly genius folks at Conectiva used APT to great benefit, and allowed massive flexibility;  upgrading in-place over a major release was a possible, easy and well-tested procedure, as I recall.  But this isn't for the Apt-RPM genius so recently localized at Conectiva and now lost to the rest of the world.

Lately there's been another change to the repository format and layout as used by YUM;  this is nothing new, as many open-source de-facto standards are developed and grow in the same kind of seeming vacuum, and the gaijin must adapt.  If you're suffering as I was, your apt-get update invocations will look like this:

E: Error occured while processing sec (CollectFileProvides)
E: Problem with MergeFileProvides /var/lib/apt/lists/archive_cobbler_repo%5fmirror_epel5-i386_repodata_277d21a2341fe766d0daff22b2846905517bcd71-primary.sqlite

I'm using Cobbler to mirror many, many repos locally; almost anywhere I have more than one machine.  The update to the createrepo invocation will either come from /etc/cobbler/settings or the createrepo flags in a particular repository item.

Here's my change:

--- /etc/cobbler/settings~      2011-09-23 07:03:38.000000000 -0700
+++ /etc/cobbler/settings       2011-09-26 14:24:32.000000000 -0700
@@ -58,7 +58,7 @@
 # enables working with Fedora repos from F11/F12 from EL-4 or
 # EL-5 without python-hashlib installed (which is not available
 # on EL-4)
-createrepo_flags: "-c cache -s sha -C --update"
+createrepo_flags: "-q -c cache -s sha --update -d"

 # if no kickstart is specified to profile add, use this template
 default_kickstart: /var/lib/cobbler/kickstarts/default.ks

You will need to adjust to suit, as the patch will not apply cleanly unless you've been mucking about there already, but take the patch and make yours match the result.  Restart cobbler when you're done and your repos will be updating madly as expected.

As always, sorry if the Blogspot editor butchers my blockquotes.

Maintaining Repos in Kickstarted Machines After Install

After you've installed a machine, its install-time repository config in /etc/yum.repos.d is pretty much set.

Bah, I say! Bah! Just keep it updated.

Kickstart (cobbler):

#set yumconfcronfilename = "/etc/cron.daily/50-yum-config-stanza"
cat << EOECYCS > $yumconfcronfilename
#!/bin/sh
$yum_config_stanza

sed -ne '
        /^baseurl=/{
                s/baseurl=/repomd /
                s://:__:
                s:/: :
                s:__://:
                p
        }
        ' /etc/yum.repos.d/cobbler-config.repo \
          > /etc/apt/sources.list.d/cobbler-config.list
EOECYCS
chmod a+x $yumconfcronfilename

If you're not running cobbler, set it into place by hand:

cat << EOECYCS > /etc/cron.daily/50-yum-config-stanza
#!/bin/sh
wget "http://archive/cblr/svc/op/yum/profile/centos5-i386-minimal" --output-document=/etc/yum.repos.d/cobbler-config.repo

sed -ne '
 /^baseurl=/{
  s/baseurl=/repomd /
  s://:__:
  s:/: :
  s:__://:
  p
 }
 ' /etc/yum.repos.d/cobbler-config.repo \
   > /etc/apt/sources.list.d/cobbler-config.list
EOECYCS

chmod a+x /etc/cron.daily/50-yum-config-stanza

That's dereferenced for you. The actual profile's going to be way off, though, so don't use that one verbatim. Find your own:

awk -F/ '/^url/{print $NF}' anaconda-ks.cfg

As usual, watch carefully for the way in which the 'new', 'better' blogspot editor makes an artistic puree of the quoted stuff;  grain of salt, kids.

Installing VMware Tools in Linux after ESX 4.1

Tried to install VMware-Tools lately?  Not sure how it goes on Windows, but on Linux it's a bit more challenging than it used to be.  Used to be there was an RPM on the ISO that popped into the Linux machine's vCDROM drive when the option to install VMware Tools was chosen.

No more.  Like ESXi4.0 before it, ESX4.1 seems to ship with an ISO containing, for Linux, a little tarball with the stuff.

Those who've worked for a distro or packaged anything will know that shipping and installing bare tarballs is really not the best way to go.  Sure, it's fine for the cowboys who maintain 1-2 machines at the home or home-office, but for anyone who has to maintain more than one, has to roll back or verify a package level, or who just wants a clean system, a good package is the only way to go.

Note that I said good package.  If you hear anyone mention "dependency hell" or anything like that, they weren't paying attention.  Stop using bad packages!  Stop mixing repositories.  The pathetic blunt object that pass for packages is the topic of many blogwails past and many blogwails to come.

The great news?  VMware seem to have a few packages for us.  In fact, they come in 4 flavours of RedHat and some even for the you-bunt-you crowd.  Seems all you have to do is mark up the apt config ...

cat <<EOF> /etc/apt/sources.list.d/esx.list

repomd  http://packages.vmware.com tools/esx/4.1/rhel$(VERSION)/$(ARCH)
EOF
apt-get update

... or yum config if you don't build your own packages and know where apt excels ...

cat <<EOF> /etc/yum.repos.d/esx.repo

[esx]
name=VMware-tools for RHEL\$releasever - \$basearch

baseurl=http://packages.vmware.com/tools/esx/4.1/rhel\$releasever/\$basearch
enabled=1
gpgcheck=1
gpgkey=http://packages.vmware.com/tools/VMWARE-PACKAGING-GPG-KEY.pub

EOF

yum update

and after refreshing, just install the package:

apt-get install vmware-tools
Reading Package Lists... Done
Building Dependency Tree... Done
The following extra packages will be installed:
   vmware-open-vm-tools (8.3.2-257589.el5)
   vmware-open-vm-tools-common (8.3.2-257589.el5)
   vmware-open-vm-tools-kmod (8.3.2-257589.el5)
   vmware-open-vm-tools-nox (8.3.2-257589.el5)
   vmware-open-vm-tools-xorg-drv-display (10.16.7.0-0.257589.el5)
   vmware-open-vm-tools-xorg-drv-mouse (12.6.4.0-0.257589.el5)
   vmware-open-vm-tools-xorg-utilities (8.3.2-257589.el5)
   vmware-tools-common (8.3.2-257589.el5)
   vmware-tools-nox (8.3.2-257589.el5)
The following packages will be REPLACED:
   xorg-x11-drv-vmmouse (12.4.0-2.1)
   (by (10.13.0-2.1)
   vmware-open-vm-tools-xorg-drv-mouse) ()
    (12.4.0-2.1)
   xorg-x11-drv-vmware (10.13.0-2.1)
   (by ()
   vmware-open-vm-tools-xorg-drv-display) (12.4.0-2.1)
The following NEW packages will be installed:
   vmware-open-vm-tools (8.3.2-257589.el5)
   vmware-open-vm-tools-common (8.3.2-257589.el5)
   vmware-open-vm-tools-kmod (8.3.2-257589.el5)
   vmware-open-vm-tools-nox (8.3.2-257589.el5)
   vmware-open-vm-tools-xorg-drv-display (10.16.7.0-0.257589.el5)
   vmware-open-vm-tools-xorg-drv-mouse (12.6.4.0-0.257589.el5)
   vmware-open-vm-tools-xorg-utilities (8.3.2-257589.el5)
   vmware-tools (8.3.2-257589.el5)
   vmware-tools-common (8.3.2-257589.el5)
   vmware-tools-nox (8.3.2-257589.el5)
0 upgraded, 10 newly installed, 2 replaced, 0 removed and 0 not upgraded.
Need to get 44.8kB/13.7MB of archives.
After unpacking 41.1MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-kmod 8.3.2-257589.el5 [524kB]
Get:2 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-common 8.3.2-257589.el5 [5283kB]
Get:3 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-nox 8.3.2-257589.el5 [2631B]
Get:4 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-xorg-drv-mouse 12.6.4.0-0.257589.el5 [17.5kB]
Get:5 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-xorg-drv-display 10.16.7.0-0.257589.el5 [33.4kB]
Get:6 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools-xorg-utilities 8.3.2-257589.el5 [7828kB]
Get:7 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-open-vm-tools 8.3.2-257589.el5 [2811B]
Get:8 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-tools-common 8.3.2-257589.el5 [39.4kB]
Get:9 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-tools-nox 8.3.2-257589.el5 [2635B]
Get:10 http://packages.vmware.com tools/esx/4.1/rhel5/x86_64/ vmware-tools 8.3.2-257589.el5 [2763B]
Fetched 44.8kB in 0s (46.5kB/s)
Committing changes...
Preparing                                ############################## [100%]
Updating / installing
  vmware-open-vm-tools-xorg-drv-display- ############################## [100%]
  vmware-open-vm-tools-xorg-drv-mouse-12 ############################## [100%]
  vmware-open-vm-tools-kmod-8.3.2-257589 ############################## [100%]
  vmware-open-vm-tools-common-8.3.2-2575 ############################## [100%]
  vmware-open-vm-tools-nox-8.3.2-257589. ############################## [100%]
  vmware-open-vm-tools-xorg-utilities-8. ############################## [100%]
  vmware-open-vm-tools-8.3.2-257589.el5. ############################## [100%]
  vmware-tools-common-8.3.2-257589.el5.x ############################## [100%]
  vmware-tools-nox-8.3.2-257589.el5.x86_ ############################## [100%]
  vmware-tools-8.3.2-257589.el5.x86_64   ############################## [100%]
Cleaning up / removing
  xorg-x11-drv-vmmouse-12.4.0-2.1.x86_64 ############################## [100%]
  xorg-x11-drv-vmware-10.13.0-2.1.x86_64 ############################## [100%]
Done.

While I avoid Yum, it should look like this:

[root@0-50-56-bc-0-7 ~]# yum install vmware-tools
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * addons: mirror.raystedman.net
 * base: mirror.skiplink.com
 * extras: mirror.skiplink.com
 * updates: styx.biochem.wfubmc.edu
Setting up Install Process
Resolving Dependencies
:
[blah blah]
:
Dependencies Resolved

==================================================================================================================================
 Package                                           Arch               Version                               Repository       Size
==================================================================================================================================
Installing:
 vmware-open-vm-tools-xorg-drv-display             x86_64             10.16.7.0-0.257589.el5                esx              33 k
     replacing  xorg-x11-drv-vmware.x86_64 10.13.0-2.1

 vmware-open-vm-tools-xorg-drv-mouse               x86_64             12.6.4.0-0.257589.el5                 esx              17 k
     replacing  xorg-x11-drv-vmmouse.x86_64 12.4.0-2.1

 vmware-tools                                      x86_64             8.3.2-257589.el5                      esx             2.7 k
Installing for dependencies:
 vmware-open-vm-tools                              x86_64             8.3.2-257589.el5                      esx             2.7 k
 vmware-open-vm-tools-common                       x86_64             8.3.2-257589.el5                      esx             5.0 M
 vmware-open-vm-tools-kmod                         x86_64             8.3.2-257589.el5                      esx             512 k
 vmware-open-vm-tools-nox                          x86_64             8.3.2-257589.el5                      esx             2.6 k
 vmware-open-vm-tools-xorg-utilities               x86_64             8.3.2-257589.el5                      esx             7.5 M
 vmware-tools-common                               x86_64             8.3.2-257589.el5                      esx              39 k
 vmware-tools-nox                                  x86_64             8.3.2-257589.el5                      esx             2.6 k

Transaction Summary
==================================================================================================================================
Install      10 Package(s)
Upgrade       0 Package(s)

Total download size: 13 M
Is this ok [y/N]: y
Downloading Packages:
:
[ more blah blah ]
:
Running Transaction
  Installing     : vmware-open-vm-tools-xorg-drv-display                                                                     1/12
  Installing     : vmware-open-vm-tools-kmod                                                                                 2/12
  Installing     : vmware-open-vm-tools-common                                                                               3/12
  Installing     : vmware-open-vm-tools-nox                                                                                  4/12
  Installing     : vmware-tools-common                                                                                       5/12
  Installing     : vmware-tools-nox                                                                                          6/12
  Installing     : vmware-open-vm-tools-xorg-drv-mouse                                                                       7/12
  Installing     : vmware-open-vm-tools-xorg-utilities                                                                       8/12
  Installing     : vmware-open-vm-tools                                                                                      9/12
  Installing     : vmware-tools                                                                                             10/12
  Erasing        : xorg-x11-drv-vmmouse                                                                                     11/12
  Erasing        : xorg-x11-drv-vmware                                                                                      12/12

Installed:
  vmware-open-vm-tools-xorg-drv-display.x86_64 0:10.16.7.0-0.257589.el5
  vmware-open-vm-tools-xorg-drv-mouse.x86_64 0:12.6.4.0-0.257589.el5
  vmware-tools.x86_64 0:8.3.2-257589.el5

Dependency Installed:
  vmware-open-vm-tools.x86_64 0:8.3.2-257589.el5                      vmware-open-vm-tools-common.x86_64 0:8.3.2-257589.el5
  vmware-open-vm-tools-kmod.x86_64 0:8.3.2-257589.el5                 vmware-open-vm-tools-nox.x86_64 0:8.3.2-257589.el5
  vmware-open-vm-tools-xorg-utilities.x86_64 0:8.3.2-257589.el5       vmware-tools-common.x86_64 0:8.3.2-257589.el5
  vmware-tools-nox.x86_64 0:8.3.2-257589.el5

Replaced:
  xorg-x11-drv-vmmouse.x86_64 0:12.4.0-2.1                        xorg-x11-drv-vmware.x86_64 0:10.13.0-2.1

Complete!

 The experience will be very similar -- if a bit more verbose.


That, my friends, may be it.  It seems convoluted but, really, once you ship a metapackage out with a proper apt+yum config, and your Linux VMs upgrade themselves overnight (well, maybe the test ones get the new metaRPMs first) then you'll be so much more pleased at the ability to do all that automatically.  Add in Spacewalk and/or cobbler and stir for best results.