Check your Certs

Check your SSL certs.

Okay, not just for fun because you're a party animal. I needed to confirm the validity of a cert on a host and I had to look it up.  Here's the one-step command:

openssl x509 -text -noout -in <(openssl s_client -connect HOSTNAME:443 <<<"" 2>/dev/null \
| sed -ne '/BEGIN/,/END/p') | grep Not
            Not Before: Mar 28 11:16:36 2014 GMT
            Not After : Nov  1 19:32:08 2015 GMT

And that's the output.  My cert's still good.  Yay!

That's it.

Testing your Syslogd Remote config

Test, test, test; right?

So you've configured syslogd or, increasingly, rsyslogd .  How do you trivially test?

It's really simple.

1. make sure nc or ncat (netcat) are installed.  It's a common tool, but maybe you don't have it on this host.  It's with nmap on my EL6 host, but a standalone on EH5.  Because EL7 has to reinvent everything, nc is in mnap-ncat.  So here's your install:
   
   yum install {/usr,}/bin/{nc,ncat}
   
   Honestly, that's the easiest way.
2. Launch tshark on the log forwarder:
   
   # tshark -plni any udp and port 514
   
3. Now hit it.  Here's your nc invocation:
   
   # ncat -u loghost 514 <<< "test logger receipt"
   
4. And here's how it looks:

   # tshark -plni any udp and port 514
   Running as user "root" and group "root". This could be dangerous.
   Capturing on Pseudo-device that captures on all interfaces
     0.000000 192.168.112.6 -> 192.168.10.251 Syslog 64 test logger receipt\n
     0.000301 192.168.10.251 -> 192.168.16.133 Syslog 83 USER.NOTICE: Jun  3 22:25:07 test logger receipt
     0.000389 192.168.10.251 -> 192.168.10.250 Syslog 83 USER.NOTICE: Jun  3 22:25:07 test logger receipt
   ^C
   

Fun fun?

Hey, look!  Our logger box is happily forwarding stuff out, too.  Double-win!

Logfile Rotation, and Counting Cars

Recently, fads changed in logfile rotation -- what a hot topic!

In RHEL5, it seems, logfiles were rotated very simply:  1, 2, 3.  Maybe gzipped too:  1.gz, 2.gz, 3.gz . This is a very simple arrangement:

1. It's simpler to predict for humans:  3 follows 2 as often as 2.gz follows 1.gz .  
2. It's simpler to use as a rotational algorithm
3. This means it's simpler to use in scripting and automation.

But the new fad is worrisome:  Logs are rotated with date suffixes, as if we can't 'head -1' or 'tail -1', and maybe need the filename to reflect one of the dates on the logfiles;  the last day it was around, because 'tail -1' is so hard, perhaps?

Now logs end up like:

/var/log/messages-20150501
/var/log/messages-20150508
/var/log/messages-20150522

Hey, now -- what happened to the logfile from 20150515?  Well, it wasn't required, as logrotate will skip rotating a logfile which wasn't full enough and thus worthy of rotation. 

If we believe it takes slightly more effort to calculate a timestamp than to simply increment a counter, we have a process evolved to where it's harder, dumber, and useless.

I'll bet this started with someone who enjoys and uses systemd.

So here's a fix.  I used it only today, so it's been tested a whopping once, (aka twice as long as systemd), but it may actually work (hey, systemd works as an eyesore of a monolithic failbot very well, thanks, so you lay off; it's the kind of drooling, three-legged marmoset that any loving mother would be proud to hang on the fridge, albeit the back).

So you've fixed the config in logrotate so it'll name things like we have a clue:

echo nodateext > /etc/logrotate.d/aa-nodateext

(Please don't ask me why we're not hacking up /etc/logrotate.conf like someone too clueless about enterprise linux to know better. I may hurt you.)

Once you've corrected the behaviour, it's time to clean up the mess.  This script seeks out badly-named logfiles - most of them - and renames them properly.  That's it.

Yeah, it looks like crap.  It's quick, and I did it when I had some spare time today.  And now I intend to run it on a bajillion machines tomorrow.  I may even hook it into the %post script in my bish-logrotate-no_dateExt_you_Idiots RPM.  It could be the perfect quick-fix.

::::::::::::::
remediate-logfile-names.sh
::::::::::::::
#!/bin/sh
# -*- compile-command: "sh remediate-logfile-names.sh" -*-
# $Id: remediate-logfile-names.sh,v 1.2 2015/05/26 17:16:07 root Exp $

# locate the logrotate files we care about
find /etc/logrotate.d \
    -name \*~ -prune -o \
    -name \*.rpm\* -prune -o \
    -name syslo\* -prune -o \
    -type f \
    | sort \
    | \
    xargs -n1 sed -ne '
# prune the open-bracket on many logfile spec lines
/^\//{s/[{]//;p
} ' \
    | \
    while read fn trash; do
        case $fn in
            (/*)
                find $fn-201[0-9][0-9][0-9][0-9][0-9] -type f 2>/dev/null | \
                    grep -n "$fn" \
                    | awk -F: -v gz=${gz:0} '
{
  f=gensub(/-201.*$/,"",1,$2)"."$1
  print "mv "$2" "f
}
'
                ;;
            (compress)
                gz=1
                ;;
        esac
    done \
        | tee /dev/stderr \
        | sh

And that's about it.   Enjoy.  And if you want to know why you don't just hand-hack logrotate.conf, then toss me a comment.  

 - bish

Dear Emacs people: Please just quit shoveling us crap without an opt-IN.

I've spent weeks considering the proper punishment for people who shovel you new 'features' which turn themselves on automatically and ruin the flow of work.  Since I jumped on a RHEL6 box - wow; there's a whole series of things I can write about the wrongness that is RHEL6 - and fired up an actual editor, you'll guess I mean the new crap in emacs without having to remember the title.

(if you're a vi freak, stop reading now.  I outgrew that 1.5-dimensional rinky-dink keypress mess in 1993)

The problem isn't the new stuff, because I like some new stuff if it's a help.  The problem is that the work flow is holy, and we need time to assimilate changes to it without it being an irritant.  What's the irritant today?  Conf mode.  I'm sure it's great, and I'm sure it'll be awesome eventually, but its behaviour is different, and instead of adding a snippet to the startup text suggesting I try it for this given text file (and maybe offer a switch-once/switch-always command) the mode I'm using to quickly edit a file causes the cursor to jump around.

I don't want it to jump around without my leave.  I mean, as long as I understand the operator/terminal relationship anyway.  *I* control *it* by default.

Now to get the behaviour I need at work, I've cruelly hacked out the mode's startup, because the alternative of vapourizing the auto-mode-alist has an even bigger crater than excising the mode in a crude, sick fashion.  Someone tell me how to use the (remove) to remove all the conf-mode entries and/or show me where someone's hacked in the kind of feature banner above, and I'll love you forever and update this page.

Here's the diff:

--- /root/.emacs~       2011-10-30 05:00:45.000000000 -0700
+++ /root/.emacs        2013-09-15 14:28:40.000000000 -0700
@@ -372,3 +372,7 @@
 ;; fix fucking new emacs' addition to fucking horizontal fucking splitting.  fuck
 (setq split-height-threshold 0)
 (setq split-width-threshold nil)
+
+;; fix fucking new emacs' addiction to fucking conf-mode.  Double-fuck.
+(defalias 'conf-space-mode 'text-mode)
+(defalias 'conf-mode 'text-mode)

And seriously, emacs devs:  sorry to rant, but adding new features and fucking with the flow is like a despicable pop-up banner on the web you just can't close.  It could not be more annoying if you also enabled at the same time.

Cobbler logfiles growing out of control!

I discovered my cobbler logs were starting to take up a noticeable portion of my /var/log/ filesystem.  It appears to be all in the pseudo-rotated log file collection in /var/log/cobbler/tasks/ .  If there's a setting to control how much is retained, I haven't seen it yet.  And I didn't have time to go looking as I had some filesystem issues to get through.

Here's my temporary fix:

# cat > /etc/cron.daily/zz-cobbler-tmpwatch
#!/bin/sh
/usr/sbin/tmpwatch -umc -f 720 /var/log/cobbler/tasks/

And that was it.

Now I can breathe easier while I dig out a better fix.

Let me know if this worked for you, or whether you found a better fix in cobbler itself.

Get your QNap to Call Home with VTun

My QNap moves around, occasionally, and I like to keep the configuration on my backups consistent.  So, I have a crappy VTun link to keep it where I want it.  When a device comes up, it calls a host and opens a channel so I can get back to it easily.

(Ideally, I'd like it to use the central server like a lookup server, to make other connections, like skype does, but that's a story for another time and some other coder)

1. install Optware
   
2. ipkg update
   

   # ipkg update
   Downloading
   http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/unstable/Packages.gz
   Inflating
   http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/unstable/Packages.gz
   Updated list of available packages in /opt/lib/ipkg/lists/cs08q1armel
   Successfully terminated.
   
   

3. ipkg install vtun
   

   # ipkg install vtun
   Installing vtun (3.0.2-2) to root...
   Downloading
   http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/unstable/vtun_3.0.2-2_arm.ipk
   Installing lzo (2.03-1) to root...
   Downloading
   http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/unstable/lzo_2.03-1_arm.ipk
   Installing openssl (0.9.8v-2) to root...
   Downloading
   http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/unstable/openssl_0.9.8v-2_arm.ipk
   Installing zlib (1.2.5-1) to root...
   Downloading
   http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/unstable/zlib_1.2.5-1_arm.ipk
   Configuring lzo
   Configuring openssl
   Configuring vtun
   Configuring zlib
   Successfully terminated.
   
   

4. configure your VPN connection

So far, so good.  But how do you make it start on boot?  That's tricky, as the QNap is largely flushed with each reboot, and we want something that is resilient and persistent -- so it's got to use something like inittab (or, if you hate Unix, maybe upstart).

So you create what they call an autorun.sh , a script.  It runs on boot due to some trickery, and we use that to make our box a little more chatty.  This hack lets you run the script you want at boot, but the timing can be a little racey.

1. qpkg.conf
   

   cat << EOF >> /etc/config/qpkg.conf
   [autorun]
   Name = autorun
   Version = 0.1
   Author = neomilium
   Date = 2013-05-06
   Enable = TRUE
   Install_Path = /share/Public/../.qpkg/autorun
   Shell = /share/Public/../.qpkg/autorun/autorun.sh
   EOF
   
   

2. autorun.sh
   

   cat << EOF > /share/Public/../.qpkg/autorun/autorun.sh
   lsmod | grep -q ^tun || insmod /lib/modules/others/tun.ko
   cat << EOT >>/etc/inittab
   null::respawn:/opt/sbin/vtund -n -f /opt/etc/vtund.conf profile server 2>/dev/null
   EOT
   EOF
   (sleep 60 ; /bin/kill -1 1) &

3. tweak
   

   mkdir -p /share/Public/../.qpkg/autorun/
   chmod a+x /share/Public/../.qpkg/autorun/autorun.sh
   

And that's kinda it.

We use the funky filename because it seems to be immune to changes between one QNap and the next -- moreso, immune to differences in drive configuration.  I think.  Let me know if it's not.

Packaging protobuf-c for EL5

How boring a title that is!  Okay, we're RPMing up Protobuf-c - a c binding for the protobuf kit from Google - for EL5.  We need it to do something far more interesting later -- so these are the giants whose shoulders we stand upon.

I cheated;  let's be honest about that, and just how much I cheat to make my life easier.  It's a lot!  In this case, I stole the protobuf-c RPM as built in RHbz593559 for EL6, and which is supposed to be started for EL5 but hasn't dropped yet.  So we'll just finish that port.

1. Go get it:
   wget http://download.fedoraproject.org/pub/epel/6/SRPMS/protobuf-c-0.15-2.el6.src.rpm
2. Unpacking it is harder because of a strange change to the package format, where they seem to ignore compatibility.  So we need to do an rpm2cpio|cpio instead of a trivial rpm-i .  It's no big deal, but watch for it:
   

   rpm2cpio protobuf-c-0.15-2.el6.src.rpm | cpio -idumv

3. Because the EL6 one has problems, we need to patch around those, but they're trivial;  it's surprising that we see this in what should be a quality product.  I worry about just how many people have this kind of blinders on.
   

   @@ -10,6 +10,7 @@

    Source1:        http://protobuf-c.googlecode.com/svn/tags/%{version}/LICENSE
   
    BuildRequires:  protobuf-devel
   +BuildRoot:     %{_tmppath}/%{name}-root
   
    %description
    Protocol Buffers are a way of encoding structured data in an efficient yet@@ -41,9 +41,13 @@

    make check
   
    %install
   +[ "$RPM_BUILD_ROOT" != "/" ] && rm -rf $RPM_BUILD_ROOT
    make install DESTDIR=$RPM_BUILD_ROOT
    rm -f $RPM_BUILD_ROOT/%{_libdir}/libprotobuf-c.la
   
   +%clean
   +[ "$RPM_BUILD_ROOT" != "/" ] && rm -rf $RPM_BUILD_ROOT
   +
    %post -p /sbin/ldconfig
    %postun -p /sbin/ldconfig

4. now build:
   

   rpmbuild -ba protobuf-c.spec

And that's kinda it.  Can you see how simple a package-port this is?

Can you see how simple code-compatibility things we learned in first-year CompSci are being ignored here to our detriment?  This is so disappointing, and it wouldn't be a problem if I haven't had this very same conversation with someone in a position to fix it.