They’re also the source of plenty of flail. What do I mean?

• They are neither code nor data, so often get overlooked, or shonkily installed, by application deployment tools
• They run with a minimal environment that can catch out the unwary: scripts that work in interactive shell sometimes don’t from cron
• The default behaviour of mailing output to the cronjob owner generates large amounts of mail that gets ignored, filtered or bounced
• Jobs can fail silently and no-one notices until, say, you need to restore that backup that hasn’t run for last six months
• Jobs that helpfully append their output to a log commonly don’t rotate that log
• It’s easy to have jobs overlapping if they get stuck or take longer than expected to complete. This is a splendid way of wedging a machine.

The mail aspect is a particular peeve. In some jobs my mailbox has enjoyed several thousand cron generated mails a day, and there’s no way I’m able to accurately look at each one and react to it. Mostly they contain expected output from successful job execution, so they’re easy to skip. But I don’t trust my eyes to get that right all the time.

One approach to this is to arrange for jobs to only send mail on error. This is an improvement, but can lead into thinking that a job is happily succeeding when in fact it’s either not running or the only-on-error logic is bust. Since cron jobs often cover essential system tasks like backing up, syncing data around and reporting it’s vital that they don’t fail silently.

I’ve worked somewhere that tackled this by collating cron-generated mails from diverse systems into a system mailbox and pattern matching them for failure signs. This seems slightly dubious — it’s fragile and labour intensive — but at least the system also flagged if expected jobs failed to arrive and got our inboxes tamed.

To tackle these problems I find myself writing wrappers for cronjobs. I’ve written several variants to meet different situation’s needs. Unhelpfully I call them all cronwrap. These wrappers sets out to

• Engage the amazingly useful lockrun utility to guard against multiple execution of stuck crons
• Place cron output into timestamped logs that can be both aged out and made available to interested parties
• Hook into local monitoring systems:
  1. On execution, update a run counter (SNMP data or some simple text file)
  2. On failure, send a SNMP trap or leave some bait for Nagios. Also, update a fail counter
  3. If lockrun has prevented a job running owing to overlap, send a SNMP trap or similarly bait Nagios
• If required, send output by mail somewhere (sometimes this is necessary, even with the concerns listed above)

So, nothing surprising there. Using such wrappers helps keep cron jobs tamed and reliable, and it’s monitoring them near to where the action occurs, rather than mediating via SMTP.

This is hardly invention either, there’s plenty of prior art with different nuances in behaviour to meet the needs of different environments. Perhaps I’ll merge the variants of my efforts and publish too.

What’s curious is that this functionality isn’t available inside the cron daemon¹ itself. It is perfectly placed to catch exit status, divert output and know if a job has overrun; and would remove the need for all this additional monkeying to make jobs reliable and well behaved. If my C wasn’t just read-only I’d have a crack at it!

There, I’ve finally condensed all my cron rant into one sustained piece.

1. To be clear, I’m talking about the BSD cron written by Paul Vixie. None of the variants I’ve seen address these concerns either. I’d love to know if there’s any I’ve missed.

The reference documentation has examples in Perl and PHP which got me so far, but I’m most comfortable in Ruby now, and was happy to find ¹ this pointer to using the soap4r library.

Chief bonus here is the wsdl2ruby.rb tool that’ll transform the WSDL data into Ruby objects with heirarchy, attribute accessors and everything else to make operating the API really comfortable. If your WSDL is a moving target during development it’ll even do this on the fly.

This meant getting the scripting done to configure the ZXTMs was pretty straightforward, without any faffing with the underlying access mech. Refreshing!

1. I was going to paste example code, but this’ll do

Update: Turns out this build is missing isol mode serial-on-LAN for IPMI 1.5 hosts¹, and building the ipmitool distribution wasn’t so hard after all. Packaging systems have made me lazy!

1. Dell R200 uses IPMI 1.5, Dell R410 uses 2.0. Both do SOL and remote power management well enough

As a same-same-but-different alternative to

• pssh
• That xterm multiplexer I can never remember the name of
• screen(1) and Terminal.app’s send-to-many option
• Quick and dirty shell scripts iterating over a host list
• Capistrano’s shell (even on hosts that don’t run any other Ruby!)

… it’s proving pretty solid.

Example:

[admin@manage3 tmp]$ pattern.py foo[1-8] | vxargs -y -o ~/tmp ssh {} varnishadm -Tlocalhost:6082 ping


Terminal clears, and shows progress:

8/8:ssh -l root foo8 varnishadm -Tlocalhost:6082 ping
Done
Done
Done
Done
Done
( 10s) 6: foo7
Done

and exits

exit code 0: 7 job(s)
exit code 1: 1 job(s)
total number of jobs: 8

Now you can inspect the output from each host, exit status and an overall failure list

[admin@manage3 tmp]$ cat ~/tmp/abnormal_list
foo5
[admin@manage3 tmp]$ cat ~/tmp/foo5.err
Warning: Permanently added 'foo5,10.221.11.0' (RSA) to the list of known hosts.
connect(): Connection refused
An error occured in receiving status.


Via Trivium

Useful for migrating sites from one host to another before or during DNS propagation.

That said, I’m using Varnish and HAProxy an increasing amount for such plumbing.

1. Yep, still running 1.3 around the place

• Between firewall pair, LAN

  [ 3] 0.0-10.0 sec 1.00 GBytes 860 Mbits/sec
[ 3] 0.0-10.0 sec 1.00 GBytes 860 Mbits/sec
[ 3] 0.0-10.0 sec 1017 MBytes 853 Mbits/sec

• Firewall to firewall between DCs, outside VPN, no PF

  [ 3] 0.0-10.0 sec 1.02 GBytes 873 Mbits/sec
[ 3] 0.0-10.0 sec 992 MBytes 832 Mbits/sec
[ 3] 0.0-10.0 sec 986 MBytes 827 Mbits/sec

• Firewall to remote internal host, outside VPN, through PF NAT (rdr)

  [ 3] 0.0-10.0 sec 260 MBytes 218 Mbits/sec
[ 3] 0.0-10.0 sec 202 MBytes 170 Mbits/sec
[ 3] 0.0-12.3 sec 333 MBytes 228 Mbits/sec

• Internal host to internal host, over IPsec VPN (ESP), through PF
  

  [ 3] 0.0-10.1 sec 43.9 MBytes 36.4 Mbits/sec
[ 3] 0.0-10.1 sec 26.2 MBytes 21.8 Mbits/sec
[ 3] 0.0-11.3 sec 28.0 MBytes 20.8 Mbits/sec

• Internal host to internal host, over OpenVPN, through PF
  

  [ 3] 0.0-10.0 sec 161 MBytes 134 Mbits/sec
[ 3] 0.0-10.0 sec 144 MBytes 121 Mbits/sec
[ 3] 0.0-10.0 sec 145 MBytes 121 Mbits/sec

Care was taken to use optimal ciphers, appropriate MTU / MSS and the TCP stack was tuned throughout.

• IPsec really hurts without hardware acceleration
• There’s a surprisingly large hit for just NAT
• Neither VPN technologies can benefit from the multiple cores available to them
• OpenVPN’s speed is appealing, but it lacks the smooth route to high availability of CARP + pfsync + sasync of IPsec on OpenBSD

• S3Fox
• Elasticfox

There’s bound to be something that draws together all the foregin keys – AMI, volume, instance, reservation etc – but I’ve yet to find that. Been putting off writing it for a while…

HPaq’s iLO is a strange beast, its functionality seems split between the SSH CLI and the web interface. Depending on your particular mission, some goals are only achievable using the whole ActiveX / Java remote console in IE. Sigh.

To get this flying from virtualised VMware, here’s the full SSH mutter to forward the ports via an access host. I’m always forgetting it, thus this post.

sudo ssh -g -L 80:${IP}:80 -L 3389:${IP}:3389 -L 443:${IP}:443 -L 17988:${IP}:17988 -L 17990:${IP}:17990 -L 23:${IP}:23 access005.example.com

IP is the iLO’s address. -g allows the port forwards to be reachable via other LAN hosts (eg: the VMware guest).  The sudo sorts the forwards for those super-privileged < 1024 ports.

Yes, that really is port 23 – telnet.

ssh -Dlocalhost:1080 -C host.example.com

You can now aim your browser’s SOCKS settting at localhost:1080. Bonus points for using a locally-hosted PAC file to determine which traffic is routed via the proxy:

function FindProxyForURL(url, host) {
  if (shExpMatch(host, "*.example.com"))
    return "SOCKS 127.0.0.1:1080"
  else
    return "DIRECT";
}

For getting sight of servers behind a firewall without remote proxies (hello Privoxy) and continual browser fiddling this is ideal. Thanks Murb!