They’re also the source of plenty of flail. What do I mean?

• They are neither code nor data, so often get overlooked, or shonkily installed, by application deployment tools
• They run with a minimal environment that can catch out the unwary: scripts that work in interactive shell sometimes don’t from cron
• The default behaviour of mailing output to the cronjob owner generates large amounts of mail that gets ignored, filtered or bounced
• Jobs can fail silently and no-one notices until, say, you need to restore that backup that hasn’t run for last six months
• Jobs that helpfully append their output to a log commonly don’t rotate that log
• It’s easy to have jobs overlapping if they get stuck or take longer than expected to complete. This is a splendid way of wedging a machine.

The mail aspect is a particular peeve. In some jobs my mailbox has enjoyed several thousand cron generated mails a day, and there’s no way I’m able to accurately look at each one and react to it. Mostly they contain expected output from successful job execution, so they’re easy to skip. But I don’t trust my eyes to get that right all the time.

One approach to this is to arrange for jobs to only send mail on error. This is an improvement, but can lead into thinking that a job is happily succeeding when in fact it’s either not running or the only-on-error logic is bust. Since cron jobs often cover essential system tasks like backing up, syncing data around and reporting it’s vital that they don’t fail silently.

I’ve worked somewhere that tackled this by collating cron-generated mails from diverse systems into a system mailbox and pattern matching them for failure signs. This seems slightly dubious — it’s fragile and labour intensive — but at least the system also flagged if expected jobs failed to arrive and got our inboxes tamed.

To tackle these problems I find myself writing wrappers for cronjobs. I’ve written several variants to meet different situation’s needs. Unhelpfully I call them all cronwrap. These wrappers sets out to

• Engage the amazingly useful lockrun utility to guard against multiple execution of stuck crons
• Place cron output into timestamped logs that can be both aged out and made available to interested parties
• Hook into local monitoring systems:
  1. On execution, update a run counter (SNMP data or some simple text file)
  2. On failure, send a SNMP trap or leave some bait for Nagios. Also, update a fail counter
  3. If lockrun has prevented a job running owing to overlap, send a SNMP trap or similarly bait Nagios
• If required, send output by mail somewhere (sometimes this is necessary, even with the concerns listed above)

So, nothing surprising there. Using such wrappers helps keep cron jobs tamed and reliable, and it’s monitoring them near to where the action occurs, rather than mediating via SMTP.

This is hardly invention either, there’s plenty of prior art with different nuances in behaviour to meet the needs of different environments. Perhaps I’ll merge the variants of my efforts and publish too.

What’s curious is that this functionality isn’t available inside the cron daemon¹ itself. It is perfectly placed to catch exit status, divert output and know if a job has overrun; and would remove the need for all this additional monkeying to make jobs reliable and well behaved. If my C wasn’t just read-only I’d have a crack at it!

There, I’ve finally condensed all my cron rant into one sustained piece.

1. To be clear, I’m talking about the BSD cron written by Paul Vixie. None of the variants I’ve seen address these concerns either. I’d love to know if there’s any I’ve missed.

The reference documentation has examples in Perl and PHP which got me so far, but I’m most comfortable in Ruby now, and was happy to find ¹ this pointer to using the soap4r library.

Chief bonus here is the wsdl2ruby.rb tool that’ll transform the WSDL data into Ruby objects with heirarchy, attribute accessors and everything else to make operating the API really comfortable. If your WSDL is a moving target during development it’ll even do this on the fly.

This meant getting the scripting done to configure the ZXTMs was pretty straightforward, without any faffing with the underlying access mech. Refreshing!

1. I was going to paste example code, but this’ll do

Update: Turns out this build is missing isol mode serial-on-LAN for IPMI 1.5 hosts¹, and building the ipmitool distribution wasn’t so hard after all. Packaging systems have made me lazy!

1. Dell R200 uses IPMI 1.5, Dell R410 uses 2.0. Both do SOL and remote power management well enough

As a same-same-but-different alternative to

• pssh
• That xterm multiplexer I can never remember the name of
• screen(1) and Terminal.app’s send-to-many option
• Quick and dirty shell scripts iterating over a host list
• Capistrano’s shell (even on hosts that don’t run any other Ruby!)

… it’s proving pretty solid.

Example:

[admin@manage3 tmp]$ pattern.py foo[1-8] | vxargs -y -o ~/tmp ssh {} varnishadm -Tlocalhost:6082 ping


Terminal clears, and shows progress:

8/8:ssh -l root foo8 varnishadm -Tlocalhost:6082 ping
Done
Done
Done
Done
Done
( 10s) 6: foo7
Done

and exits

exit code 0: 7 job(s)
exit code 1: 1 job(s)
total number of jobs: 8

Now you can inspect the output from each host, exit status and an overall failure list

[admin@manage3 tmp]$ cat ~/tmp/abnormal_list
foo5
[admin@manage3 tmp]$ cat ~/tmp/foo5.err
Warning: Permanently added 'foo5,10.221.11.0' (RSA) to the list of known hosts.
connect(): Connection refused
An error occured in receiving status.


Via Trivium

The author of the piece linked above has scratched this itch with Maatkit, a suite of replication-focused MySQL tools which include some for checksumming tables.

This can run with standalone instances but most appealing is the way it runs across a replication chain: the statements to perform the table checksums are themselves introduced into the replication and store their results in a table alongside the master’s checksums. You can then compare and contrast on each slave: neat. All straightforward to roll into Nagios or other monitoring.

Particulary cool is that they will let you at the source of your network. You can’t then wander off and run it elsewhere, it sits atop of their core web framework that you can’t see. With the source (which is encouragingly well written PHP) you can do perform all manner of modifications to bend the template network to your will.

Ning provide their own authentication system, and there’s no API to hook in someone else’s which is a hassle if you’re trying to build a social network alongside another site: maintaining a login for each site is going to annoy the users and be a nightmare to manage. Nil points.

One solution is to build a single sign-on system around what Ning already provide, which is robust, tested and better presented than anything we could achieve! I’d sketched out such a system for a proposal years ago but never had the opportunity to build it. A current project provided the perfect excuse to try this out.

Layout

Operation
The numbers here match those in the diagram.

1. User visits www.example.com (which for us happens to be a Rails) site.
2. The PageController notices the browser supplied no cookie and must therefore log on to Ning before proceeding. The site returns a redirect to Ning’s authentication page.
3. The browser follows this redirect to Ning.
4. Ning authenticates the user. The login code is modified from the original Ning behaviour. Here, it issues a redirect to sso.example.com with some parameters, including the user’s Ning ID and a salted hash to prevent spoofing. In this redirect Ning sets a range of cookies, including one that identifies the user to Ning.
5. The browser follows this redirect to the SSO server (which happens to be a Merb site – I wanted to try it out!)
6. The SSO app checks the provided hash against its own idea of what it should be. Assuming they match it considers the Ning ID to be valid. Finally, the SSO app issues another redirect along with cookies for just .example.com. This cookie identifies the user to the external site
7. In passing, the SSO app keeps track of users it has seen, and if this is a new user it will make an API request to Ning to fetch that user’s profile data and create a matching user on the www.example.com site.
8. The browser follows this last redirect back to Ning. It could be back to www.example.com or Ning depending on the situation.
9. Ning knows who the user is by merit of its cookies.
10. As does our external site.

Notes

• Having the SSO app different from the www.example.com site is perhaps a bit baroque. It works because the SSO app issues a cookie for .example.com which the browser will offer to both sso.example.com and www.example.com. In favour of this approach is that the SSO app is simple, and thus less likely to fail during development iterations than the nascent www.example.com site. A failure in the SSO app is bad, becaise it locks people out of Ning too. That Rails and Merb can share session data (and a database) is cool.
• The SSO app’s fetching of Ning profile data allows us to maintain a local version of a user’s profile to avoid the need to fetch it from Ning every time we need. There’s a nuance of Ning API authentication that meant I had to write a custom widget (Ning site component) to handle that.

Specifics
I was about to paste bits of modified Ning code, but I’ll need to check if I can under the various blurbs you agree too when signing up! Anyhow, the information above should help answer some of the requests for details from the Ning developers forum.

• There’s no invention here, I believe this setup is very common.
• High availability isn’t the same thing as load balanced. There is nothing here to intelligently shared load across the frontend servers, and one backend server is essentially idle all the time.
• This cluster is built with a bunch of open-source software on non-fancy kit. As such it doesn’t have the enormous capacity of clusters built upon commercial shared-storage products, SAN kit, layer 7 web switches etc. Its ambition is to run a few busy Rails sites well whilst coping with hardware failure gracefully.

Layout

Operation

• Web traffic is spread across the managed frontend interfaces by multiple A records in the DNS.
• Wackamole uses a Spread messaging network to ensure these multiple A record IPs are always present across the frontend. It achieves this by managing the hosts’ interfaces when it detects hosts joining or leaving the cluster.
• A pair of MySQL servers run in master:master configuration on the backend hosts
• The backend hosts use DRBD to maintain a mirrored block device between them.
• These block devices back a NFS filesystem.
• Heartbeat runs on the backend hosts to do several tasks:
  1. Manage which host is the DRBD primary and therefore can be written to.
  2. Manage which host has the DRBD filesystem mounted and exported with NFS.
  3. Manage the IP through which the frontend mounts the filesystem and talks to MySQL.
• With all this in place, Nginx accepts web connections and serves static assets off the NFS mount and passess other requests to Mongrel, a HTTP server that’s well suited to running a Rails instance.

Notes

• One of the main hazards of MySQL master:master setups is primary key collision if an INSERT occurs on both hosts at once. We avoid that here by letting Hearbeat manage the IP that the frontends connect to.
• I’ve built two of these clusters to date. The second one is now four servers wide on the frontend.

Future work

• DRBD can now run in dual-primary mode, allowing both hosts to accept writes. This makes it a candidate for filesystems like GFS that use shared storage to present a filesystem that can be written to on multiple hosts. More here.
• To add some load balancing I’m considering using HAProxy or LVS to actively distribute traffic across the frontends.
• HA aside, there’s also some cool things like evented Mongrel that it would be interesting to try.

However the magic doesn’t touch everything. I was so bold as to move Dictionary.app from its default home in /Applications into the Utilities subdirectory. This broke the Ctrl-Cmd-D system-wide (Cocoa-wide?) lookup box. I don’t use it that much but that irked me nonetheless.

Console logs included this:

com.apple.launchd[335] (com.apple.DictionaryPanelAgent[490]): posix_spawn("/Applications/Dictionary.app/Contents/SharedSupport/DictionaryPanel.app/Contents/MacOS/DictionaryPanel", ...): No such file or directory

Clearly something thinks it knows where the nested DictionaryPanel application should live. This thing is a launchd-managed process used by the keyboard shortcut, configured out of /System/Library/LaunchAgents/com.apple.DictionaryPanelAgent.plist.

The hoojah to tweak this does of course involve XML:


$ launchctl unload /System/Library/LaunchAgents/com.apple.DictionaryPanelAgent.plist

$ plutil -convert xml1 -o - /System/Library/LaunchAgents/com.apple.DictionaryPanelAgent.plist | perl -pe 's|/Applications/Dictionary.app/|/Applications/Utilities/Dictionary.app/|;' > /var/tmp/com.apple.DictionaryPanelAgent.plist

$ plutil -lint /var/tmp/com.apple.DictionaryPanelAgent.plist
/var/tmp/com.apple.DictionaryPanelAgent.plist: OK

$ plutil -convert binary1 /var/tmp/com.apple.DictionaryPanelAgent.plist

$ mkdir /Library/LaunchAgents-orig

$ sudo mv /System/Library/LaunchAgents/com.apple.DictionaryPanelAgent.plist /Library/LaunchAgents-orig && sudo cp /var/tmp/com.apple.DictionaryPanelAgent.plist /System/Library/LaunchAgents/com.apple.DictionaryPanelAgent.plist

$ launchctl load /System/Library/LaunchAgents/com.apple.DictionaryPanelAgent.plist

The last step was intended to make the change work this session, but it didn’t work. The DictionaryPanelAgent still loads but didn’t do anything until logged out and in again. I think this is something to do with launchctl domains / sessiontype. Blunder factor is high.

This violates the “don’t frig with stuff in /System” principle but I don’t know how else to solve it. The modified plist could go in /Library/LaunchAgents of course, but I’d still need to disable the system version (with launchctl unload -w) which is equally naughty. I think.

Happily “the vm_kern.c.2 patch” (it’s well known!) sorted things. I stressed ZFS-backed IO massively without any further wobbles.

Now I can micro-quota all my jail(8)s with obsession!