I cut my UNIX teeth at a cryptoanarchist shop whose culture of paranoia makes me wary of Apple’s own firewall with its emphasis on letting all the hot shinyness Just Work rather than being overly fussy about inbound connections. Furthermore, with IPv6 tunnels like aiccu, you aren’t behind the warm fuzzy comfort of NAT, you’re just there on the net with all the fun that entails. So, worth having some extra protection I reckon.

10.6 (and earlier) came with ipfw, a packet filter that’s knocked around the BSD world for some time. It works but isn’t overly featuresome (for example, it doesn’t support NAT in-kernel, so you monkey about passing packets to an external daemon). But it was Good Enough for an end system so I supplemented the system’s “Application Firewall” with an additional ipfw ruleset to give an approximation of safety when out and about, and way more permissive when on networks I trust.

On 10.6 I used WaterRoof to sort-of manage the ipfw rules: in that I only really used its launchd loader and would hack on the rules by hand. In the spirit of decrufting I figured I’d sort that myself and went to remind myself how to load ipfw rules enmasse. I noticed at the top of man page:

NAME
ipfw -- IP firewall and traffic shaper control program (DEPRECATED)
...
DESCRIPTION
Note that use of this utility is DEPRECATED. Please use pfctl(8) instead


Deprecated? Use pfctl instead? Good news everyone – OS X 10.7 now comes with pf, another BSD packet filter that I’ve chosen of ipfw on BSD hosts for years off the back of its featureset (native NAT, state syncing between failover firewall pairs, traffic queing…).

Anyway, the point of this post is to point out a few things I noticed which are intriguing. Firstly, pf is not enabled by default. Further, Apple have added some moving parts around how it is enabled. From /etc/pf.conf:

# This file contains the main ruleset, which gets automatically loaded
# at startup. PF will not be automatically enabled, however. Instead,
# each component which utilizes PF is responsible for enabling and disabling
# PF via -E and -X as documented in pfctl(8). That will ensure that PF
# is disabled only when the last enable reference is released.

These two flags, -E and -X, are absent from pf on BSD. Here’s how they’re documented on OS X:

-E Enable the packet filter and increment the pf enable reference count.
-X token
Release the pf enable reference represented by the token passed.

This suggests that different system components might choose to enable and disable pf, and this is the mechanism to coordinate that. There’s a clue about which components in /etc/pf.anchors/com.apple, which is loaded by the main /etc/pf.conf. It defines additional rule anchors:

anchor "100.InternetSharing/*"
anchor "200.AirDrop/*"
anchor "250.ApplicationFirewall/*"


Interestingly, this host’s ApplicationFirewall has a bunch of entries in when viewed in the Preferences GUI, yet the pf anchor of the same name is empty (and pf was disabled when I started out):

$ sudo pfctl -a com.apple/250.ApplicationFirewall -s rules
Password:
No ALTQ support in kernel
ALTQ related functions disabled

so I’m unsure what the status of this mechanism is. I’ve not had occasion to use AirDrop or connection sharing, but would be curious to see if either use these anchors and enable pf temporarily.

Finally, what’s the token that’s passed to -X? You can ask pfctl for the current tokens:

$ sudo pfctl -s References
No ALTQ support in kernel
ALTQ related functions disabled
TOKENS:
PID Process Name TOKEN TIMESTAMP
17013 pfctl 18446743524308110600 0 days 01:05:50

I enabled pf with pfctl, so that makes sense. When I did so it didn’t inform me of the token, but I suppose an enabling process would spelunk the token shortly after enabling pf by merit of its name and PID and pass it back when it’s finished with pf.

Now, on with the actual job of ruleset writing and puzzling out the launchd voodoo required to enable it at boot.

Minor whinge: Apple could do with updating /etc/protocols:

# $FreeBSD: src/etc/protocols,v 1.14 2000/09/24 11:20:27 asmodai Exp $

Why whinge? It doesn’t know icmp6 is a valid alias for ipv6-icmp. Yep, minor.

I then found then whenever I sent mail using authenticated SMTP my mail server would lock up with saslauthd chewing the CPU. This authentication daemon is the glue between the MTA (Exim) and the IMAP server (Courier) – it logs into the IMAP service to test the SMTP user’s credentials. This little kink of indirection comes about because the IMAP daemon is downstream from the Exim host, in a BSD jail host, so its own authentication mechanisms aren’t visible to the MTA.

My new mail password contained a double-quote mark, which made me wonder if the password wasn’t being quoted properly. Testing a bit with openssl:

$ openssl s_client -starttls smtp -connect localhost:25
CONNECTED(00000003)
---
250 HELP
EHLO localhost
250-svc9.zomo.co.uk Hello localhost [127.0.0.1]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250 HELP
AUTH PLAIN AGZvbwAi < -- this is Base64 for username foo, password "

Compiling a -g debug variant of the daemon and aiming gdb at it:

$ sudo gdb /usr/local/sbin/saslauthd-debug 97103
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
...
(gdb) bt
#0 0x284250d1 in strchr () from /lib/libc.so.7
#1 0x0804a823 in qstring ()
#2 0x0804ac45 in auth_rimap ()
#3 0x0804f8e3 in do_auth ()
#4 0x0804e1f4 in do_request ()
#5 0x0804e53b in ipc_loop ()
#6 0x0805018d in main ()


What’s qstring()? It’s a function for escaping the quote marks in strings passed to the IMAP daemon. Turns out count-the-quotemark logic wasn’t properly advancing along the string, so it would sit there spinning forever.

Trivial patch¹ fixes:

$ openssl s_client -starttls smtp -connect localhost:25
CONNECTED(00000003)
---
250 HELP
EHLO localhost
250-svc9.zomo.co.uk Hello localhost [127.0.0.1]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250 HELP
AUTH PLAIN AGZvbwAi
535 Incorrect authentication data


Better :)

1. Gist if it’s not inlined above

Recently a client was receiving complaints that their busy server hosting both their WordPress sites and their OpenX¹ banner delivery was underperforming. Specifically, sites including their banners were seeing page loads hang on them. If you’re in the business of selling banners this is bad news. There were reports of the WordPress sites being slow too, but mostly from administrators² rather than site visitors.

I sorted out a bunch of request amplification issues but still things still weren’t right, so I added a second server to help out. Instead of just chucking the combined traffic at both servers I used HAProxy to separate out the traffic to each, with a view to adding more OpenX servers as necessary.

Here’s what HAProxy’s stats had to say after some time running the sites split:

Some of these I found unsurprising – WordPress serves a higher volume of data, it is content heavy compared to banner delivery and related click handling. Conversely the inbound data volume for OpenX is up because it’s loaded with click information.

What’s interesting is that the WordPress sites have a higher maximum concurrent session count, yet the total sessions is far higher for the OpenX banners. This illustrates the benefit of separating out different server loads: one server is churning away pushing out fat content and even when heavily cached this burns enough resource that requests get queued and gum up, whilst another is fielding quick-in quick-out requests. When it’s not contending with its laggard sibling it can get on with its business unhindered.

Ultimately the visibility HAProxy affords beats an Apache scoreboard when that Apache is fielding two differently focused workloads.

1. advertising is a necessary evil, right?
2. and that turned out to be a pagination issue

“Integration points are the number-one killer of systems” – Release It!, Michael Nygard

Last week I had two different web systems fail in a similar way. One was a single box running two busy WordPress sites, another was a largish multi-tier publishing cluster. Both dropped off air because they didn’t handle the failure of a remote system very well. In particular, both had the webserver waiting on a HTTP request to a foreign site to complete before returning a page to the client.

The architecture issues with this kind of request amplification are reasonably clear, and can sometimes be avoided. Data ingest from other systems can often execute asynchronously in another process. The cluster mentioned has a set of machines dedicated to just this, distributing the data they retrieve from remote systems via database and shared filesystem ready for the webservers to use when building pages.

However, in the situations where it is necessary for a webserver to dial out to another system at request time then it’s worth being really paranoid about how that outbound request works and untrusting of the reply. The particular problem here was lack of timeouts.

Here, both systems were running mpm_prefork Apache. Both were serving a request that made a HTTP call somewhere else before returning the page to the client¹. Last week the remote sites that both these systems contact had outages, leaving Apache waiting for the reply up until the request timed out.

However, with no timeout configured that wait is effectively infinite. Long enough for these hanging Apache processes to consume all the available slots of the webserver, resulting an interesting set of observations:

• New TCP connections to the server just hang.
• Server load is low, because the load measures runnable processes (and, on Linux, uninterruptible sleeping processes), and these hanging processes are just sleeping, waiting on poll(2) or select(2).
• No errors are logged by Apache or the application code because they haven’t failed yet.²

Here’s some vmstat during such a wedge:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 140580 56528 58168 516184 0 0 0 0 1026 78 0 0 100 0 0
0 0 140580 56528 58168 516184 0 0 0 0 1032 80 0 0 100 0 0
0 0 140580 56528 58168 516184 0 0 0 0 1031 78 0 0 100 0 0

“It was quiet. Too quiet”. Hook up gdb and you can see what’s going on:

(gdb) bt
#0 0x00002aafffb7d14f in poll () from /lib64/libc.so.6
#1 0x00002ab0090b3930 in Curl_select () from /usr/lib64/libcurl.so.3
#2 0x00002ab0090ac4bc in Curl_perform () from /usr/lib64/libcurl.so.3
#3 0x00002ab00850591b in zif_curl_exec () from /etc/httpd/modules/libphp5.so
#4 0x00002ab0086663b2 in ?? () from /etc/httpd/modules/libphp5.so
#5 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#6 0x00002ab00865dc09 in ?? () from /etc/httpd/modules/libphp5.so
#7 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#8 0x00002ab008688750 in ?? () from /etc/httpd/modules/libphp5.so
#9 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#10 0x00002ab00865dc09 in ?? () from /etc/httpd/modules/libphp5.so
#11 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#12 0x00002ab00865c11e in ?? () from /etc/httpd/modules/libphp5.so
#13 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#14 0x00002ab0086395de in zend_execute_scripts () from /etc/httpd/modules/libphp5.so
#15 0x00002ab0085fe697 in php_execute_script () from /etc/httpd/modules/libphp5.so
#16 0x00002ab0086b6ad6 in ?? () from /etc/httpd/modules/libphp5.so
#17 0x00002aaffdbb7a4a in ap_run_handler ()
#18 0x00002aaffdbbaed8 in ap_invoke_handler ()
#19 0x00002aaffdbc578a in ap_internal_redirect ()
#20 0x00002ab0069ffbc0 in ap_make_dirstr_parent () from /etc/httpd/modules/mod_rewrite.so
#21 0x00002aaffdbb7a4a in ap_run_handler ()
#22 0x00002aaffdbbaed8 in ap_invoke_handler ()
#23 0x00002aaffdbc5938 in ap_process_request ()
#24 0x00002aaffdbc2b70 in ?? ()
#25 0x00002aaffdbbecd2 in ap_run_process_connection ()
#26 0x00002aaffdbc9789 in ?? ()
#27 0x00002aaffdbc9a1a in ?? ()
#28 0x00002aaffdbc9ad0 in ?? ()
#29 0x00002aaffdbca7bb in ap_mpm_run ()
#30 0x00002aaffdba4e48 in main ()


Another kink is that a graceful restart of Apache, via SIGUSR1, doesn’t work since a graceful restart waits for a request to finish – and these ones aren’t finishing. apachectl or service(8) scripts will exit but the hung processes remain. These long running wedged processes are also visible in ps:

apache 580 0.0 0.3 30332 5492 ? S Jan02 0:00
/usr/sbin/httpd

and can be matched up with the hung connections via netstat -anp

tcp 0 0 172.18.74.113:50129 192.0.32.10:80
ESTABLISHED 580/httpd


This is a mostly a long winded way of pointing out the importance of timeouts on code that executes during a request, particularly when doing something non-local. Fail fast!

To keep the WordPress sites running I ran through the code adding CURLOPT_CONNECTTIMEOUT, CURLOPT_TIMEOUT settings to all the cURL calls I could find, and the development team took care of the code running on the cluster. Both have been stable since.

There are plenty of other pitfalls in these integration points between systems. Michael Nygard’s book, linked at top, makes a good survey of them alongside other stability antipatterns in complex systems. It’s a recommended read.

1. One of the WordPress sites was doing this in every page’s footer. And not caching the result. Better still, it was contacting the other WordPress site on the same server. This almost guarantees the request won’t return when the shared webserver gets busy since there may be no spare server slots to accept the second request. Ouch.
2. If you don’t run Apache hot, then you’d see a “I’ve reached MaxClients” message in the error log, but not much else.

Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
        The Regents of the University of California.  All rights reserved.

FreeBSD 8.0-RELEASE-p4 (GENERIC) #0: Mon Jul 12 20:22:27 UTC 2010

/libexec/ld-elf.so.1: Shared object "libintl.so.8" not found, required by "bash"
$

What’s happened here is that gettext (which provides libintl.so) has been upgraded without its dependencies noticing (read: I messed up some package maintenance). These dependencies are linked against a specific version of this library that’s now gone. One of these dependencies is bash, my preferred shell.

Having your shell asplode like that is ungood. If bash has been set as my default shell then I wouldn’t be able to log in at all! This is a risk of relying on software that isn’t part of FreeBSD’s core to absolutely always work across system maintenance. bash is a third-party port, and it’s prudent to anticipate package management fail¹.

To side-step this risk I tend to set my shell to something that is native to a FreeBSD release (plain ole’ /bin/sh) and adding the following in my .profile, to test if bash is indeed working before running it:

/usr/local/bin/bash -c true && exec /usr/local/bin/bash

And to fix all the other ports that are now broken:

$ sudo portmaster -r -R devel/gettext

1. because all package management sucks at some point, yes?

They’re also the source of plenty of flail. What do I mean?

• They are neither code nor data, so often get overlooked, or shonkily installed, by application deployment tools
• They run with a minimal environment that can catch out the unwary: scripts that work in interactive shell sometimes don’t from cron
• The default behaviour of mailing output to the cronjob owner generates large amounts of mail that gets ignored, filtered or bounced
• Jobs can fail silently and no-one notices until, say, you need to restore that backup that hasn’t run for last six months
• Jobs that helpfully append their output to a log commonly don’t rotate that log
• It’s easy to have jobs overlapping if they get stuck or take longer than expected to complete. This is a splendid way of wedging a machine.

The mail aspect is a particular peeve. In some jobs my mailbox has enjoyed several thousand cron generated mails a day, and there’s no way I’m able to accurately look at each one and react to it. Mostly they contain expected output from successful job execution, so they’re easy to skip. But I don’t trust my eyes to get that right all the time.

One approach to this is to arrange for jobs to only send mail on error. This is an improvement, but can lead into thinking that a job is happily succeeding when in fact it’s either not running or the only-on-error logic is bust. Since cron jobs often cover essential system tasks like backing up, syncing data around and reporting it’s vital that they don’t fail silently.

I’ve worked somewhere that tackled this by collating cron-generated mails from diverse systems into a system mailbox and pattern matching them for failure signs. This seems slightly dubious — it’s fragile and labour intensive — but at least the system also flagged if expected jobs failed to arrive and got our inboxes tamed.

To tackle these problems I find myself writing wrappers for cronjobs. I’ve written several variants to meet different situation’s needs. Unhelpfully I call them all cronwrap. These wrappers sets out to

• Engage the amazingly useful lockrun utility to guard against multiple execution of stuck crons
• Place cron output into timestamped logs that can be both aged out and made available to interested parties
• Hook into local monitoring systems:
  1. On execution, update a run counter (SNMP data or some simple text file)
  2. On failure, send a SNMP trap or leave some bait for Nagios. Also, update a fail counter
  3. If lockrun has prevented a job running owing to overlap, send a SNMP trap or similarly bait Nagios
• If required, send output by mail somewhere (sometimes this is necessary, even with the concerns listed above)

So, nothing surprising there. Using such wrappers helps keep cron jobs tamed and reliable, and it’s monitoring them near to where the action occurs, rather than mediating via SMTP.

This is hardly invention either, there’s plenty of prior art with different nuances in behaviour to meet the needs of different environments. Perhaps I’ll merge the variants of my efforts and publish too.

What’s curious is that this functionality isn’t available inside the cron daemon¹ itself. It is perfectly placed to catch exit status, divert output and know if a job has overrun; and would remove the need for all this additional monkeying to make jobs reliable and well behaved. If my C wasn’t just read-only I’d have a crack at it!

There, I’ve finally condensed all my cron rant into one sustained piece.

Update: I posted a cron wrapper at https://github.com/zomo/cronwrap.

1. To be clear, I’m talking about the BSD cron written by Paul Vixie. None of the variants I’ve seen address these concerns either. I’d love to know if there’s any I’ve missed.

The reference documentation has examples in Perl and PHP which got me so far, but I’m most comfortable in Ruby now, and was happy to find 1 this pointer to using the soap4r library.

Chief bonus here is the wsdl2ruby.rb tool that’ll transform the WSDL data into Ruby objects with heirarchy, attribute accessors and everything else to make operating the API really comfortable. If your WSDL is a moving target during development it’ll even do this on the fly.

This meant getting the scripting done to configure the ZXTMs was pretty straightforward, without any faffing with the underlying access mech. Refreshing!

1. I was going to paste example code, but this’ll do

Update: Turns out this build is missing isol mode serial-on-LAN for IPMI 1.5 hosts¹, and building the ipmitool distribution wasn’t so hard after all. Packaging systems have made me lazy!

1. Dell R200 uses IPMI 1.5, Dell R410 uses 2.0. Both do SOL and remote power management well enough

As a same-same-but-different alternative to

• pssh
• That xterm multiplexer I can never remember the name of
• screen(1) and Terminal.app‘s send-to-many option
• Quick and dirty shell scripts iterating over a host list
• Capistrano‘s shell (even on hosts that don’t run any other Ruby!)

… it’s proving pretty solid.

Example:

[admin@manage3 tmp]$ pattern.py foo[1-8] | vxargs -y -o ~/tmp ssh {} varnishadm -Tlocalhost:6082 ping


Terminal clears, and shows progress:

8/8:ssh -l root foo8 varnishadm -Tlocalhost:6082 ping
Done
Done
Done
Done
Done
( 10s) 6: foo7
Done

and exits

exit code 0: 7 job(s)
exit code 1: 1 job(s)
total number of jobs: 8

Now you can inspect the output from each host, exit status and an overall failure list

[admin@manage3 tmp]$ cat ~/tmp/abnormal_list
foo5
[admin@manage3 tmp]$ cat ~/tmp/foo5.err
Warning: Permanently added 'foo5,10.221.11.0' (RSA) to the list of known hosts.
connect(): Connection refused
An error occured in receiving status.


Via Trivium