I’ve often wondered about how to do this without monkeying in Objective C. It’s doable in (ugh) Applescript with some property list editing. Whimsical XKCD handler such that xkcd://627 behaves like http://xckd.com/627:


on open location uri
  set delimiter to AppleScript's text item delimiters
  set AppleScript's text item delimiters to "//"
  set comic to text item 2 of uri
  set page to "http://xkcd.com/" & comic
  tell application "Safari"
    activate
    open location page
  end tell
  set AppleScript's text item delimiters to delimiter
end open location


Save as an Application rather than a Script. Open its info.plist and add

 <key>CFBundleIdentifier</key>
 <string>uk.co.zomo.xkcd-handler</string>
 <key>CFBundleURLTypes</key>
 <array>
  <dict>
   <key>CFBundleURLName</key>
   <string>XKCD helper</string>
   <key>CFBundleURLSchemes</key>
   <array>
    <string>xkcd</string>
   </array>
  </dict>
 </array>


To nudge OS X to notice changed a plist:

/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -v -f ~/Library/Scripts/xkcd-handler.app


Links:

• http://applescript123.com/linktrigger/
• http://stackoverflow.com/questions/8981749/how-do-i-refresh-application-metadata-url-handlers-web-browsers-after-insta
• http://www.rubicode.com/Software/RCDefaultApp/

• Set open files to 65535 via ulimit -n
• Recompile httperf with /usr/include/bits/typesizes.h‘s __FD_SETSIZE dialled up to 65535¹
• Twiddle the TCP stack for faster socket recycling²³ :
  
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse


1. http://gom-jabbar.org/articles/2009/02/04/httperf-and-file-descriptors
2. http://www.speedguide.net/articles/linux-tweaking-121
3. http://serverfault.com/questions/331513/httperf-hangs-when-using-hog

• Moving virtual machines from Xen to KVM
• The storage for both is under LVM
• Under Xen the storage for each VM isn’t partitioned as a disk, it’s just a filesystem. GRUB under KVM will need to see a boot sector and a partition table.
• Under Xen the storage for each VM doesn’t contain a kernel. GRUB under KVM will need one to boot the VM.

Sequence

• Measure disks on Xen server, create on KVM server. Create the new root disk 10MB larger than its counterpart.¹
  
kyero@xen03 ~ $ sudo lvscan
ACTIVE '/dev/vg4/mail-root' [20.00 GB] inherit
ACTIVE '/dev/vg4/mail-swap' [1.00 GB] inherit


[kyero@vm02 ~]$ sudo lvcreate -L20490M -n mail-root vg4
[kyero@vm02 ~]$ sudo lvcreate -L2G -n mail-swap vg4

• Create a boot disk on the KVM server. We make it 4 cylinders long, roughly 32MB.²
  
[kyero@vm02 ~]$ sudo dd if=/dev/zero of=/var/boot-images/mail-boot bs=32901120 count=1

• Partition the boot disk with one partition starting at cylinder zero
  
  [kyero@vm02 ~]$ sudo sfdisk /var/boot-images/mail-boot
  last_lba(): I don't know how to handle files with mode 81a4
  Warning: /var/boot-images/mail-boot is not a block device
  Disk /var/boot-images/mail-boot: cannot get geometry

  Disk /var/boot-images/mail-boot: 4 cylinders, 255 heads, 63 sectors/track

  sfdisk: ERROR: sector 0 does not have an msdos signature
  /var/boot-images/mail-boot: unrecognized partition table type
  Old situation:
  No partitions found
  Input in the following format; absent fields get a default value.
  <start> <size> <type [E,S,L,X,hex]> <bootable [-,*]> <c,h,s> <c,h,s>\nUsually you only need to specify <start> and <size> (and perhaps <type>).

  /var/boot-images/mail-boot1 :0
  /var/boot-images/mail-boot1 0+ 3 4- 32129+ 83 Linux
  /var/boot-images/mail-boot2 :
  /var/boot-images/mail-boot2 0 - 0 0 0 Empty
  /var/boot-images/mail-boot3 :
  /var/boot-images/mail-boot3 0 - 0 0 0 Empty
  /var/boot-images/mail-boot4 :
  /var/boot-images/mail-boot4 0 - 0 0 0 Empty
  New situation:
  Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

  Device Boot Start End #cyls #blocks Id System
  /var/boot-images/mail-boot1 0+ 3 4- 32129+ 83 Linux
  /var/boot-images/mail-boot2 0 - 0 0 0 Empty
  /var/boot-images/mail-boot3 0 - 0 0 0 Empty
  /var/boot-images/mail-boot4 0 - 0 0 0 Empty
  Warning: no primary partition is marked bootable (active)
  This does not matter for LILO, but the DOS MBR will not boot this disk.
  Do you want to write this to disk? [ynq] y
  Successfully wrote the new partition table

  Re-reading the partition table ...
  BLKRRPART: Inappropriate ioctl for device

  If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
  to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1
  (See fdisk(8).)

• Format and mount the partition
  
  [kyero@vm02 ~]$ sudo kpartx -av /var/boot-images/mail-boot
  add map loop1p1 : 0 64259 linear /dev/loop1 1

  [kyero@vm02 ~]$ sudo mke2fs /dev/mapper/loop1p1
  mke2fs 1.39 (29-May-2006)
  Filesystem label=
  OS type: Linux
  Block size=1024 (log=0)
  Fragment size=1024 (log=0)
  8032 inodes, 32128 blocks
  1606 blocks (5.00%) reserved for the super user
  First data block=1
  Maximum filesystem blocks=33030144
  4 block groups
  8192 blocks per group, 8192 fragments per group
  2008 inodes per group
  Superblock backups stored on blocks:
  8193, 24577

  Writing inode tables: done
  Writing superblocks and filesystem accounting information: done

  This filesystem will be automatically checked every 34 mounts or
  180 days, whichever comes first. Use tune2fs -c or -i to override.

  [kyero@vm02 ~]$ sudo mount /dev/mapper/loop1p1 /mnt/boot/

  [kyero@vm02 ~]$ df -h /mnt/boot
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/loop1p1 31M 389K 29M 2% /mnt/boot


• Copy a kernel to it, configure GRUB’s notion of the root device and install the boot sector.
  
  [kyero@vm02 ~]$ sudo rsync -a /boot/ /mnt/boot/
  [kyero@vm02 ~]$ sudo vi /mnt/boot/grub/menu.lst
  [kyero@vm02 ~]$ sudo touch /mnt/boot/kvm-boot-image
  [kyero@vm02 ~]$ sudo grub --device-map=/dev/null

  GNU GRUB version 0.97 (640K lower / 3072K upper memory)

  [ Minimal BASH-like line editing is supported. For the first word, TAB
  lists possible command completions. Anywhere else TAB lists the possible
  completions of a device/filename.]
  grub> device (hd0) /var/boot-images/mail-boot
  device (hd0) /var/boot-images/mail-boot
  grub> find /kvm-boot-image
  find /kvm-boot-image
  (hd0,0)
  grub> root (hd0,0)
  root (hd0,0)
  Filesystem type is ext2fs, partition type 0x83
  grub> setup (hd0)
  setup (hd0)
  Checking if "/boot/grub/stage1" exists... no
  Checking if "/grub/stage1" exists... yes
  Checking if "/grub/stage2" exists... yes
  Checking if "/grub/e2fs_stage1_5" exists... yes
  Running "embed /grub/e2fs_stage1_5 (hd0)"... failed (this is not fatal)
  Running "embed /grub/e2fs_stage1_5 (hd0,0)"... failed (this is not fatal)
  Done.
  grub> quit

  [kyero@vm02 ~]$ sudo umount /mnt/boot
  [kyero@vm02 ~]$ sudo kpartx -d /var/boot-images/mail-boot
  loop deleted : /dev/loop1

• Populate the new root disk with one cylinder of zeros and a copy of the old root disk.
  The first cylinder will contain the MBR and not much else. The padding means it’s easy to align the old root disk at cylinder one.
  Use pv so we can see the copy progress.
  The dd-lvm-out script is mostly just a sudo-blessed wrapper.
  [kyero@vm02 ~]$ ( dd if=/dev/zero bs=8225280 count=1 ; ssh xen03.local -l kyero sudo /data/system/bin/dd-lvm-out.sh mail-root ) | pv -e -r -s 21474836480 | sudo dd bs=1048577 of=/dev/vg4/mail-root
1+0 records in
1+0 records out
8225280 bytes (8.2 MB) copied, 0.03395 seconds, 242 MB/s
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB) copied, 611.215 s, 35.1 MB/s
0+662630 records in
0+662630 records out
21483061795 bytes (21 GB) copied, 615.249 seconds, 34.9 MB/s

• Partition the new root disk with a single partition starting at cylinder one.
  [kyero@vm02 ~]$ sudo sfdisk /dev/vg4/mail-root
  Checking that no-one is using this disk right now ...
  BLKRRPART: Invalid argument
  OK

  Disk /dev/vg4/mail-root: 2614 cylinders, 255 heads, 63 sectors/track

  sfdisk: ERROR: sector 0 does not have an msdos signature
  /dev/vg4/mail-root: unrecognized partition table type
  Old situation:
  No partitions found
  Input in the following format; absent fields get a default value.
  <start> <size> <type [E,S,L,X,hex]> <bootable [-,*]> <c,h,s> <c,h,s>\nUsually you only need to specify <start> and <size> (and perhaps <type>).

  /dev/vg4/mail-root1 :1
  /dev/vg4/mail-root1 1 2613 2613 20988922+ 83 Linux
  /dev/vg4/mail-root2 :
  /dev/vg4/mail-root2 0+ 0 1- 8032 83 Linux
  /dev/vg4/mail-root3 :
  /dev/vg4/mail-root3 0 - 0 0 0 Empty
  /dev/vg4/mail-root4 :
  /dev/vg4/mail-root4 0 - 0 0 0 Empty
  New situation:
  Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

  Device Boot Start End #cyls #blocks Id System
  /dev/vg4/mail-root1 1 2613 2613 20988922+ 83 Linux
  /dev/vg4/mail-root2 0+ 0 1- 8032 83 Linux
  /dev/vg4/mail-root3 0 - 0 0 0 Empty
  /dev/vg4/mail-root4 0 - 0 0 0 Empty
  Warning: no primary partition is marked bootable (active)
  This does not matter for LILO, but the DOS MBR will not boot this disk.
  Do you want to write this to disk? [ynq] y
  Successfully wrote the new partition table

  Re-reading the partition table ...
  BLKRRPART: Invalid argument

  If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
  to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1
  (See fdisk(8).)

• Mount the new root filesystem
  [kyero@vm02 ~]$ sudo kpartx -av /dev/vg4/mail-root
  add map mail-root1 : 0 41977845 linear /dev/vg4/mail-root 16065
  add map mail-root2 : 0 16064 linear /dev/vg4/mail-root 1
  [kyero@vm02 ~]$ sudo e2fsck /dev/mapper/mail-root1
  e2fsck 1.39 (29-May-2006)
  /dev/mapper/mail-root1: clean, 435213/1310720 files, 4086961/5242880 blocks

  [kyero@vm02 ~]$ sudo mount /dev/mapper/mail-root1 /mnt/tmp


• Add the new kernel’s modules, tweak filesystem mounts, disable swap, check console.
  
  [kyero@vm02 ~]$ sudo rsync -a /lib/modules/ /mnt/tmp/lib/modules/

  [kyero@vm02 ~]$ sudo vi /mnt/tmp/etc/fstab

  [kyero@vm02 ~]$ sudo vi /mnt/tmp/etc/inittab


• Create the KVM guest, assigning the boot, root and swap storage
  [kyero@vm02 ~]$ sudo virt-install -n mail -r 1024 --vcpus 1 --os-type linux --os-variant generic26 --accelerate --disk path=/var/boot-images/mail-boot --disk path=/dev/mapper/vg4-mail--root,bus=ide,format=raw --disk path=/dev/mapper/vg4-mail--swap,bus=ide,format=raw --network bridge:br0 --network bridge:br1 --import --noreboot
  Starting install...
  Guest installation complete... you can restart your domain
  by running 'virsh start mail'

  [kyero@vm02 ~]$ sudo virsh dumpxml mail
...
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/var/boot-images/mail-boot'/>
<target dev='hda' bus='ide'/>
<address type='drive' controller='0' bus='0' unit='0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='/dev/mapper/vg4-mail--root'/>
<target dev='hdb' bus='ide'/>
<address type='drive' controller='0' bus='0' unit='1'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='/dev/mapper/vg4-mail--swap'/>
<target dev='hdd' bus='ide'/>
<address type='drive' controller='0' bus='1' unit='1'/>
</disk>
...


• Boot the KVM guest
  
[kyero@vm02 ~]$ sudo virsh start mail
[kyero@vm02 ~]$ sudo virsh autostart mail
[kyero@vm02 ~]$ sudo virsh console mail

• Log into KVM guest, partition /dev/hdd and configure it as swap, update /etc/fstab.

1. Later we align the original root filesystem at cylinder 1 on this new disk. A LVM cylinder is roughly 8MB
2. File backed disk is used rather than LVM owing to grief with GRUB and LVM (Error 22: No such partition)

This log directory was allowed to grow indefinitely (ahem) until I trimmed it this morning. Now it is mostly empty (1.7MB), but the directory entry is still huge (at 1.6MB, compared to the default 4kB). I had a vague idea of why this happens but realised I was unclear on the specifics so it was time for The Learning.

On traditional filesystems (here, FFS on FreeBSD and ext2 on GNU/Linux) directories look much like files, just with special content. They have an inode that points to either blocks on disk or to indirection blocks that themselves point to blocks (or to other indirection blocks, up to three levels of indirection). The contents of these directory blocks are reasonably straightforward: they’re a sequence of dirent structures that generally contain a filename and that file’s inode number along with some small housekeeping data.

As a directory fills this sequence of dirents grows to fill all the direct blocks, then starts on the indirect blocks and so on. Each time the sequence crosses a block boundary the directory’s own size, rather than the total of the files it contains, grows by another block. Since filenames have variable length a single block will contain a variable number of dirents.

So far so good. What happens you delete a file from a directory? Other than freeing the actual file’s inode and respective blocks something must happen to its dirent entry in the directory’s data. What doesn’t happen is that the dirent is removed and all the ones after it get bumped towards the front of the list, which would lead to fewer overall directory blocks given enough deletions and ultimately smaller directories.

Why this doesn’t happen partly relates to the guarantees that telldir and friends make to programs with the directory open: a file’s relative position in the sequence of dirent will not change for any process that has the directory open. The deletion process can’t compact the directory itself without breaking that guarantee. Both FFS and ext2 blank out the file’s inode in the dirent record (ext2 with zero, FFS with a special value). Subsequent directory operations will ignore this blanked dirent, and processes which knew its sequence number via telldir will error if they attempt operations on it.

To recover the disk space used by a directory that has ballooned and then contracted its simplest to simply delete it entirely and recreate it. ext2 has an option (-D) in its filesystem check to compact the directory offline.

Interestingly this behaviour isn’t present on all filesystems, particularly those that use indexes instead of special files for directories: for example ZFS and HFS+. They can remove a file’s entry in the directory index yet still keep the remaining files appearing at their same positions by informed updates to the index.

Here is these four filesystem’s compared by

• Creating an empty directory
• Filling it with a million files
• Removing all but one of those files

The comparison shows how on ext2 and FFS the directory itself remains at its peak size after the test files are removed, whereas ZFS and HFS+ contract.

ext2

[build@centos-x86-64-build tmp]$ df -ih .
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sdb 1.3M 11 1.3M 1% /mnt/tmp
[build@centos-x86-64-build tmp]$ ls -ild .
2 drwxr-xr-x. 3 build build 4096 Mar 19 08:58 .

[build@centos-x86-64-build tmp]$ seq --format 'f%06g' 0 999998 | xargs rm
[build@centos-x86-64-build tmp]$ df -ih .
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sdb 1.3M 12 1.3M 1% /mnt/tmp
[build@centos-x86-64-build tmp]$ ls -ild .
2 drwxr-xr-x. 3 build build 23224320 Mar 19 09:02 .


FFS

[lemon@zest ~/tmp]$ df -ih .
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/mirror/gm0s1a 19G 10G 7.5G 58% 587k 2.1M 22% /
[lemon@zest ~/tmp]$ ls -ild .
521826 drwxr-xr-x 2 lemon lemon 512 Mar 19 14:50 .

[lemon@zest ~/tmp]$ jot -w "f%06d" 999999 0 | xargs rm
[lemon@zest ~/tmp]$ df -ih .
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/mirror/gm0s1a 19G 10G 7.5G 58% 587k 2.1M 22% /
[lemon@zest ~/tmp]$ ls -ild .
521826 drwxr-xr-x 2 lemon lemon 16000512 Mar 20 01:44 .


ZFS

[lemon@zest /mnt/ark/tmp]$ df -ih .
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
ark 207G 686M 206G 0% 58k 432M 0% /mnt/ark
[lemon@zest /mnt/ark/tmp]$ ls -ild
5 drwxrwsr-x 2 root wheel 2 Mar 20 06:18 .

[lemon@zest /mnt/ark/tmp]$ jot -w "f%06d" 999999 0 | xargs rm
[lemon@zest /mnt/ark/tmp]$ df -ih .
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
ark 207G 727M 206G 0% 58k 432M 0% /mnt/ark
[lemon@zest /mnt/ark/tmp]$ ls -ild
5 drwxrwsr-x 2 root wheel 3 Mar 20 09:27 .


HFS+


[lemon@further tmp] 0 $ df -ih .
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk1 233Gi 185Gi 48Gi 80% 48506927 12484688 80% /
[lemon@further tmp] 0 $ ls -ild .
12149971 drwxr-xr-x 2 lemon staff 68 19 Mar 16:06 .

[lemon@further tmp] 0 $ jot -w "f%06d" 999999 0 | xargs rm
[lemon@further tmp] 0 $ df -ih .
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk1 233Gi 185Gi 48Gi 80% 48524492 12467123 80% /
[lemon@further tmp] 0 $ ls -ild .
12149971 drwxr-xr-x 2 lemon staff 102 19 Mar 19:08 .


I cut my UNIX teeth at a cryptoanarchist shop whose culture of paranoia makes me wary of Apple’s own firewall with its emphasis on letting all the hot shinyness Just Work rather than being overly fussy about inbound connections. Furthermore, with IPv6 tunnels like aiccu, you aren’t behind the warm fuzzy comfort of NAT, you’re just there on the net with all the fun that entails. So, worth having some extra protection I reckon.

10.6 (and earlier) came with ipfw, a packet filter that’s knocked around the BSD world for some time. It works but isn’t overly featuresome (for example, it doesn’t support NAT in-kernel, so you monkey about passing packets to an external daemon). But it was Good Enough for an end system so I supplemented the system’s “Application Firewall” with an additional ipfw ruleset to give an approximation of safety when out and about, and way more permissive when on networks I trust.

On 10.6 I used WaterRoof to sort-of manage the ipfw rules: in that I only really used its launchd loader and would hack on the rules by hand. In the spirit of decrufting I figured I’d sort that myself and went to remind myself how to load ipfw rules enmasse. I noticed at the top of man page:

NAME
ipfw -- IP firewall and traffic shaper control program (DEPRECATED)
...
DESCRIPTION
Note that use of this utility is DEPRECATED. Please use pfctl(8) instead


Deprecated? Use pfctl instead? Good news everyone – OS X 10.7 now comes with pf, another BSD packet filter that I’ve chosen of ipfw on BSD hosts for years off the back of its featureset (native NAT, state syncing between failover firewall pairs, traffic queing…).

Anyway, the point of this post is to point out a few things I noticed which are intriguing. Firstly, pf is not enabled by default. Further, Apple have added some moving parts around how it is enabled. From /etc/pf.conf:

# This file contains the main ruleset, which gets automatically loaded
# at startup. PF will not be automatically enabled, however. Instead,
# each component which utilizes PF is responsible for enabling and disabling
# PF via -E and -X as documented in pfctl(8). That will ensure that PF
# is disabled only when the last enable reference is released.

These two flags, -E and -X, are absent from pf on BSD. Here’s how they’re documented on OS X:

-E Enable the packet filter and increment the pf enable reference count.
-X token
Release the pf enable reference represented by the token passed.

This suggests that different system components might choose to enable and disable pf, and this is the mechanism to coordinate that. There’s a clue about which components in /etc/pf.anchors/com.apple, which is loaded by the main /etc/pf.conf. It defines additional rule anchors:

anchor "100.InternetSharing/*"
anchor "200.AirDrop/*"
anchor "250.ApplicationFirewall/*"


Interestingly, this host’s ApplicationFirewall has a bunch of entries in when viewed in the Preferences GUI, yet the pf anchor of the same name is empty (and pf was disabled when I started out):

$ sudo pfctl -a com.apple/250.ApplicationFirewall -s rules
Password:
No ALTQ support in kernel
ALTQ related functions disabled

so I’m unsure what the status of this mechanism is. I’ve not had occasion to use AirDrop or connection sharing, but would be curious to see if either use these anchors and enable pf temporarily.

Finally, what’s the token that’s passed to -X? You can ask pfctl for the current tokens:

$ sudo pfctl -s References
No ALTQ support in kernel
ALTQ related functions disabled
TOKENS:
PID Process Name TOKEN TIMESTAMP
17013 pfctl 18446743524308110600 0 days 01:05:50

I enabled pf with pfctl, so that makes sense. When I did so it didn’t inform me of the token, but I suppose an enabling process would spelunk the token shortly after enabling pf by merit of its name and PID and pass it back when it’s finished with pf.

Now, on with the actual job of ruleset writing and puzzling out the launchd voodoo required to enable it at boot.

Minor whinge: Apple could do with updating /etc/protocols:

# $FreeBSD: src/etc/protocols,v 1.14 2000/09/24 11:20:27 asmodai Exp $

Why whinge? It doesn’t know icmp6 is a valid alias for ipv6-icmp. Yep, minor.

I then found then whenever I sent mail using authenticated SMTP my mail server would lock up with saslauthd chewing the CPU. This authentication daemon is the glue between the MTA (Exim) and the IMAP server (Courier) – it logs into the IMAP service to test the SMTP user’s credentials. This little kink of indirection comes about because the IMAP daemon is downstream from the Exim host, in a BSD jail host, so its own authentication mechanisms aren’t visible to the MTA.

My new mail password contained a double-quote mark, which made me wonder if the password wasn’t being quoted properly. Testing a bit with openssl:

$ openssl s_client -starttls smtp -connect localhost:25
CONNECTED(00000003)
---
250 HELP
EHLO localhost
250-svc9.zomo.co.uk Hello localhost [127.0.0.1]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250 HELP
AUTH PLAIN AGZvbwAi < -- this is Base64 for username foo, password "

Compiling a -g debug variant of the daemon and aiming gdb at it:

$ sudo gdb /usr/local/sbin/saslauthd-debug 97103
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
...
(gdb) bt
#0 0x284250d1 in strchr () from /lib/libc.so.7
#1 0x0804a823 in qstring ()
#2 0x0804ac45 in auth_rimap ()
#3 0x0804f8e3 in do_auth ()
#4 0x0804e1f4 in do_request ()
#5 0x0804e53b in ipc_loop ()
#6 0x0805018d in main ()


What’s qstring()? It’s a function for escaping the quote marks in strings passed to the IMAP daemon. Turns out count-the-quotemark logic wasn’t properly advancing along the string, so it would sit there spinning forever.

Trivial patch¹ fixes:

$ openssl s_client -starttls smtp -connect localhost:25
CONNECTED(00000003)
---
250 HELP
EHLO localhost
250-svc9.zomo.co.uk Hello localhost [127.0.0.1]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250 HELP
AUTH PLAIN AGZvbwAi
535 Incorrect authentication data


Better :)

1. Gist if it’s not inlined above

Recently a client was receiving complaints that their busy server hosting both their WordPress sites and their OpenX¹ banner delivery was underperforming. Specifically, sites including their banners were seeing page loads hang on them. If you’re in the business of selling banners this is bad news. There were reports of the WordPress sites being slow too, but mostly from administrators² rather than site visitors.

I sorted out a bunch of request amplification issues but still things still weren’t right, so I added a second server to help out. Instead of just chucking the combined traffic at both servers I used HAProxy to separate out the traffic to each, with a view to adding more OpenX servers as necessary.

Here’s what HAProxy’s stats had to say after some time running the sites split:

Some of these I found unsurprising – WordPress serves a higher volume of data, it is content heavy compared to banner delivery and related click handling. Conversely the inbound data volume for OpenX is up because it’s loaded with click information.

What’s interesting is that the WordPress sites have a higher maximum concurrent session count, yet the total sessions is far higher for the OpenX banners. This illustrates the benefit of separating out different server loads: one server is churning away pushing out fat content and even when heavily cached this burns enough resource that requests get queued and gum up, whilst another is fielding quick-in quick-out requests. When it’s not contending with its laggard sibling it can get on with its business unhindered.

Ultimately the visibility HAProxy affords beats an Apache scoreboard when that Apache is fielding two differently focused workloads.

1. advertising is a necessary evil, right?
2. and that turned out to be a pagination issue

“Integration points are the number-one killer of systems” – Release It!, Michael Nygard

Last week I had two different web systems fail in a similar way. One was a single box running two busy WordPress sites, another was a largish multi-tier publishing cluster. Both dropped off air because they didn’t handle the failure of a remote system very well. In particular, both had the webserver waiting on a HTTP request to a foreign site to complete before returning a page to the client.

The architecture issues with this kind of request amplification are reasonably clear, and can sometimes be avoided. Data ingest from other systems can often execute asynchronously in another process. The cluster mentioned has a set of machines dedicated to just this, distributing the data they retrieve from remote systems via database and shared filesystem ready for the webservers to use when building pages.

However, in the situations where it is necessary for a webserver to dial out to another system at request time then it’s worth being really paranoid about how that outbound request works and untrusting of the reply. The particular problem here was lack of timeouts.

Here, both systems were running mpm_prefork Apache. Both were serving a request that made a HTTP call somewhere else before returning the page to the client¹. Last week the remote sites that both these systems contact had outages, leaving Apache waiting for the reply up until the request timed out.

However, with no timeout configured that wait is effectively infinite. Long enough for these hanging Apache processes to consume all the available slots of the webserver, resulting an interesting set of observations:

• New TCP connections to the server just hang.
• Server load is low, because the load measures runnable processes (and, on Linux, uninterruptible sleeping processes), and these hanging processes are just sleeping, waiting on poll(2) or select(2).
• No errors are logged by Apache or the application code because they haven’t failed yet.²

Here’s some vmstat during such a wedge:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 140580 56528 58168 516184 0 0 0 0 1026 78 0 0 100 0 0
0 0 140580 56528 58168 516184 0 0 0 0 1032 80 0 0 100 0 0
0 0 140580 56528 58168 516184 0 0 0 0 1031 78 0 0 100 0 0

“It was quiet. Too quiet”. Hook up gdb and you can see what’s going on:

(gdb) bt
#0 0x00002aafffb7d14f in poll () from /lib64/libc.so.6
#1 0x00002ab0090b3930 in Curl_select () from /usr/lib64/libcurl.so.3
#2 0x00002ab0090ac4bc in Curl_perform () from /usr/lib64/libcurl.so.3
#3 0x00002ab00850591b in zif_curl_exec () from /etc/httpd/modules/libphp5.so
#4 0x00002ab0086663b2 in ?? () from /etc/httpd/modules/libphp5.so
#5 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#6 0x00002ab00865dc09 in ?? () from /etc/httpd/modules/libphp5.so
#7 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#8 0x00002ab008688750 in ?? () from /etc/httpd/modules/libphp5.so
#9 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#10 0x00002ab00865dc09 in ?? () from /etc/httpd/modules/libphp5.so
#11 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#12 0x00002ab00865c11e in ?? () from /etc/httpd/modules/libphp5.so
#13 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#14 0x00002ab0086395de in zend_execute_scripts () from /etc/httpd/modules/libphp5.so
#15 0x00002ab0085fe697 in php_execute_script () from /etc/httpd/modules/libphp5.so
#16 0x00002ab0086b6ad6 in ?? () from /etc/httpd/modules/libphp5.so
#17 0x00002aaffdbb7a4a in ap_run_handler ()
#18 0x00002aaffdbbaed8 in ap_invoke_handler ()
#19 0x00002aaffdbc578a in ap_internal_redirect ()
#20 0x00002ab0069ffbc0 in ap_make_dirstr_parent () from /etc/httpd/modules/mod_rewrite.so
#21 0x00002aaffdbb7a4a in ap_run_handler ()
#22 0x00002aaffdbbaed8 in ap_invoke_handler ()
#23 0x00002aaffdbc5938 in ap_process_request ()
#24 0x00002aaffdbc2b70 in ?? ()
#25 0x00002aaffdbbecd2 in ap_run_process_connection ()
#26 0x00002aaffdbc9789 in ?? ()
#27 0x00002aaffdbc9a1a in ?? ()
#28 0x00002aaffdbc9ad0 in ?? ()
#29 0x00002aaffdbca7bb in ap_mpm_run ()
#30 0x00002aaffdba4e48 in main ()


Another kink is that a graceful restart of Apache, via SIGUSR1, doesn’t work since a graceful restart waits for a request to finish – and these ones aren’t finishing. apachectl or service(8) scripts will exit but the hung processes remain. These long running wedged processes are also visible in ps:

apache 580 0.0 0.3 30332 5492 ? S Jan02 0:00
/usr/sbin/httpd

and can be matched up with the hung connections via netstat -anp

tcp 0 0 172.18.74.113:50129 192.0.32.10:80
ESTABLISHED 580/httpd


This is a mostly a long winded way of pointing out the importance of timeouts on code that executes during a request, particularly when doing something non-local. Fail fast!

To keep the WordPress sites running I ran through the code adding CURLOPT_CONNECTTIMEOUT, CURLOPT_TIMEOUT settings to all the cURL calls I could find, and the development team took care of the code running on the cluster. Both have been stable since.

There are plenty of other pitfalls in these integration points between systems. Michael Nygard’s book, linked at top, makes a good survey of them alongside other stability antipatterns in complex systems. It’s a recommended read.

1. One of the WordPress sites was doing this in every page’s footer. And not caching the result. Better still, it was contacting the other WordPress site on the same server. This almost guarantees the request won’t return when the shared webserver gets busy since there may be no spare server slots to accept the second request. Ouch.
2. If you don’t run Apache hot, then you’d see a “I’ve reached MaxClients” message in the error log, but not much else.

Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
        The Regents of the University of California.  All rights reserved.

FreeBSD 8.0-RELEASE-p4 (GENERIC) #0: Mon Jul 12 20:22:27 UTC 2010

/libexec/ld-elf.so.1: Shared object "libintl.so.8" not found, required by "bash"
$

What’s happened here is that gettext (which provides libintl.so) has been upgraded without its dependencies noticing (read: I messed up some package maintenance). These dependencies are linked against a specific version of this library that’s now gone. One of these dependencies is bash, my preferred shell.

Having your shell asplode like that is ungood. If bash has been set as my default shell then I wouldn’t be able to log in at all! This is a risk of relying on software that isn’t part of FreeBSD’s core to absolutely always work across system maintenance. bash is a third-party port, and it’s prudent to anticipate package management fail¹.

To side-step this risk I tend to set my shell to something that is native to a FreeBSD release (plain ole’ /bin/sh) and adding the following in my .profile, to test if bash is indeed working before running it:

/usr/local/bin/bash -c true && exec /usr/local/bin/bash

And to fix all the other ports that are now broken:

$ sudo portmaster -r -R devel/gettext

1. because all package management sucks at some point, yes?

People

I went with a bunch of colleagues from the BBC, hooked up with some people I’d not seen for ages, unexpectedly met in real life people I’d worked with online previously and met a whole load of talented and interested engineers.

Talks

The conference was split between set talks and open space sessions. The talk I got the most out of was Stephen Nelson Smith‘s talk on his experience bringing devops-shaped thinking to a large government site refresh. His thoughts about what worked and what didn’t were interesting and well made.

John Willis gave an excellent presentation on Chef. I’ve monkeyed a bit with Chef but am only using Puppet in real world situations so far. John’s a powerful evangelist and definitely got me enthused about Chef. I deliberately didn’t ask about Chef’s lack of dry-run mode since I’ve banged on about it on the chef-users list twice now¹, and because I wanted to see if anyone else flagged it as a concern – they did. All that aside, I am certainly going to get it up in a lab soon.

Link dump

• Marionette Collective (mcollective) – message bus framework for mass host control. I’m really interested by this. Seems similar to Engine Yard‘s Vertebrae thing, but perhaps more immediately useable for me.
• Fabric – a more traditional (ie: centralised) mass control mechanism. We’ve all rolled our own variant of this at some point, right?
• Zookeeper – mentioned in passing, need to check this out.
• cucumber-puppet – not currently doing anything like this, probably should!
• Circonus – flagged in passing, monitoring.
• Web Operations – seems to be a good compliment to Release It!, will grab and read.

Après-ski

The conference laid on a boat trip around Hamburg’s canals and harbour on the Friday night with beer and soup. Brilliant! Afterwards we maxed out the lounge area of the Chilli Club. After sufficient beers and cocktails we headed off to the Bunker club for an epic all nighter². Clearly we didn’t know about the school gym dress theme but hey that didn’t matter. The utterly bizarre story that unfolded on stage throughout the night was hilarious. House of Pain MCd by a large sweaty man in a baby costume? Oh yes! (more outrageous moments omitted…)

Saturday night included more Brownian motion around the Reeperbahn, plus Mike and I hanging out in a bar 0wned by St Pauli supporters. I don’t know much about football but these guys are clearly very dedicated to their cause! Edgy. More drinks into the dawn, then tactical sleep before flying home.

Hamburgers

I was consistently amazed how friendly everyone is. Wandering around Hamburg you could strike up conversation with pretty much anyone and find something of interest to jib about. At several points strangers would change the course of their evening just to steer our slightly cat-herdesque group of geeks towards fun.

Thanks

Huge thanks to everyone who helped make this an excellent weekend, especially Marcel and Patrick.

1. and, for the record, I understand why such a mode would be imperfect and not aligned with Chef’s overall philosophy
2. but no, I didn’t quite last the whole night!