If you’re using freebsd-update to upgrade FreeBSD and you run a custom kernel and temporarily using the generic kernel isn’t an option and it’s been ages since you’ve done this, then welcome!

$ sudo freebsd-update upgrade -r 9.1-RELEASE

$ sudo freebsd-update install
$ sudo reboot


The kernel-toolchain bit is the important one. In the Old Days¹ of buildworld the new kernel is built with the new world’s compiler, linker etc instead of what’s currently installed. Going directly to buildkernel during the freebsd-update dance means the custom kernel builds with the toolchain that’s about to be replaced.

Trying to build the new kernel with the old toolchain results in

• warnings-as-errors in random modules
• old kernel linker unable to parse new kernel link data

and no kernel.

1. not really, but binary updating is way nicer

Because every Saturday morning needs some XML, right?

I’ve often wondered about how to do this without monkeying in Objective C. It’s doable in (ugh) Applescript with some property list editing. Whimsical XKCD handler such that xkcd://627 behaves like http://xckd.com/627:


on open location uri
  set delimiter to AppleScript's text item delimiters
  set AppleScript's text item delimiters to "//"
  set comic to text item 2 of uri
  set page to "http://xkcd.com/" & comic
  tell application "Safari"
    activate
    open location page
  end tell
  set AppleScript's text item delimiters to delimiter
end open location


Save as an Application rather than a Script. Open its info.plist and add

 <key>CFBundleIdentifier</key>
 <string>uk.co.zomo.xkcd-handler</string>
 <key>CFBundleURLTypes</key>
 <array>
  <dict>
   <key>CFBundleURLName</key>
   <string>XKCD helper</string>
   <key>CFBundleURLSchemes</key>
   <array>
    <string>xkcd</string>
   </array>
  </dict>
 </array>


To nudge OS X to notice changed a plist:

/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -v -f ~/Library/Scripts/xkcd-handler.app


Links:

• http://applescript123.com/linktrigger/
• http://stackoverflow.com/questions/8981749/how-do-i-refresh-application-metadata-url-handlers-web-browsers-after-insta
• http://www.rubicode.com/Software/RCDefaultApp/

• Set open files to 65535 via ulimit -n
• Recompile httperf with /usr/include/bits/typesizes.h‘s __FD_SETSIZE dialled up to 65535¹
• Twiddle the TCP stack for faster socket recycling²³ :
  
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse


1. http://gom-jabbar.org/articles/2009/02/04/httperf-and-file-descriptors
2. http://www.speedguide.net/articles/linux-tweaking-121
3. http://serverfault.com/questions/331513/httperf-hangs-when-using-hog

• Moving virtual machines from Xen to KVM
• The storage for both is under LVM
• Under Xen the storage for each VM isn’t partitioned as a disk, it’s just a filesystem. GRUB under KVM will need to see a boot sector and a partition table.
• Under Xen the storage for each VM doesn’t contain a kernel. GRUB under KVM will need one to boot the VM.

Sequence

• Measure disks on Xen server, create on KVM server. Create the new root disk 10MB larger than its counterpart.¹
  
kyero@xen03 ~ $ sudo lvscan
ACTIVE '/dev/vg4/mail-root' [20.00 GB] inherit
ACTIVE '/dev/vg4/mail-swap' [1.00 GB] inherit


[kyero@vm02 ~]$ sudo lvcreate -L20490M -n mail-root vg4
[kyero@vm02 ~]$ sudo lvcreate -L2G -n mail-swap vg4

• Create a boot disk on the KVM server. We make it 4 cylinders long, roughly 32MB.²
  
[kyero@vm02 ~]$ sudo dd if=/dev/zero of=/var/boot-images/mail-boot bs=32901120 count=1

• Partition the boot disk with one partition starting at cylinder zero
  
  [kyero@vm02 ~]$ sudo sfdisk /var/boot-images/mail-boot
  last_lba(): I don't know how to handle files with mode 81a4
  Warning: /var/boot-images/mail-boot is not a block device
  Disk /var/boot-images/mail-boot: cannot get geometry

  Disk /var/boot-images/mail-boot: 4 cylinders, 255 heads, 63 sectors/track

  sfdisk: ERROR: sector 0 does not have an msdos signature
  /var/boot-images/mail-boot: unrecognized partition table type
  Old situation:
  No partitions found
  Input in the following format; absent fields get a default value.
  <start> <size> <type [E,S,L,X,hex]> <bootable [-,*]> <c,h,s> <c,h,s>\nUsually you only need to specify <start> and <size> (and perhaps <type>).

  /var/boot-images/mail-boot1 :0
  /var/boot-images/mail-boot1 0+ 3 4- 32129+ 83 Linux
  /var/boot-images/mail-boot2 :
  /var/boot-images/mail-boot2 0 - 0 0 0 Empty
  /var/boot-images/mail-boot3 :
  /var/boot-images/mail-boot3 0 - 0 0 0 Empty
  /var/boot-images/mail-boot4 :
  /var/boot-images/mail-boot4 0 - 0 0 0 Empty
  New situation:
  Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

  Device Boot Start End #cyls #blocks Id System
  /var/boot-images/mail-boot1 0+ 3 4- 32129+ 83 Linux
  /var/boot-images/mail-boot2 0 - 0 0 0 Empty
  /var/boot-images/mail-boot3 0 - 0 0 0 Empty
  /var/boot-images/mail-boot4 0 - 0 0 0 Empty
  Warning: no primary partition is marked bootable (active)
  This does not matter for LILO, but the DOS MBR will not boot this disk.
  Do you want to write this to disk? [ynq] y
  Successfully wrote the new partition table

  Re-reading the partition table ...
  BLKRRPART: Inappropriate ioctl for device

  If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
  to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1
  (See fdisk(8).)

• Format and mount the partition
  
  [kyero@vm02 ~]$ sudo kpartx -av /var/boot-images/mail-boot
  add map loop1p1 : 0 64259 linear /dev/loop1 1

  [kyero@vm02 ~]$ sudo mke2fs /dev/mapper/loop1p1
  mke2fs 1.39 (29-May-2006)
  Filesystem label=
  OS type: Linux
  Block size=1024 (log=0)
  Fragment size=1024 (log=0)
  8032 inodes, 32128 blocks
  1606 blocks (5.00%) reserved for the super user
  First data block=1
  Maximum filesystem blocks=33030144
  4 block groups
  8192 blocks per group, 8192 fragments per group
  2008 inodes per group
  Superblock backups stored on blocks:
  8193, 24577

  Writing inode tables: done
  Writing superblocks and filesystem accounting information: done

  This filesystem will be automatically checked every 34 mounts or
  180 days, whichever comes first. Use tune2fs -c or -i to override.

  [kyero@vm02 ~]$ sudo mount /dev/mapper/loop1p1 /mnt/boot/

  [kyero@vm02 ~]$ df -h /mnt/boot
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/loop1p1 31M 389K 29M 2% /mnt/boot


• Copy a kernel to it, configure GRUB’s notion of the root device and install the boot sector.
  
  [kyero@vm02 ~]$ sudo rsync -a /boot/ /mnt/boot/
  [kyero@vm02 ~]$ sudo vi /mnt/boot/grub/menu.lst
  [kyero@vm02 ~]$ sudo touch /mnt/boot/kvm-boot-image
  [kyero@vm02 ~]$ sudo grub --device-map=/dev/null

  GNU GRUB version 0.97 (640K lower / 3072K upper memory)

  [ Minimal BASH-like line editing is supported. For the first word, TAB
  lists possible command completions. Anywhere else TAB lists the possible
  completions of a device/filename.]
  grub> device (hd0) /var/boot-images/mail-boot
  device (hd0) /var/boot-images/mail-boot
  grub> find /kvm-boot-image
  find /kvm-boot-image
  (hd0,0)
  grub> root (hd0,0)
  root (hd0,0)
  Filesystem type is ext2fs, partition type 0x83
  grub> setup (hd0)
  setup (hd0)
  Checking if "/boot/grub/stage1" exists... no
  Checking if "/grub/stage1" exists... yes
  Checking if "/grub/stage2" exists... yes
  Checking if "/grub/e2fs_stage1_5" exists... yes
  Running "embed /grub/e2fs_stage1_5 (hd0)"... failed (this is not fatal)
  Running "embed /grub/e2fs_stage1_5 (hd0,0)"... failed (this is not fatal)
  Done.
  grub> quit

  [kyero@vm02 ~]$ sudo umount /mnt/boot
  [kyero@vm02 ~]$ sudo kpartx -d /var/boot-images/mail-boot
  loop deleted : /dev/loop1

• Populate the new root disk with one cylinder of zeros and a copy of the old root disk.
  The first cylinder will contain the MBR and not much else. The padding means it’s easy to align the old root disk at cylinder one.
  Use pv so we can see the copy progress.
  The dd-lvm-out script is mostly just a sudo-blessed wrapper.
  [kyero@vm02 ~]$ ( dd if=/dev/zero bs=8225280 count=1 ; ssh xen03.local -l kyero sudo /data/system/bin/dd-lvm-out.sh mail-root ) | pv -e -r -s 21474836480 | sudo dd bs=1048577 of=/dev/vg4/mail-root
1+0 records in
1+0 records out
8225280 bytes (8.2 MB) copied, 0.03395 seconds, 242 MB/s
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB) copied, 611.215 s, 35.1 MB/s
0+662630 records in
0+662630 records out
21483061795 bytes (21 GB) copied, 615.249 seconds, 34.9 MB/s

• Partition the new root disk with a single partition starting at cylinder one.
  [kyero@vm02 ~]$ sudo sfdisk /dev/vg4/mail-root
  Checking that no-one is using this disk right now ...
  BLKRRPART: Invalid argument
  OK

  Disk /dev/vg4/mail-root: 2614 cylinders, 255 heads, 63 sectors/track

  sfdisk: ERROR: sector 0 does not have an msdos signature
  /dev/vg4/mail-root: unrecognized partition table type
  Old situation:
  No partitions found
  Input in the following format; absent fields get a default value.
  <start> <size> <type [E,S,L,X,hex]> <bootable [-,*]> <c,h,s> <c,h,s>\nUsually you only need to specify <start> and <size> (and perhaps <type>).

  /dev/vg4/mail-root1 :1
  /dev/vg4/mail-root1 1 2613 2613 20988922+ 83 Linux
  /dev/vg4/mail-root2 :
  /dev/vg4/mail-root2 0+ 0 1- 8032 83 Linux
  /dev/vg4/mail-root3 :
  /dev/vg4/mail-root3 0 - 0 0 0 Empty
  /dev/vg4/mail-root4 :
  /dev/vg4/mail-root4 0 - 0 0 0 Empty
  New situation:
  Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

  Device Boot Start End #cyls #blocks Id System
  /dev/vg4/mail-root1 1 2613 2613 20988922+ 83 Linux
  /dev/vg4/mail-root2 0+ 0 1- 8032 83 Linux
  /dev/vg4/mail-root3 0 - 0 0 0 Empty
  /dev/vg4/mail-root4 0 - 0 0 0 Empty
  Warning: no primary partition is marked bootable (active)
  This does not matter for LILO, but the DOS MBR will not boot this disk.
  Do you want to write this to disk? [ynq] y
  Successfully wrote the new partition table

  Re-reading the partition table ...
  BLKRRPART: Invalid argument

  If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
  to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1
  (See fdisk(8).)

• Mount the new root filesystem
  [kyero@vm02 ~]$ sudo kpartx -av /dev/vg4/mail-root
  add map mail-root1 : 0 41977845 linear /dev/vg4/mail-root 16065
  add map mail-root2 : 0 16064 linear /dev/vg4/mail-root 1
  [kyero@vm02 ~]$ sudo e2fsck /dev/mapper/mail-root1
  e2fsck 1.39 (29-May-2006)
  /dev/mapper/mail-root1: clean, 435213/1310720 files, 4086961/5242880 blocks

  [kyero@vm02 ~]$ sudo mount /dev/mapper/mail-root1 /mnt/tmp


• Add the new kernel’s modules, tweak filesystem mounts, disable swap, check console.
  
  [kyero@vm02 ~]$ sudo rsync -a /lib/modules/ /mnt/tmp/lib/modules/

  [kyero@vm02 ~]$ sudo vi /mnt/tmp/etc/fstab

  [kyero@vm02 ~]$ sudo vi /mnt/tmp/etc/inittab


• Create the KVM guest, assigning the boot, root and swap storage
  [kyero@vm02 ~]$ sudo virt-install -n mail -r 1024 --vcpus 1 --os-type linux --os-variant generic26 --accelerate --disk path=/var/boot-images/mail-boot --disk path=/dev/mapper/vg4-mail--root,bus=ide,format=raw --disk path=/dev/mapper/vg4-mail--swap,bus=ide,format=raw --network bridge:br0 --network bridge:br1 --import --noreboot
  Starting install...
  Guest installation complete... you can restart your domain
  by running 'virsh start mail'

  [kyero@vm02 ~]$ sudo virsh dumpxml mail
...
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/var/boot-images/mail-boot'/>
<target dev='hda' bus='ide'/>
<address type='drive' controller='0' bus='0' unit='0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='/dev/mapper/vg4-mail--root'/>
<target dev='hdb' bus='ide'/>
<address type='drive' controller='0' bus='0' unit='1'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='/dev/mapper/vg4-mail--swap'/>
<target dev='hdd' bus='ide'/>
<address type='drive' controller='0' bus='1' unit='1'/>
</disk>
...


• Boot the KVM guest
  
[kyero@vm02 ~]$ sudo virsh start mail
[kyero@vm02 ~]$ sudo virsh autostart mail
[kyero@vm02 ~]$ sudo virsh console mail

• Log into KVM guest, partition /dev/hdd and configure it as swap, update /etc/fstab.

1. Later we align the original root filesystem at cylinder 1 on this new disk. A LVM cylinder is roughly 8MB
2. File backed disk is used rather than LVM owing to grief with GRUB and LVM (Error 22: No such partition)

This log directory was allowed to grow indefinitely (ahem) until I trimmed it this morning. Now it is mostly empty (1.7MB), but the directory entry is still huge (at 1.6MB, compared to the default 4kB). I had a vague idea of why this happens but realised I was unclear on the specifics so it was time for The Learning.

On traditional filesystems (here, FFS on FreeBSD and ext2 on GNU/Linux) directories look much like files, just with special content. They have an inode that points to either blocks on disk or to indirection blocks that themselves point to blocks (or to other indirection blocks, up to three levels of indirection). The contents of these directory blocks are reasonably straightforward: they’re a sequence of dirent structures that generally contain a filename and that file’s inode number along with some small housekeeping data.

As a directory fills this sequence of dirents grows to fill all the direct blocks, then starts on the indirect blocks and so on. Each time the sequence crosses a block boundary the directory’s own size, rather than the total of the files it contains, grows by another block. Since filenames have variable length a single block will contain a variable number of dirents.

So far so good. What happens you delete a file from a directory? Other than freeing the actual file’s inode and respective blocks something must happen to its dirent entry in the directory’s data. What doesn’t happen is that the dirent is removed and all the ones after it get bumped towards the front of the list, which would lead to fewer overall directory blocks given enough deletions and ultimately smaller directories.

Why this doesn’t happen partly relates to the guarantees that telldir and friends make to programs with the directory open: a file’s relative position in the sequence of dirent will not change for any process that has the directory open. The deletion process can’t compact the directory itself without breaking that guarantee. Both FFS and ext2 blank out the file’s inode in the dirent record (ext2 with zero, FFS with a special value). Subsequent directory operations will ignore this blanked dirent, and processes which knew its sequence number via telldir will error if they attempt operations on it.

To recover the disk space used by a directory that has ballooned and then contracted its simplest to simply delete it entirely and recreate it. ext2 has an option (-D) in its filesystem check to compact the directory offline.

Interestingly this behaviour isn’t present on all filesystems, particularly those that use indexes instead of special files for directories: for example ZFS and HFS+. They can remove a file’s entry in the directory index yet still keep the remaining files appearing at their same positions by informed updates to the index.

Here is these four filesystem’s compared by

• Creating an empty directory
• Filling it with a million files
• Removing all but one of those files

The comparison shows how on ext2 and FFS the directory itself remains at its peak size after the test files are removed, whereas ZFS and HFS+ contract.

ext2

[build@centos-x86-64-build tmp]$ df -ih .
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sdb 1.3M 11 1.3M 1% /mnt/tmp
[build@centos-x86-64-build tmp]$ ls -ild .
2 drwxr-xr-x. 3 build build 4096 Mar 19 08:58 .

[build@centos-x86-64-build tmp]$ seq --format 'f%06g' 0 999998 | xargs rm
[build@centos-x86-64-build tmp]$ df -ih .
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sdb 1.3M 12 1.3M 1% /mnt/tmp
[build@centos-x86-64-build tmp]$ ls -ild .
2 drwxr-xr-x. 3 build build 23224320 Mar 19 09:02 .


FFS

[lemon@zest ~/tmp]$ df -ih .
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/mirror/gm0s1a 19G 10G 7.5G 58% 587k 2.1M 22% /
[lemon@zest ~/tmp]$ ls -ild .
521826 drwxr-xr-x 2 lemon lemon 512 Mar 19 14:50 .

[lemon@zest ~/tmp]$ jot -w "f%06d" 999999 0 | xargs rm
[lemon@zest ~/tmp]$ df -ih .
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/mirror/gm0s1a 19G 10G 7.5G 58% 587k 2.1M 22% /
[lemon@zest ~/tmp]$ ls -ild .
521826 drwxr-xr-x 2 lemon lemon 16000512 Mar 20 01:44 .


ZFS

[lemon@zest /mnt/ark/tmp]$ df -ih .
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
ark 207G 686M 206G 0% 58k 432M 0% /mnt/ark
[lemon@zest /mnt/ark/tmp]$ ls -ild
5 drwxrwsr-x 2 root wheel 2 Mar 20 06:18 .

[lemon@zest /mnt/ark/tmp]$ jot -w "f%06d" 999999 0 | xargs rm
[lemon@zest /mnt/ark/tmp]$ df -ih .
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
ark 207G 727M 206G 0% 58k 432M 0% /mnt/ark
[lemon@zest /mnt/ark/tmp]$ ls -ild
5 drwxrwsr-x 2 root wheel 3 Mar 20 09:27 .


HFS+


[lemon@further tmp] 0 $ df -ih .
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk1 233Gi 185Gi 48Gi 80% 48506927 12484688 80% /
[lemon@further tmp] 0 $ ls -ild .
12149971 drwxr-xr-x 2 lemon staff 68 19 Mar 16:06 .

[lemon@further tmp] 0 $ jot -w "f%06d" 999999 0 | xargs rm
[lemon@further tmp] 0 $ df -ih .
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk1 233Gi 185Gi 48Gi 80% 48524492 12467123 80% /
[lemon@further tmp] 0 $ ls -ild .
12149971 drwxr-xr-x 2 lemon staff 102 19 Mar 19:08 .


I cut my UNIX teeth at a cryptoanarchist shop whose culture of paranoia makes me wary of Apple’s own firewall with its emphasis on letting all the hot shinyness Just Work rather than being overly fussy about inbound connections. Furthermore, with IPv6 tunnels like aiccu, you aren’t behind the warm fuzzy comfort of NAT, you’re just there on the net with all the fun that entails. So, worth having some extra protection I reckon.

10.6 (and earlier) came with ipfw, a packet filter that’s knocked around the BSD world for some time. It works but isn’t overly featuresome (for example, it doesn’t support NAT in-kernel, so you monkey about passing packets to an external daemon). But it was Good Enough for an end system so I supplemented the system’s “Application Firewall” with an additional ipfw ruleset to give an approximation of safety when out and about, and way more permissive when on networks I trust.

On 10.6 I used WaterRoof to sort-of manage the ipfw rules: in that I only really used its launchd loader and would hack on the rules by hand. In the spirit of decrufting I figured I’d sort that myself and went to remind myself how to load ipfw rules enmasse. I noticed at the top of man page:

NAME
ipfw -- IP firewall and traffic shaper control program (DEPRECATED)
...
DESCRIPTION
Note that use of this utility is DEPRECATED. Please use pfctl(8) instead


Deprecated? Use pfctl instead? Good news everyone – OS X 10.7 now comes with pf, another BSD packet filter that I’ve chosen of ipfw on BSD hosts for years off the back of its featureset (native NAT, state syncing between failover firewall pairs, traffic queing…).

Anyway, the point of this post is to point out a few things I noticed which are intriguing. Firstly, pf is not enabled by default. Further, Apple have added some moving parts around how it is enabled. From /etc/pf.conf:

# This file contains the main ruleset, which gets automatically loaded
# at startup. PF will not be automatically enabled, however. Instead,
# each component which utilizes PF is responsible for enabling and disabling
# PF via -E and -X as documented in pfctl(8). That will ensure that PF
# is disabled only when the last enable reference is released.

These two flags, -E and -X, are absent from pf on BSD. Here’s how they’re documented on OS X:

-E Enable the packet filter and increment the pf enable reference count.
-X token
Release the pf enable reference represented by the token passed.

This suggests that different system components might choose to enable and disable pf, and this is the mechanism to coordinate that. There’s a clue about which components in /etc/pf.anchors/com.apple, which is loaded by the main /etc/pf.conf. It defines additional rule anchors:

anchor "100.InternetSharing/*"
anchor "200.AirDrop/*"
anchor "250.ApplicationFirewall/*"


Interestingly, this host’s ApplicationFirewall has a bunch of entries in when viewed in the Preferences GUI, yet the pf anchor of the same name is empty (and pf was disabled when I started out):

$ sudo pfctl -a com.apple/250.ApplicationFirewall -s rules
Password:
No ALTQ support in kernel
ALTQ related functions disabled

so I’m unsure what the status of this mechanism is. I’ve not had occasion to use AirDrop or connection sharing, but would be curious to see if either use these anchors and enable pf temporarily.

Finally, what’s the token that’s passed to -X? You can ask pfctl for the current tokens:

$ sudo pfctl -s References
No ALTQ support in kernel
ALTQ related functions disabled
TOKENS:
PID Process Name TOKEN TIMESTAMP
17013 pfctl 18446743524308110600 0 days 01:05:50

I enabled pf with pfctl, so that makes sense. When I did so it didn’t inform me of the token, but I suppose an enabling process would spelunk the token shortly after enabling pf by merit of its name and PID and pass it back when it’s finished with pf.

Now, on with the actual job of ruleset writing and puzzling out the launchd voodoo required to enable it at boot.

Minor whinge: Apple could do with updating /etc/protocols:

# $FreeBSD: src/etc/protocols,v 1.14 2000/09/24 11:20:27 asmodai Exp $

Why whinge? It doesn’t know icmp6 is a valid alias for ipv6-icmp. Yep, minor.

I then found then whenever I sent mail using authenticated SMTP my mail server would lock up with saslauthd chewing the CPU. This authentication daemon is the glue between the MTA (Exim) and the IMAP server (Courier) – it logs into the IMAP service to test the SMTP user’s credentials. This little kink of indirection comes about because the IMAP daemon is downstream from the Exim host, in a BSD jail host, so its own authentication mechanisms aren’t visible to the MTA.

My new mail password contained a double-quote mark, which made me wonder if the password wasn’t being quoted properly. Testing a bit with openssl:

$ openssl s_client -starttls smtp -connect localhost:25
CONNECTED(00000003)
---
250 HELP
EHLO localhost
250-svc9.zomo.co.uk Hello localhost [127.0.0.1]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250 HELP
AUTH PLAIN AGZvbwAi < -- this is Base64 for username foo, password "

Compiling a -g debug variant of the daemon and aiming gdb at it:

$ sudo gdb /usr/local/sbin/saslauthd-debug 97103
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
...
(gdb) bt
#0 0x284250d1 in strchr () from /lib/libc.so.7
#1 0x0804a823 in qstring ()
#2 0x0804ac45 in auth_rimap ()
#3 0x0804f8e3 in do_auth ()
#4 0x0804e1f4 in do_request ()
#5 0x0804e53b in ipc_loop ()
#6 0x0805018d in main ()


What’s qstring()? It’s a function for escaping the quote marks in strings passed to the IMAP daemon. Turns out count-the-quotemark logic wasn’t properly advancing along the string, so it would sit there spinning forever.

Trivial patch¹ fixes:

$ openssl s_client -starttls smtp -connect localhost:25
CONNECTED(00000003)
---
250 HELP
EHLO localhost
250-svc9.zomo.co.uk Hello localhost [127.0.0.1]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250 HELP
AUTH PLAIN AGZvbwAi
535 Incorrect authentication data


Better :)

1. Gist if it’s not inlined above

Recently a client was receiving complaints that their busy server hosting both their WordPress sites and their OpenX¹ banner delivery was underperforming. Specifically, sites including their banners were seeing page loads hang on them. If you’re in the business of selling banners this is bad news. There were reports of the WordPress sites being slow too, but mostly from administrators² rather than site visitors.

I sorted out a bunch of request amplification issues but still things still weren’t right, so I added a second server to help out. Instead of just chucking the combined traffic at both servers I used HAProxy to separate out the traffic to each, with a view to adding more OpenX servers as necessary.

Here’s what HAProxy’s stats had to say after some time running the sites split:

Some of these I found unsurprising – WordPress serves a higher volume of data, it is content heavy compared to banner delivery and related click handling. Conversely the inbound data volume for OpenX is up because it’s loaded with click information.

What’s interesting is that the WordPress sites have a higher maximum concurrent session count, yet the total sessions is far higher for the OpenX banners. This illustrates the benefit of separating out different server loads: one server is churning away pushing out fat content and even when heavily cached this burns enough resource that requests get queued and gum up, whilst another is fielding quick-in quick-out requests. When it’s not contending with its laggard sibling it can get on with its business unhindered.

Ultimately the visibility HAProxy affords beats an Apache scoreboard when that Apache is fielding two differently focused workloads.

1. advertising is a necessary evil, right?
2. and that turned out to be a pagination issue