The reference documentation has examples in Perl and PHP which got me so far, but I’m most comfortable in Ruby now, and was happy to find ¹ this pointer to using the soap4r library.

Chief bonus here is the wsdl2ruby.rb tool that’ll transform the WSDL data into Ruby objects with heirarchy, attribute accessors and everything else to make operating the API really comfortable. If your WSDL is a moving target during development it’ll even do this on the fly.

This meant getting the scripting done to configure the ZXTMs was pretty straightforward, without any faffing with the underlying access mech. Refreshing!

1. I was going to paste example code, but this’ll do

ssh -Dlocalhost:1080 -C host.example.com

You can now aim your browser’s SOCKS settting at localhost:1080. Bonus points for using a locally-hosted PAC file to determine which traffic is routed via the proxy:

function FindProxyForURL(url, host) {
  if (shExpMatch(host, "*.example.com"))
    return "SOCKS 127.0.0.1:1080"
  else
    return "DIRECT";
}

For getting sight of servers behind a firewall without remote proxies (hello Privoxy) and continual browser fiddling this is ideal. Thanks Murb!

Particulary cool is that they will let you at the source of your network. You can’t then wander off and run it elsewhere, it sits atop of their core web framework that you can’t see. With the source (which is encouragingly well written PHP) you can do perform all manner of modifications to bend the template network to your will.

Ning provide their own authentication system, and there’s no API to hook in someone else’s which is a hassle if you’re trying to build a social network alongside another site: maintaining a login for each site is going to annoy the users and be a nightmare to manage. Nil points.

One solution is to build a single sign-on system around what Ning already provide, which is robust, tested and better presented than anything we could achieve! I’d sketched out such a system for a proposal years ago but never had the opportunity to build it. A current project provided the perfect excuse to try this out.

Layout

Operation
The numbers here match those in the diagram.

1. User visits www.example.com (which for us happens to be a Rails) site.
2. The PageController notices the browser supplied no cookie and must therefore log on to Ning before proceeding. The site returns a redirect to Ning’s authentication page.
3. The browser follows this redirect to Ning.
4. Ning authenticates the user. The login code is modified from the original Ning behaviour. Here, it issues a redirect to sso.example.com with some parameters, including the user’s Ning ID and a salted hash to prevent spoofing. In this redirect Ning sets a range of cookies, including one that identifies the user to Ning.
5. The browser follows this redirect to the SSO server (which happens to be a Merb site – I wanted to try it out!)
6. The SSO app checks the provided hash against its own idea of what it should be. Assuming they match it considers the Ning ID to be valid. Finally, the SSO app issues another redirect along with cookies for just .example.com. This cookie identifies the user to the external site
7. In passing, the SSO app keeps track of users it has seen, and if this is a new user it will make an API request to Ning to fetch that user’s profile data and create a matching user on the www.example.com site.
8. The browser follows this last redirect back to Ning. It could be back to www.example.com or Ning depending on the situation.
9. Ning knows who the user is by merit of its cookies.
10. As does our external site.

Notes

• Having the SSO app different from the www.example.com site is perhaps a bit baroque. It works because the SSO app issues a cookie for .example.com which the browser will offer to both sso.example.com and www.example.com. In favour of this approach is that the SSO app is simple, and thus less likely to fail during development iterations than the nascent www.example.com site. A failure in the SSO app is bad, becaise it locks people out of Ning too. That Rails and Merb can share session data (and a database) is cool.
• The SSO app’s fetching of Ning profile data allows us to maintain a local version of a user’s profile to avoid the need to fetch it from Ning every time we need. There’s a nuance of Ning API authentication that meant I had to write a custom widget (Ning site component) to handle that.

Specifics
I was about to paste bits of modified Ning code, but I’ll need to check if I can under the various blurbs you agree too when signing up! Anyhow, the information above should help answer some of the requests for details from the Ning developers forum.

• There’s no invention here, I believe this setup is very common.
• High availability isn’t the same thing as load balanced. There is nothing here to intelligently shared load across the frontend servers, and one backend server is essentially idle all the time.
• This cluster is built with a bunch of open-source software on non-fancy kit. As such it doesn’t have the enormous capacity of clusters built upon commercial shared-storage products, SAN kit, layer 7 web switches etc. Its ambition is to run a few busy Rails sites well whilst coping with hardware failure gracefully.

Layout

Operation

• Web traffic is spread across the managed frontend interfaces by multiple A records in the DNS.
• Wackamole uses a Spread messaging network to ensure these multiple A record IPs are always present across the frontend. It achieves this by managing the hosts’ interfaces when it detects hosts joining or leaving the cluster.
• A pair of MySQL servers run in master:master configuration on the backend hosts
• The backend hosts use DRBD to maintain a mirrored block device between them.
• These block devices back a NFS filesystem.
• Heartbeat runs on the backend hosts to do several tasks:
  1. Manage which host is the DRBD primary and therefore can be written to.
  2. Manage which host has the DRBD filesystem mounted and exported with NFS.
  3. Manage the IP through which the frontend mounts the filesystem and talks to MySQL.
• With all this in place, Nginx accepts web connections and serves static assets off the NFS mount and passess other requests to Mongrel, a HTTP server that’s well suited to running a Rails instance.

Notes

• One of the main hazards of MySQL master:master setups is primary key collision if an INSERT occurs on both hosts at once. We avoid that here by letting Hearbeat manage the IP that the frontends connect to.
• I’ve built two of these clusters to date. The second one is now four servers wide on the frontend.

Future work

• DRBD can now run in dual-primary mode, allowing both hosts to accept writes. This makes it a candidate for filesystems like GFS that use shared storage to present a filesystem that can be written to on multiple hosts. More here.
• To add some load balancing I’m considering using HAProxy or LVS to actively distribute traffic across the frontends.
• HA aside, there’s also some cool things like evented Mongrel that it would be interesting to try.

Extra credit for assigning a keyword (eg ‘ktx’) in the bookmark properties, allowing you to just type, eg, ‘ktx 1234′ in the Location bar to achieve the same.

This is a rework of a similar hack for a much older ticketer, PTS, which is amazingly still in use at one of my previous workplaces. You can gauge its age by the fact that it was ported to PHP3 and was pretty open to most injection attacks!

I do however find myself forever doing stuff like
$ ls -tr /a/log/dir | tail
error_log.1129593600
error_log.1129680000
error_log.1129766400
$ tail -f error_log.1129766400 to diagnose a problem.

What would be cool was if rotatelogs linked, say, error_log.current to the, er, current log. If it used a hard link, you could probably use tail’s -F flag to seemlessly glide over rotations.

Update: I looked at the source; what immediately occurs to me is that both soft and hard linking aren’t particularly portable, and Apache runs on many different platforms. It’s probably more straightforward to do this with a shell script or similar.

PHP defaults to sending nocache headers, and this makes IE aggressively zap any temporary files that arise, even if they’re destined for another app locally.

We fixed this by setting PHP’s cache limiter knob on a per-download basis, although it can be done system-wide also. Here’s the page in the PHP manual.

Whilst investigating this, I encountered the ugly truth about IE’s mime-type handling. Seems fiddly!