Recently a client was receiving complaints that their busy server hosting both their WordPress sites and their OpenX¹ banner delivery was underperforming. Specifically, sites including their banners were seeing page loads hang on them. If you’re in the business of selling banners this is bad news. There were reports of the WordPress sites being slow too, but mostly from administrators² rather than site visitors.

I sorted out a bunch of request amplification issues but still things still weren’t right, so I added a second server to help out. Instead of just chucking the combined traffic at both servers I used HAProxy to separate out the traffic to each, with a view to adding more OpenX servers as necessary.

Here’s what HAProxy’s stats had to say after some time running the sites split:

Some of these I found unsurprising – WordPress serves a higher volume of data, it is content heavy compared to banner delivery and related click handling. Conversely the inbound data volume for OpenX is up because it’s loaded with click information.

What’s interesting is that the WordPress sites have a higher maximum concurrent session count, yet the total sessions is far higher for the OpenX banners. This illustrates the benefit of separating out different server loads: one server is churning away pushing out fat content and even when heavily cached this burns enough resource that requests get queued and gum up, whilst another is fielding quick-in quick-out requests. When it’s not contending with its laggard sibling it can get on with its business unhindered.

Ultimately the visibility HAProxy affords beats an Apache scoreboard when that Apache is fielding two differently focused workloads.

1. advertising is a necessary evil, right?
2. and that turned out to be a pagination issue

“Integration points are the number-one killer of systems” – Release It!, Michael Nygard

Last week I had two different web systems fail in a similar way. One was a single box running two busy WordPress sites, another was a largish multi-tier publishing cluster. Both dropped off air because they didn’t handle the failure of a remote system very well. In particular, both had the webserver waiting on a HTTP request to a foreign site to complete before returning a page to the client.

The architecture issues with this kind of request amplification are reasonably clear, and can sometimes be avoided. Data ingest from other systems can often execute asynchronously in another process. The cluster mentioned has a set of machines dedicated to just this, distributing the data they retrieve from remote systems via database and shared filesystem ready for the webservers to use when building pages.

However, in the situations where it is necessary for a webserver to dial out to another system at request time then it’s worth being really paranoid about how that outbound request works and untrusting of the reply. The particular problem here was lack of timeouts.

Here, both systems were running mpm_prefork Apache. Both were serving a request that made a HTTP call somewhere else before returning the page to the client¹. Last week the remote sites that both these systems contact had outages, leaving Apache waiting for the reply up until the request timed out.

However, with no timeout configured that wait is effectively infinite. Long enough for these hanging Apache processes to consume all the available slots of the webserver, resulting an interesting set of observations:

• New TCP connections to the server just hang.
• Server load is low, because the load measures runnable processes (and, on Linux, uninterruptible sleeping processes), and these hanging processes are just sleeping, waiting on poll(2) or select(2).
• No errors are logged by Apache or the application code because they haven’t failed yet.²

Here’s some vmstat during such a wedge:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 140580 56528 58168 516184 0 0 0 0 1026 78 0 0 100 0 0
0 0 140580 56528 58168 516184 0 0 0 0 1032 80 0 0 100 0 0
0 0 140580 56528 58168 516184 0 0 0 0 1031 78 0 0 100 0 0

“It was quiet. Too quiet”. Hook up gdb and you can see what’s going on:

(gdb) bt
#0 0x00002aafffb7d14f in poll () from /lib64/libc.so.6
#1 0x00002ab0090b3930 in Curl_select () from /usr/lib64/libcurl.so.3
#2 0x00002ab0090ac4bc in Curl_perform () from /usr/lib64/libcurl.so.3
#3 0x00002ab00850591b in zif_curl_exec () from /etc/httpd/modules/libphp5.so
#4 0x00002ab0086663b2 in ?? () from /etc/httpd/modules/libphp5.so
#5 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#6 0x00002ab00865dc09 in ?? () from /etc/httpd/modules/libphp5.so
#7 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#8 0x00002ab008688750 in ?? () from /etc/httpd/modules/libphp5.so
#9 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#10 0x00002ab00865dc09 in ?? () from /etc/httpd/modules/libphp5.so
#11 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#12 0x00002ab00865c11e in ?? () from /etc/httpd/modules/libphp5.so
#13 0x00002ab00865651c in execute () from /etc/httpd/modules/libphp5.so
#14 0x00002ab0086395de in zend_execute_scripts () from /etc/httpd/modules/libphp5.so
#15 0x00002ab0085fe697 in php_execute_script () from /etc/httpd/modules/libphp5.so
#16 0x00002ab0086b6ad6 in ?? () from /etc/httpd/modules/libphp5.so
#17 0x00002aaffdbb7a4a in ap_run_handler ()
#18 0x00002aaffdbbaed8 in ap_invoke_handler ()
#19 0x00002aaffdbc578a in ap_internal_redirect ()
#20 0x00002ab0069ffbc0 in ap_make_dirstr_parent () from /etc/httpd/modules/mod_rewrite.so
#21 0x00002aaffdbb7a4a in ap_run_handler ()
#22 0x00002aaffdbbaed8 in ap_invoke_handler ()
#23 0x00002aaffdbc5938 in ap_process_request ()
#24 0x00002aaffdbc2b70 in ?? ()
#25 0x00002aaffdbbecd2 in ap_run_process_connection ()
#26 0x00002aaffdbc9789 in ?? ()
#27 0x00002aaffdbc9a1a in ?? ()
#28 0x00002aaffdbc9ad0 in ?? ()
#29 0x00002aaffdbca7bb in ap_mpm_run ()
#30 0x00002aaffdba4e48 in main ()


Another kink is that a graceful restart of Apache, via SIGUSR1, doesn’t work since a graceful restart waits for a request to finish – and these ones aren’t finishing. apachectl or service(8) scripts will exit but the hung processes remain. These long running wedged processes are also visible in ps:

apache 580 0.0 0.3 30332 5492 ? S Jan02 0:00
/usr/sbin/httpd

and can be matched up with the hung connections via netstat -anp

tcp 0 0 172.18.74.113:50129 192.0.32.10:80
ESTABLISHED 580/httpd


This is a mostly a long winded way of pointing out the importance of timeouts on code that executes during a request, particularly when doing something non-local. Fail fast!

To keep the WordPress sites running I ran through the code adding CURLOPT_CONNECTTIMEOUT, CURLOPT_TIMEOUT settings to all the cURL calls I could find, and the development team took care of the code running on the cluster. Both have been stable since.

There are plenty of other pitfalls in these integration points between systems. Michael Nygard’s book, linked at top, makes a good survey of them alongside other stability antipatterns in complex systems. It’s a recommended read.

1. One of the WordPress sites was doing this in every page’s footer. And not caching the result. Better still, it was contacting the other WordPress site on the same server. This almost guarantees the request won’t return when the shared webserver gets busy since there may be no spare server slots to accept the second request. Ouch.
2. If you don’t run Apache hot, then you’d see a “I’ve reached MaxClients” message in the error log, but not much else.

The reference documentation has examples in Perl and PHP which got me so far, but I’m most comfortable in Ruby now, and was happy to find 1 this pointer to using the soap4r library.

Chief bonus here is the wsdl2ruby.rb tool that’ll transform the WSDL data into Ruby objects with heirarchy, attribute accessors and everything else to make operating the API really comfortable. If your WSDL is a moving target during development it’ll even do this on the fly.

This meant getting the scripting done to configure the ZXTMs was pretty straightforward, without any faffing with the underlying access mech. Refreshing!

1. I was going to paste example code, but this’ll do

ssh -Dlocalhost:1080 -C host.example.com

You can now aim your browser’s SOCKS settting at localhost:1080. Bonus points for using a locally-hosted PAC file to determine which traffic is routed via the proxy:

function FindProxyForURL(url, host) {
  if (shExpMatch(host, "*.example.com"))
    return "SOCKS 127.0.0.1:1080"
  else
    return "DIRECT";
}

For getting sight of servers behind a firewall without remote proxies (hello Privoxy) and continual browser fiddling this is ideal. Thanks Murb!

Particulary cool is that they will let you at the source of your network. You can’t then wander off and run it elsewhere, it sits atop of their core web framework that you can’t see. With the source (which is encouragingly well written PHP) you can do perform all manner of modifications to bend the template network to your will.

Ning provide their own authentication system, and there’s no API to hook in someone else’s which is a hassle if you’re trying to build a social network alongside another site: maintaining a login for each site is going to annoy the users and be a nightmare to manage. Nil points.

One solution is to build a single sign-on system around what Ning already provide, which is robust, tested and better presented than anything we could achieve! I’d sketched out such a system for a proposal years ago but never had the opportunity to build it. A current project provided the perfect excuse to try this out.

Layout

Operation
The numbers here match those in the diagram.

1. User visits www.example.com (which for us happens to be a Rails) site.
2. The PageController notices the browser supplied no cookie and must therefore log on to Ning before proceeding. The site returns a redirect to Ning’s authentication page.
3. The browser follows this redirect to Ning.
4. Ning authenticates the user. The login code is modified from the original Ning behaviour. Here, it issues a redirect to sso.example.com with some parameters, including the user’s Ning ID and a salted hash to prevent spoofing. In this redirect Ning sets a range of cookies, including one that identifies the user to Ning.
5. The browser follows this redirect to the SSO server (which happens to be a Merb site – I wanted to try it out!)
6. The SSO app checks the provided hash against its own idea of what it should be. Assuming they match it considers the Ning ID to be valid. Finally, the SSO app issues another redirect along with cookies for just .example.com. This cookie identifies the user to the external site
7. In passing, the SSO app keeps track of users it has seen, and if this is a new user it will make an API request to Ning to fetch that user’s profile data and create a matching user on the www.example.com site.
8. The browser follows this last redirect back to Ning. It could be back to www.example.com or Ning depending on the situation.
9. Ning knows who the user is by merit of its cookies.
10. As does our external site.

Notes

• Having the SSO app different from the www.example.com site is perhaps a bit baroque. It works because the SSO app issues a cookie for .example.com which the browser will offer to both sso.example.com and www.example.com. In favour of this approach is that the SSO app is simple, and thus less likely to fail during development iterations than the nascent www.example.com site. A failure in the SSO app is bad, becaise it locks people out of Ning too. That Rails and Merb can share session data (and a database) is cool.
• The SSO app’s fetching of Ning profile data allows us to maintain a local version of a user’s profile to avoid the need to fetch it from Ning every time we need. There’s a nuance of Ning API authentication that meant I had to write a custom widget (Ning site component) to handle that.

Specifics
I was about to paste bits of modified Ning code, but I’ll need to check if I can under the various blurbs you agree too when signing up! Anyhow, the information above should help answer some of the requests for details from the Ning developers forum.

• There’s no invention here, I believe this setup is very common.
• High availability isn’t the same thing as load balanced. There is nothing here to intelligently shared load across the frontend servers, and one backend server is essentially idle all the time.
• This cluster is built with a bunch of open-source software on non-fancy kit. As such it doesn’t have the enormous capacity of clusters built upon commercial shared-storage products, SAN kit, layer 7 web switches etc. Its ambition is to run a few busy Rails sites well whilst coping with hardware failure gracefully.

Layout

Operation

• Web traffic is spread across the managed frontend interfaces by multiple A records in the DNS.
• Wackamole uses a Spread messaging network to ensure these multiple A record IPs are always present across the frontend. It achieves this by managing the hosts’ interfaces when it detects hosts joining or leaving the cluster.
• A pair of MySQL servers run in master:master configuration on the backend hosts
• The backend hosts use DRBD to maintain a mirrored block device between them.
• These block devices back a NFS filesystem.
• Heartbeat runs on the backend hosts to do several tasks:
  1. Manage which host is the DRBD primary and therefore can be written to.
  2. Manage which host has the DRBD filesystem mounted and exported with NFS.
  3. Manage the IP through which the frontend mounts the filesystem and talks to MySQL.
• With all this in place, Nginx accepts web connections and serves static assets off the NFS mount and passess other requests to Mongrel, a HTTP server that’s well suited to running a Rails instance.

Notes

• One of the main hazards of MySQL master:master setups is primary key collision if an INSERT occurs on both hosts at once. We avoid that here by letting Hearbeat manage the IP that the frontends connect to.
• I’ve built two of these clusters to date. The second one is now four servers wide on the frontend.

Future work

• DRBD can now run in dual-primary mode, allowing both hosts to accept writes. This makes it a candidate for filesystems like GFS that use shared storage to present a filesystem that can be written to on multiple hosts. More here.
• To add some load balancing I’m considering using HAProxy or LVS to actively distribute traffic across the frontends.
• HA aside, there’s also some cool things like evented Mongrel that it would be interesting to try.

Extra credit for assigning a keyword (eg ‘ktx’) in the bookmark properties, allowing you to just type, eg, ‘ktx 1234′ in the Location bar to achieve the same.

This is a rework of a similar hack for a much older ticketer, PTS, which is amazingly still in use at one of my previous workplaces. You can gauge its age by the fact that it was ported to PHP3 and was pretty open to most injection attacks!

I do however find myself forever doing stuff like
$ ls -tr /a/log/dir | tail
error_log.1129593600
error_log.1129680000
error_log.1129766400
$ tail -f error_log.1129766400 to diagnose a problem.

What would be cool was if rotatelogs linked, say, error_log.current to the, er, current log. If it used a hard link, you could probably use tail‘s -F flag to seemlessly glide over rotations.

Update: I looked at the source; what immediately occurs to me is that both soft and hard linking aren’t particularly portable, and Apache runs on many different platforms. It’s probably more straightforward to do this with a shell script or similar.

PHP defaults to sending nocache headers, and this makes IE aggressively zap any temporary files that arise, even if they’re destined for another app locally.

We fixed this by setting PHP’s cache limiter knob on a per-download basis, although it can be done system-wide also. Here’s the page in the PHP manual.

Whilst investigating this, I encountered the ugly truth about IE’s mime-type handling. Seems fiddly!