Weird Looking: interrupts in python io

Graham Dumpleton just wrote a blog post recommending mod_wsgi’s daemon mode, I guess on the grounds that it’s harder to screw up than apache’s mpm configurations.  All the talk of mpms reminded me that I’d never posted anything about this one problem I had a while back.

See, apache’s worker mpm looks pretty appealing when you’re trying to squeeze a little more performance out of a mod_python or mod_wsgi app.  Serving requests in individual threads running on the same Python interpreter should reduce the overall memory footprint and would let you keep expensive resources like persistent database connections in a per-interpreter pool that threads access as needed.  SQLAlchemy’s queue pool is threadsafe for exactly that sort of application.

But there’s a snag if you use any networked services:  Apache children get signaled when they hit MaxRequestsPerChild (with SIGUSR2 or SIGRTMIN or something along those lines), and maybe on some other events – that’s just the one I could readily reproduce.  If any of the child’s threads are in blocking system calls (i.e. network i/o) when that signal hits, the call will be interrupted and a Python exception raised.

In real life you have to write networking code so that it can treat interrupts as non-fatal errors, but Python’s standard library makes that difficult.  The popular sendall and readline methods on sockets can’t be used period, since they’ll lose important state if they’re interrupted.  Most libraries that use the network, including anything built on top of httplib (such as the clients from xmlrpclib, zsi, suds, etc.), fail to treat interrupts as non-fatal errors.  Fortunately, the database libraries I’ve looked at have more robust networking.

In some cases, you could wrap an entire high-level request in a try/except, check if the error argument is EINTR (or EWOULDBLOCK due to a bug in Python < 2.5), then re-try the whole request.  But that’s expensive, and if you happened to be streaming transient data to a remote server or something, you’re screwed.

I’ve had some success monkey patching the socket object with methods that re-try on interrupt, but the socket module’s layout makes that difficult, fragile and slow.  I also tried writing a module that re-registered all signal handlers for the process with the SA_RESTART set, but that caused more problems than it solved (and some system calls can’t be restarted anyway).  We’ve considered patching the socket library to re-try i/o on interrupt at the C level, but the administrative overhead of maintaining a forked Python just isn’t worth it.

So it looks like pre-fork mpm is the only workable choice for mod_python if you use any networked services.  Maybe mod_wsgi in daemon mode would insulate the Python interpreter from any signals, but honestly it’s pretty unlikely that our app will be ported any time soon.

Name

Email

Link

Comment

Accepts BBCode with a few enhancements.