interrupts in python io

Michael Barton — Mon, 09 Mar 2009 12:55:55 +0000

Graham Dumpleton just wrote a blog post recommending mod_wsgi’s daemon mode, I guess on the grounds that it’s harder to screw up than apache’s mpm configurations. All the talk of mpms reminded me that I’d never posted anything about this one problem I had a while back.

See, apache’s worker mpm looks pretty appealing when you’re trying to squeeze a little more performance out of a mod_python or mod_wsgi app. Serving requests in individual threads running on the same Python interpreter should reduce the overall memory footprint and would let you keep expensive resources like persistent database connections in a per-interpreter pool that threads access as needed. SQLAlchemy’s queue pool is threadsafe for exactly that sort of application.

But there’s a snag if you use any networked services: Apache children get signaled when they hit MaxRequestsPerChild (with SIGUSR2 or SIGRTMIN or something along those lines), and maybe on some other events – that’s just the one I could readily reproduce. If any of the child’s threads are in blocking system calls (i.e. network i/o) when that signal hits, the call will be interrupted and a Python exception raised.

In real life you have to write networking code so that it can treat interrupts as non-fatal errors, but Python’s standard library makes that difficult. The popular sendall and readline methods on sockets can’t be used period, since they’ll lose important state if they’re interrupted. Most libraries that use the network, including anything built on top of httplib (such as the clients from xmlrpclib, zsi, suds, etc.), fail to treat interrupts as non-fatal errors. Fortunately, the database libraries I’ve looked at have more robust networking.

In some cases, you could wrap an entire high-level request in a try/except, check if the error argument is EINTR (or EWOULDBLOCK due to a bug in Python < 2.5), then re-try the whole request. But that’s expensive, and if you happened to be streaming transient data to a remote server or something, you’re screwed.

I’ve had some success monkey patching the socket object with methods that re-try on interrupt, but the socket module’s layout makes that difficult, fragile and slow. I also tried writing a module that re-registered all signal handlers for the process with the SA_RESTART set, but that caused more problems than it solved (and some system calls can’t be restarted anyway). We’ve considered patching the socket library to re-try i/o on interrupt at the C level, but the administrative overhead of maintaining a forked Python just isn’t worth it.

So it looks like pre-fork mpm is the only workable choice for mod_python if you use any networked services. Maybe mod_wsgi in daemon mode would insulate the Python interpreter from any signals, but honestly it’s pretty unlikely that our app will be ported any time soon.

some sign code

Michael Barton — Fri, 03 Oct 2008 16:54:44 +0000

So, what I’ve got here is my implementation of the favotech sign’s protocol in python. As I’ve mentioned, it’s horrible. From what I can tell, the sign supports basically the same protocol over TCP, UDP, and serial. I’ve been using UDP over a crossover cable from a whitebox with two NICs, and that’s worked fine.

I’m using it inside a twisted app, mostly for all the batteries it includes. I have a UDP server class that inherits from this class, but implements its own send_to_sign method. This layout made reusing the code for different connection media a breeze. Basic usage then looks something like this:

   self.test_reset()
   self.pause()
   self.time_sync()
   self.set_frame_count(3)
   self.set_text(0, '{y}OH HAI')
   self.set_text(1, 'I AM {f}THE{/f} SIGN')
   self.set_text(2, '{0}{ma} {dd}  {12}')
   self.resume()

To dumb down the interface, a frame count of n actually creates a playlist containing n two-letter filenames, 'AA’…’ZZ’. Calling set_text with a frame ID then sets the text of the corresponding file. The set_text method has a dead simple markup language it can use to set text colors and stuff.

More to come, probably…

WeirdLooking.com: Michael Barton's Blog

interrupts in python io

some sign code