interrupts in python io

Michael Barton — Mon, 09 Mar 2009 12:55:55 +0000

Graham Dumpleton just wrote a blog post recommending mod_wsgi’s daemon mode, I guess on the grounds that it’s harder to screw up than apache’s mpm configurations. All the talk of mpms reminded me that I’d never posted anything about this one problem I had a while back.

See, apache’s worker mpm looks pretty appealing when you’re trying to squeeze a little more performance out of a mod_python or mod_wsgi app. Serving requests in individual threads running on the same Python interpreter should reduce the overall memory footprint and would let you keep expensive resources like persistent database connections in a per-interpreter pool that threads access as needed. SQLAlchemy’s queue pool is threadsafe for exactly that sort of application.

But there’s a snag if you use any networked services: Apache children get signaled when they hit MaxRequestsPerChild (with SIGUSR2 or SIGRTMIN or something along those lines), and maybe on some other events – that’s just the one I could readily reproduce. If any of the child’s threads are in blocking system calls (i.e. network i/o) when that signal hits, the call will be interrupted and a Python exception raised.

In real life you have to write networking code so that it can treat interrupts as non-fatal errors, but Python’s standard library makes that difficult. The popular sendall and readline methods on sockets can’t be used period, since they’ll lose important state if they’re interrupted. Most libraries that use the network, including anything built on top of httplib (such as the clients from xmlrpclib, zsi, suds, etc.), fail to treat interrupts as non-fatal errors. Fortunately, the database libraries I’ve looked at have more robust networking.

In some cases, you could wrap an entire high-level request in a try/except, check if the error argument is EINTR (or EWOULDBLOCK due to a bug in Python < 2.5), then re-try the whole request. But that’s expensive, and if you happened to be streaming transient data to a remote server or something, you’re screwed.

I’ve had some success monkey patching the socket object with methods that re-try on interrupt, but the socket module’s layout makes that difficult, fragile and slow. I also tried writing a module that re-registered all signal handlers for the process with the SA_RESTART set, but that caused more problems than it solved (and some system calls can’t be restarted anyway). We’ve considered patching the socket library to re-try i/o on interrupt at the C level, but the administrative overhead of maintaining a forked Python just isn’t worth it.

So it looks like pre-fork mpm is the only workable choice for mod_python if you use any networked services. Maybe mod_wsgi in daemon mode would insulate the Python interpreter from any signals, but honestly it’s pretty unlikely that our app will be ported any time soon.

glass box unit testing can bite me

Michael Barton — Thu, 24 Apr 2008 09:32:37 +0000

I’ve seen unit tests where everything touched by the code is mocked out, and the mocks are intelligent in such a way that the tests fail if the mocked methods aren’t all called with correct arguments. It winds up being nearly a line for line check of the implementation of the code, and is about the most horrible thing I think I’ve ever heard of.

I have basically the same problem with tests against private methods.

Changing implementation details without changing behavior called “refactoring” and is one of the selling points of unit tests. If a class behaves exactly the same today as it did yesterday, its unit tests shouldn’t suddenly fail, even if someone was mucking around inside it.

If you just want to make sure nobody changes the code, why not get a checksum of the file? It’ll be a lot less work and have about the same effect. Ta freakin' da.

And I’m not saying that there aren’t exceptions, just that they are exceptions.

WeirdLooking.com: Michael Barton's Blog

interrupts in python io

glass box unit testing can bite me