WeirdLooking.com: Michael Barton's Blog

interrupts in python io

Michael Barton — Mon, 09 Mar 2009 12:55:55 +0000

Graham Dumpleton just wrote a blog post recommending mod_wsgi’s daemon mode, I guess on the grounds that it’s harder to screw up than apache’s mpm configurations. All the talk of mpms reminded me that I’d never posted anything about this one problem I had a while back.

See, apache’s worker mpm looks pretty appealing when you’re trying to squeeze a little more performance out of a mod_python or mod_wsgi app. Serving requests in individual threads running on the same Python interpreter should reduce the overall memory footprint and would let you keep expensive resources like persistent database connections in a per-interpreter pool that threads access as needed. SQLAlchemy’s queue pool is threadsafe for exactly that sort of application.

But there’s a snag if you use any networked services: Apache children get signaled when they hit MaxRequestsPerChild (with SIGUSR2 or SIGRTMIN or something along those lines), and maybe on some other events – that’s just the one I could readily reproduce. If any of the child’s threads are in blocking system calls (i.e. network i/o) when that signal hits, the call will be interrupted and a Python exception raised.

In real life you have to write networking code so that it can treat interrupts as non-fatal errors, but Python’s standard library makes that difficult. The popular sendall and readline methods on sockets can’t be used period, since they’ll lose important state if they’re interrupted. Most libraries that use the network, including anything built on top of httplib (such as the clients from xmlrpclib, zsi, suds, etc.), fail to treat interrupts as non-fatal errors. Fortunately, the database libraries I’ve looked at have more robust networking.

In some cases, you could wrap an entire high-level request in a try/except, check if the error argument is EINTR (or EWOULDBLOCK due to a bug in Python < 2.5), then re-try the whole request. But that’s expensive, and if you happened to be streaming transient data to a remote server or something, you’re screwed.

I’ve had some success monkey patching the socket object with methods that re-try on interrupt, but the socket module’s layout makes that difficult, fragile and slow. I also tried writing a module that re-registered all signal handlers for the process with the SA_RESTART set, but that caused more problems than it solved (and some system calls can’t be restarted anyway). We’ve considered patching the socket library to re-try i/o on interrupt at the C level, but the administrative overhead of maintaining a forked Python just isn’t worth it.

So it looks like pre-fork mpm is the only workable choice for mod_python if you use any networked services. Maybe mod_wsgi in daemon mode would insulate the Python interpreter from any signals, but honestly it’s pretty unlikely that our app will be ported any time soon.

sign's new home

Michael Barton — Sat, 25 Oct 2008 08:09:53 +0000

sign's new home, originally uploaded by red_bo.

The favotech sign is now mounted in our living room, where it shares
news headlines, stock quotes and weather forecasts scraped from the
web. It's a very depressing sign.

I ordered a Pro-Lite sign to play around with at work, but it won't
get here until next week.

some sign code

Michael Barton — Fri, 03 Oct 2008 16:54:44 +0000

So, what I’ve got here is my implementation of the favotech sign’s protocol in python. As I’ve mentioned, it’s horrible. From what I can tell, the sign supports basically the same protocol over TCP, UDP, and serial. I’ve been using UDP over a crossover cable from a whitebox with two NICs, and that’s worked fine.

I’m using it inside a twisted app, mostly for all the batteries it includes. I have a UDP server class that inherits from this class, but implements its own send_to_sign method. This layout made reusing the code for different connection media a breeze. Basic usage then looks something like this:

   self.test_reset()
   self.pause()
   self.time_sync()
   self.set_frame_count(3)
   self.set_text(0, '{y}OH HAI')
   self.set_text(1, 'I AM {f}THE{/f} SIGN')
   self.set_text(2, '{0}{ma} {dd}  {12}')
   self.resume()

To dumb down the interface, a frame count of n actually creates a playlist containing n two-letter filenames, 'AA’…’ZZ’. Calling set_text with a frame ID then sets the text of the corresponding file. The set_text method has a dead simple markup language it can use to set text colors and stuff.

More to come, probably…

hacking the sign

Michael Barton — Fri, 05 Sep 2008 13:11:28 +0000

So right now, I have a simple python app that queries servers over SNMP and displays their load averages on the sign. And the text can flash if a load average seems high, and stuff like that. I’ll do more later, I just hacked that together yesterday to get something working.

If you try this, you may notice that the sign’s protocol documentation kind of sucks. Oh, also, its Sigma software only uses that protocol as part of the format of its text files. It has another, more horrible protocol, in which lots of things are framed as file transfers. The files being transfered are also in various ugly formats.

Maybe I’ll post some code later, because this would get super-boring if I started throwing out hex codes and what endian integers to use and how it calculates checksums.

I will just say that I’m kind of wishing I’d held out for a better-known brand of sign with a better-documented protocol.

it's saying hi

Michael Barton — Mon, 01 Sep 2008 05:17:18 +0000

Is that an LED sign on my desk? I believe it is.

It’s a FavoTech model I picked up on eBay that can be programmed by ethernet, USB, serial and IR. My weekend project has been reverse engineering enough of its network protocol to control it programatically. The protocol is all binary stuff over UDP, but I’ve made quite a bit of progress in deciphering it.

It’s not all mai tais and yahtzee, though. I somehow broke the sign so that when it first boots up, it spams the network with constant ARP packets until I’ve uploaded a playlist. I think it’s because I removed the playlist on the flash drive, but I haven’t figured out how to get one back on there yet. Against all reason, it’s apparently not “select the flash drive then upload a playlist”.

I know it has a speaker, because it’s forever beeping when it resets. I just haven’t figured out any message I can send it to make it beep on cue. I emailed their tech support but haven’t heard back yet.

I’m not even being original here – one of my co-workers has a nearly identical LED sign that used to stream news feeds and OA scores in our office.

It’s pretty fun, though. And I think there’s a lot of neat information you could convey in sign form. The only limit is my imagination! Oh, hell.

hot asian girls

Michael Barton — Thu, 29 May 2008 12:41:02 +0000

This is my second-favorite ad from Facebook. It’s pretty well-targeted. I mean, how did they know that I like hot asian girls?

My #1 favorite ad I ever saw (but failed to get a screenshot of) had a picture of some dude without a shirt on and said, “Try something different!”

par 3 golf

Michael Barton — Thu, 29 May 2008 11:56:39 +0000

So, it turns out that Greg's darkest hour was one of my brighter hours.

The thing to take away from our scorecard is not that I lost at par 3 golf.
It’s that I lost at par 3 golf by the least that I ever lost by before!

I’m pretty sure I understand how Arnold Palmer feels now. You know, like I’ll be remembered for something. Sure, I’ll be remembered for golf while he’s remembered for mixing lemonade and iced tea, but the analogy still holds.

i just liked this picture...

Michael Barton — Sun, 18 May 2008 01:43:09 +0000

developer conference at work, originally uploaded by red_bo.

You can tell nobody said "cheese!" first.

That's Nels, me, and Greg at a company developers' conference.

glass box unit testing can bite me

Michael Barton — Thu, 24 Apr 2008 09:32:37 +0000

I’ve seen unit tests where everything touched by the code is mocked out, and the mocks are intelligent in such a way that the tests fail if the mocked methods aren’t all called with correct arguments. It winds up being nearly a line for line check of the implementation of the code, and is about the most horrible thing I think I’ve ever heard of.

I have basically the same problem with tests against private methods.

Changing implementation details without changing behavior called “refactoring” and is one of the selling points of unit tests. If a class behaves exactly the same today as it did yesterday, its unit tests shouldn’t suddenly fail, even if someone was mucking around inside it.

If you just want to make sure nobody changes the code, why not get a checksum of the file? It’ll be a lot less work and have about the same effect. Ta freakin' da.

And I’m not saying that there aren’t exceptions, just that they are exceptions.

so it began...

Michael Barton — Mon, 17 Mar 2008 07:16:36 +0000

so it began..., originally uploaded by red_bo.

Here, Brian begins his rapid downward spiral by loitering in front of
a "no loitering" sign.

He would not be seen or heard from for three days.