Weird Looking: Michael Barton's Web Site

March 9, 2009 7:55am (6 years, 10 months and 4 weeks ago)
Graham Dumpleton just wrote a blog post recommending mod_wsgi’s daemon mode, I guess on the grounds that it’s harder to screw up than apache’s mpm configurations.  All the talk of mpms reminded me that I’d never posted anything about this one problem I had a while back.

See, apache’s worker mpm looks pretty appealing when you’re trying to squeeze a little more performance out of a mod_python or mod_wsgi app.  Serving requests in individual threads running on the same Python interpreter should reduce the overall memory footprint and would let you keep expensive resources like persistent database connections in a per-interpreter pool that threads access as needed.  SQLAlchemy’s queue pool is threadsafe for exactly that sort of application.

But there’s a snag if you use any networked services:  Apache children get signaled when they hit MaxRequestsPerChild (with SIGUSR2 or SIGRTMIN or something along those lines), and maybe on some other events – that’s just the one I could readily reproduce.  If any of the child’s threads are in blocking system calls (i.e. network i/o) when that signal hits, the call will be interrupted and a Python exception raised.

In real life you have to write networking code so that it can treat interrupts as non-fatal errors, but Python’s standard library makes that difficult.  The popular sendall and readline methods on sockets can’t be used period, since they’ll lose important state if they’re interrupted.  Most libraries that use the network, including anything built on top of httplib (such as the clients from xmlrpclib, zsi, suds, etc.), fail to treat interrupts as non-fatal errors.  Fortunately, the database libraries I’ve looked at have more robust networking.

In some cases, you could wrap an entire high-level request in a try/except, check if the error argument is EINTR (or EWOULDBLOCK due to a bug in Python < 2.5), then re-try the whole request.  But that’s expensive, and if you happened to be streaming transient data to a remote server or something, you’re screwed.

I’ve had some success monkey patching the socket object with methods that re-try on interrupt, but the socket module’s layout makes that difficult, fragile and slow.  I also tried writing a module that re-registered all signal handlers for the process with the SA_RESTART set, but that caused more problems than it solved (and some system calls can’t be restarted anyway).  We’ve considered patching the socket library to re-try i/o on interrupt at the C level, but the administrative overhead of maintaining a forked Python just isn’t worth it.

So it looks like pre-fork mpm is the only workable choice for mod_python if you use any networked services.  Maybe mod_wsgi in daemon mode would insulate the Python interpreter from any signals, but honestly it’s pretty unlikely that our app will be ported any time soon.
October 25, 2008 3:09am (7 years, 3 months and 1 week ago)

sign's new home, originally uploaded by red_bo.

The favotech sign is now mounted in our living room, where it shares
news headlines, stock quotes and weather forecasts scraped from the
web. It's a very depressing sign.

I ordered a Pro-Lite sign to play around with at work, but it won't
get here until next week.

October 3, 2008 11:54am (7 years, 4 months and 4 days ago)
So, what I’ve got here is my implementation of the favotech sign’s protocol in python.  As I’ve mentioned, it’s horrible.  From what I can tell, the sign supports basically the same protocol over TCP, UDP, and serial.  I’ve been using UDP over a crossover cable from a whitebox with two NICs, and that’s worked fine.

I’m using it inside a twisted app, mostly for all the batteries it includes.  I have a UDP server class that inherits from this class, but implements its own send_to_sign method.  This layout made reusing the code for different connection media a breeze.  Basic usage then looks something like this:

  self.set_text(0, '{y}OH HAI')
  self.set_text(1, 'I AM {f}THE{/f} SIGN')
  self.set_text(2, '{0}{ma} {dd}  {12}')

To dumb down the interface, a frame count of n actually creates a playlist containing n two-letter filenames, 'AA’…’ZZ’.  Calling set_text with a frame ID then sets the text of the corresponding file.  The set_text method has a dead simple markup language it can use to set text colors and stuff.

More to come, probably…
September 5, 2008 8:11am (7 years, 5 months and 2 days ago)
So right now, I have a simple python app that queries servers over SNMP and displays their load averages on the sign. And the text can flash if a load average seems high, and stuff like that.  I’ll do more later, I just hacked that together yesterday to get something working.

If you try this, you may notice that the sign’s protocol documentation kind of sucks.  Oh, also, its Sigma software only uses that protocol as part of the format of its text files.  It has another, more horrible protocol, in which lots of things are framed as file transfers.  The files being transfered are also in various ugly formats.

Maybe I’ll post some code later, because this would get super-boring if I started throwing out hex codes and what endian integers to use and how it calculates checksums.

I will just say that I’m kind of wishing I’d held out for a better-known brand of sign with a better-documented protocol.
September 1, 2008 12:17am (7 years, 5 months and 6 days ago)
Is that an LED sign on my desk?  I believe it is.

It’s a FavoTech model I picked up on eBay that can be programmed by ethernet, USB, serial and IR.  My weekend project has been reverse engineering enough of its network protocol to control it programatically.  The protocol is all binary stuff over UDP, but I’ve made quite a bit of progress in deciphering it.

It’s not all mai tais and yahtzee, though.  I somehow broke the sign so that when it first boots up, it spams the network with constant ARP packets until I’ve uploaded a playlist.  I think it’s because I removed the playlist on the flash drive, but I haven’t figured out how to get one back on there yet.  Against all reason, it’s apparently not “select the flash drive then upload a playlist”.

I know it has a speaker, because it’s forever beeping when it resets.  I just haven’t figured out any message I can send it to make it beep on cue.  I emailed their tech support but haven’t heard back yet.

I’m not even being original here – one of my co-workers has a nearly identical LED sign that used to stream news feeds and OA scores in our office.

It’s pretty fun, though.  And I think there’s a lot of neat information you could convey in sign form.  The only limit is my imagination!  Oh, hell.
May 29, 2008 6:56am (7 years, 8 months and 1 week ago)
So, it turns out that Greg's darkest hour was one of my brighter hours.

The thing to take away from our scorecard is not that I lost at par 3 golf.
It’s that I lost at par 3 golf by the least that I ever lost by before!

I’m pretty sure I understand how Arnold Palmer feels now.  You know, like I’ll be remembered for something.  Sure, I’ll be remembered for golf while he’s remembered for mixing lemonade and iced tea, but the analogy still holds.
May 17, 2008 8:43pm (7 years, 8 months and 2 weeks ago)

developer conference at work, originally uploaded by red_bo.

You can tell nobody said "cheese!" first.

That's Nels, me, and Greg at a company developers' conference.