nginx and flup
I hope this will be helpful to someone, someday:
If you're using flup to serve fastCGI to nginx, don't enable multiplexing!
Background:
The Bugzilla Dashboard uses web.py as its back-end, for authentication, caching, and storage of divisions, teams, and members. web.py, in turn, uses flup, a relatively popular CGI, WSGI, and fastCGI library. It was relatively easy to write my back end, and hooking it up to nginx wasn't too bad either.
However over time, I noticed the flup would crash, not being able to create a new thread. Looking closer, my app (which runs in a separate process thanks to fastCGI) had many, many running threads (I forget the exact number, but after an hour or two it was well over 100). There were also quite a few "connection reset by peer" exceptions thrown by the flup code responsible for cleanup--so likely the client was occasionally hanging up just a bit before flup.
Switching to a forked, multiprocess model didn't help the situation--the machine ended up with piles of process instead of threads. Out of desperation, I tried disabling multiplexing, the interleaving of fastCGI requests on one connection. Poof, all my extra threads disappeared, and everything appears stable!
The worst part of this was that the web.py code defaults to turning on multiplexing:
def runfcgi(func, addr=('localhost', 8000)):
"""Runs a WSGI function as a FastCGI server."""
import flup.server.fcgi as flups
return flups.WSGIServer(func, multiplexed=True, bindAddress=addr, debug=False).run()
This is despite what flup says:
Set multiplexed to True if you want to handle multiple requests
per connection. Some FastCGI backends (namely mod_fastcgi) don't
multiplex requests at all, so by default this is off (which saves
on thread creation/locking overhead).
Luckily it was easy enough to disable multiplexing by writing my own runfcgi function.
So either nginx does support multiplexing but improperly, or, more likely, there's a bug in flup (probably in the MultiplexedConnection class, since this problem occurs with both the threaded and forking servers--I bet some request threads aren't being dealt with if the connection terminates suddenly). At some point I want to take a closer look, but for now, beware if you using nginx with flup!
If you're using flup to serve fastCGI to nginx, don't enable multiplexing!
Background:
The Bugzilla Dashboard uses web.py as its back-end, for authentication, caching, and storage of divisions, teams, and members. web.py, in turn, uses flup, a relatively popular CGI, WSGI, and fastCGI library. It was relatively easy to write my back end, and hooking it up to nginx wasn't too bad either.
However over time, I noticed the flup would crash, not being able to create a new thread. Looking closer, my app (which runs in a separate process thanks to fastCGI) had many, many running threads (I forget the exact number, but after an hour or two it was well over 100). There were also quite a few "connection reset by peer" exceptions thrown by the flup code responsible for cleanup--so likely the client was occasionally hanging up just a bit before flup.
Switching to a forked, multiprocess model didn't help the situation--the machine ended up with piles of process instead of threads. Out of desperation, I tried disabling multiplexing, the interleaving of fastCGI requests on one connection. Poof, all my extra threads disappeared, and everything appears stable!
The worst part of this was that the web.py code defaults to turning on multiplexing:
def runfcgi(func, addr=('localhost', 8000)):
"""Runs a WSGI function as a FastCGI server."""
import flup.server.fcgi as flups
return flups.WSGIServer(func, multiplexed=True, bindAddress=addr, debug=False).run()
This is despite what flup says:
Set multiplexed to True if you want to handle multiple requests
per connection. Some FastCGI backends (namely mod_fastcgi) don't
multiplex requests at all, so by default this is off (which saves
on thread creation/locking overhead).
Luckily it was easy enough to disable multiplexing by writing my own runfcgi function.
So either nginx does support multiplexing but improperly, or, more likely, there's a bug in flup (probably in the MultiplexedConnection class, since this problem occurs with both the threaded and forking servers--I bet some request threads aren't being dealt with if the connection terminates suddenly). At some point I want to take a closer look, but for now, beware if you using nginx with flup!