Sprinkle time.sleep(0) liberally throughout your code where you think natural processing breaks should be. Even in while loops. It's lame but is the only way to make Python multithreading task switch fairly. Your compute intensive tasks need a time.sleep(0) in their loops. This prevents starvation and makes overall processing and responsiveness seem properly multithreaded. This is a hand optimization so you have to play with the location and amount of time.sleep(0)s. You'll know when you've found a problematic spot when the queues stop growing/overflowing.
Put the dns lookup on a separate thread pool with it's own growing queue with lots of time.sleep(0)s sprinkled in. The dns lookups don't have to be real time and you can easily cache them with a timestamp attached. This is the thread pool where more is better and threads should be aggressively terminated for having a long running process time. This also requires lots of hand tuning for dynamically managing the number of threads needed to process the queue in a reasonable time if you find it hard to aggressively kill threads. I think there is a way to launch threads that only give them a maximum lifetime. The problem you will hit while tuning may require allocating more file handles for all the hung sockets. The DNS lookup is one of those things that may make sense to run as a separate daemon process that listens on a socket. You make one connection that feeds in the ip addresses. The daemon process feeds back ip address/host name combinations out of order. Your main process/connection thread builds a serialized access dict with timestamps. The main processes threads make their requests asynchronously and sleep while waiting for the response to appear in the dict. They terminate after a certain time if they don't see their response. Requires hand/algorithmic tweaking for this to work correctly across different machines. On Fri, Apr 27, 2012 at 2:54 PM, John Nagle <na...@animats.com> wrote: > I have a multi-threaded CPython program, which has up to four > threads. One thread is simply a wait loop monitoring the other > three and waiting for them to finish, so it can give them more > work to do. When the work threads, which read web pages and > then parse them, are compute-bound, I've had the monitoring thread > starved of CPU time for as long as 120 seconds. > It's sleeping for 0.5 seconds, then checking on the other threads > and for new work do to, so the work thread isn't using much > compute time. > > I know that the CPython thread dispatcher sucks, but I didn't > realize it sucked that bad. Is there a preference for running > threads at the head of the list (like UNIX, circa 1979) or > something like that? > > (And yes, I know about "multiprocessing". These threads are already > in one of several service processes. I don't want to launch even more > copies of the Python interpreter. The threads are usually I/O bound, > but when they hit unusually long web pages, they go compute-bound > during parsing.) > > Setting "sys.setcheckinterval" from the default to 10000 seems > to have little effect. This is on Windows 7. > > John Nagle > -- > http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list