On Mon, May 23, 2011 at 7:06 AM, John Ladasky <lada...@my-deja.com> wrote: > If I spawn N worker sub-processes, my application in fact has N+1 > processes in all, because there's also the master process itself.
This would definitely be correct. How much impact the master process has depends on how much it's doing. > I'd still appreciate hearing from anyone else who has more experience > with multiprocessing. If there are general rules about how to do this > best, I haven't seen them posted anywhere. This may not be a Python- > specific issue, of course. I don't have much experience with Python's multiprocessing model, but I've done concurrent programming on a variety of platforms, and there are some common issues. Each CPU (or core) has its own execution cache. If you can keep one thread running on the same core all the time, it will benefit more from that cache than if it has to keep flitting from one to another. You undoubtedly will have other processes in the system, too. As well as your master, there'll be processes over which you have no control (unless you're on a bare-bones system). Some of them may preempt your processes. Leaving one CPU/core available for "everything else" may allow the OS to keep each thread on its own core. Having as many workers as cores means that every time there's something else to do, one of your workers has to be kicked off its nice warm CPU and sent out into the cold for a while. If all your workers are at the same priority, it will then grab a timeslice off one of the other cores, kicking its incumbent off... rinse and repeat. This is a tradeoff, though. If the rest of your system is going to use 0.01 of a core, then 1% thrashing is worth having one more core available 99% of the time. If the numbers are reversed, it's equally obvious that you should leave one core available. In your case, it's probably turning out that the contention causes more overhead than the extra worker is worth. That's just some general concepts, without an in-depth analysis of your code and your entire system. It's probably easier to analyse by results rather than inspection. Chris Angelico -- http://mail.python.org/mailman/listinfo/python-list
