Following up to my own post... Flickr informs me that quite a few of you have been looking at my graphs of performance vs. the number of sub-processes employed in a parallelizable task:
On May 21, 8:58 pm, John Ladasky <lada...@my-deja.com> wrote: > http://www.flickr.com/photos/15579975@N00/5744093219 [...] > I'll quickly ask my question first, to avoid a TL;DR problem: when you > have a multi-core CPU with N cores, is it common to see the > performance peak at N-1, or even N-2 processes? And so, should you > avoid using quite as many processes as there are cores? I was > expecting diminishing returns for each additional core -- but not > outright declines. But no one has offered any insight yet? Well, I slept on it, and I had a thought. Please feel free to shoot it down. If I spawn N worker sub-processes, my application in fact has N+1 processes in all, because there's also the master process itself. If the master process has anything significant to do (and mine does, and I would surmise that many multi-core applications would be that way), then the master process may sometimes find itself competing for time on a CPU core with a worker sub-process. This could impact performance even when the demands from the operating system and/or other applications are modest. I'd still appreciate hearing from anyone else who has more experience with multiprocessing. If there are general rules about how to do this best, I haven't seen them posted anywhere. This may not be a Python- specific issue, of course. Tag, you're it! -- http://mail.python.org/mailman/listinfo/python-list
