Re: Multiprocessing: don't push the pedal to the metal?

2011-05-23 Thread Adam Tauno Williams
On Mon, 2011-05-23 at 12:51 -0700, John Ladasky wrote: > On May 23, 2:50 am, Adam Tauno Williams > wrote: > > I develop an app that uses multiprocessing heavily. Remember that all > > these processes are processes - so you can use all the OS facilities > > regarding processes on them. This inclu

Re: Multiprocessing: don't push the pedal to the metal?

2011-05-23 Thread John Ladasky
On May 23, 2:50 am, Adam Tauno Williams wrote: > I develop an app that uses multiprocessing heavily.  Remember that all > these processes are processes - so you can use all the OS facilities > regarding processes on them.  This includes setting nice values, > schedular options, CPU pinning, etc..

Re: Multiprocessing: don't push the pedal to the metal?

2011-05-23 Thread Adam Tauno Williams
On Mon, 2011-05-23 at 10:32 +1000, Chris Angelico wrote: > On Mon, May 23, 2011 at 7:06 AM, John Ladasky wrote: > > If I spawn N worker sub-processes, my application in fact has N+1 > > processes in all, because there's also the master process itself. > > I'd still appreciate hearing from anyone e

Re: Multiprocessing: don't push the pedal to the metal?

2011-05-22 Thread Chris Angelico
On Mon, May 23, 2011 at 7:06 AM, John Ladasky wrote: > If I spawn N worker sub-processes, my application in fact has N+1 > processes in all, because there's also the master process itself. This would definitely be correct. How much impact the master process has depends on how much it's doing. >

Re: Multiprocessing: don't push the pedal to the metal?

2011-05-22 Thread John Ladasky
Following up to my own post... Flickr informs me that quite a few of you have been looking at my graphs of performance vs. the number of sub-processes employed in a parallelizable task: On May 21, 8:58 pm, John Ladasky wrote: > http://www.flickr.com/photos/15579975@N00/5744093219 [...] > I'll qu

Multiprocessing: don't push the pedal to the metal?

2011-05-21 Thread John Ladasky
Hello again, everyone. I'm developing some custom neural network code. I'm using Python 2.6, Numpy 1.5, and Ubuntu Linux 10.10. I have an AMD 1090T six-core CPU. About six weeks ago, I asked some questions about multiprocessing in Python, and I got some very helpful responses from you all. ht