multiprocessing: pool with blocking queue

2009-07-02 Thread masher
Hi, I am trying to implement a multiprocessing pool that assigns tasks from a blocking queue. My situation is a pretty classic producer/ consumer conundrum, where the producer can produce much faster than the consumers can consume. The wrinkle in the story is that the producer produces objects

Re: multiprocessing: pool with blocking queue

2009-07-02 Thread J Kenneth King
masher vertesp...@gmail.com writes: My questions, then, is: Is there a more elegant/pythonic way of doing what I am trying to do with the current Pool class? Forgive me, I may not fully understand what you are trying to do here (I've never really used multiprocessing all that much)... But

Re: multiprocessing: pool with blocking queue

2009-07-02 Thread masher
On Jul 2, 12:06 pm, J Kenneth King ja...@agentultra.com wrote: masher vertesp...@gmail.com writes: My questions, then, is: Is there a more elegant/pythonic way of doing what I am trying to do with the current Pool class? Forgive me, I may not fully understand what you are trying to do here

Re: multiprocessing: pool with blocking queue

2009-07-02 Thread J Kenneth King
masher vertesp...@gmail.com writes: On Jul 2, 12:06 pm, J Kenneth King ja...@agentultra.com wrote: masher vertesp...@gmail.com writes: My questions, then, is: Is there a more elegant/pythonic way of doing what I am trying to do with the current Pool class? Forgive me, I may not fully

Re: multiprocessing: pool with blocking queue

2009-07-02 Thread ryles
On Jul 2, 11:09 am, masher vertesp...@gmail.com wrote: My questions, then, is: Is there a more elegant/pythonic way of doing what I am trying to do with the current Pool class? Another thing you might try is to subclass Pool and add an apply_async () wrapper which would wait for