On Jan 16, 11:39 pm, TomF <tomf.sess...@gmail.com> wrote: > One difficulty is that there is a queue of work to be done and a queue > of results to be incorporated back into the parent; there is no > one-to-one correspondence between the two. It's not obvious to me how > to coordinate the queues in a natural way to avoid deadlock or > starvation. >
Depends on what you are doing. If you can enqueue all the jobs before waiting for your results, then two queues are adequate. The first queue is jobs to be accomplished, the second queue is the results. The items you put on the result queue have both the result and some sort of id so the results can be ordered after the fact. Your parent thread of execution (thread hereafter) then: 1. Adds jobs to the queue 2. Blocks until all the results are returned. Given that you suggested that there isn't a 1:1 correspondence between jobs and results, have the queue support a message saying, 'Job X is done'. You're finished when all jobs send such a message. 3. Sorts the results into the desired ordered. 4. Acts on them. If you cannot enqueue all the jobs before waiting for the results, I suggest turning the problem into a pipeline, such that the thread submitting the jobs and the thread acting on the results are different: submitter -> job processor -> results processor. Again though, the devil is in the details and without more details, it's hard to suggest an explicit approach. The simplest way to avoid contention between two queues is to just remove it entirely (by converting the processing to a single pipeline like I suggested). If that is not possible, then I suggest moving to pipes (or some other form of I/O based IPC) and asynchronous I/O. But I'd only do that if I really couldn't write a pipeline. Adam -- http://mail.python.org/mailman/listinfo/python-list