Tim Arnold wrote:
Hi, I've been using the threading module with each thread as a key in a
dictionary. I've been reading about Queues though and it looks like that's
what I should be using instead. Just checking here to see if I'm on the
right path.
The code I have currently compiles a bunch of chapters in a book (no more
than 80 jobs at a time) and just waits for them all to finish:
max_running = 80
threads = dict()
current = 1
chaps = [x.config['name'] for x in self.document.chapter_objects]
while current <= len(chaps):
running = len([x for x in threads.keys() if
threads[x].isAlive()])
if running == max_running:
time.sleep(10)
else:
chap = chaps[current - 1]
c = self.compiler(self.document.config['name'], chap)
threads[chap] = threading.Thread(target=c.compile)
threads[chap].start()
current += 1
for thread in threads.keys():
threads[thread].join(3600.0)
---------------------------------
but I think Queue could do a lot of the above work for me. Here is
pseudocode for what I'm thinking:
q = Queue(maxsize=80)
for chap in [x.config['name'] for x in self.document.chapter_objects]:
c = self.compiler(self.document.config['name'], chap)
t = threading.Thread(target=c.compile)
t.start()
q.put(t)
q.join()
is that the right idea?
I don't need that many threads; just create a few to do the work and let
each do multiple chapters, something like this:
class CompilerTask(object):
def __init__(self, chapter_queue):
self.chapter_queue = chapter_queue
def __call__(self):
while True:
chapter = self.chapter_queue.get()
if chapter is None:
# A None indicates that there are no more chapters.
break
chapter.compile()
# Put back the None so that the next thread will also see it.
self.chapter_queue.put(None)
MAX_RUNNING = 10
# Put the chapters into a queue, ending with a None.
chapter_queue = Queue()
for c in self.document.chapter_objects:
chapter_queue.put(self.compiler(self.document.config['name'],
c.config['name']))
chapter_queue.put(None)
# Start the threads to do the work.
for i in range(MAX_RUNNING):
t = threading.Thread(target=CompilerTask(chapter_queue))
t.start()
thread_list.append(t)
# The threads will finish when they see the None in the queue.
for t in thread_list:
t.join()
--
http://mail.python.org/mailman/listinfo/python-list
