Thomas Grainger writes: > Would a work stealing approach work better for you here? Then the only > signalling overhead would be when a core runs out of work
Not sure what you're talking about with "work stealing". It sounds conceptually more complex than the queue + worker pool approach, which is already implemented in both the threading and multiprocessing modules. The overhead of creating hundreds of multiprocessing tasks is going to be barely human-perceptible. The other "overhead" is the programmer effort in assembling the finished product (assuming order matters, or there are interdependencies between chunks that require keeping per-chunk state). But I don't see how such programmer effort would be much greater for the "many chunks in a queue" approach vs. the chunk-per-core approach. So it seems to me that multiprocessing with a worker pool is a low programmer effort, very high efficiency gain approach to this problem. The remaining question is "how many chunks?" If that's relevant, ISTM a few simple experiments will show where the sweet spot is. Try a queue of 64 chunks, then 128 chunks, and refine guesses from there. I may be missing something, but that's the thinking that led to my previous post. Steve _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/YUEBSYCWGU4ZJIJOE6XXMIA6N3KHK75I/ Code of Conduct: http://python.org/psf/codeofconduct/
