[issue6407] multiprocessing Pool should allow custom task queue

Irit Katriel added the comment: It should be possible to do this now through the context, but it's not documented: >>> import multiprocessing as mp >>> ctx = mp.get_context() >>> ctx.SimpleQueue > >>> ctx.SimpleQueue = list >>> ctx.SimpleQueue -- nosy: +iritkatriel

[issue6407] multiprocessing Pool should allow custom task queue

Ask Solem added the comment: @swindmill, if you provide a doc/test patch then this can probably be merged. @pitrou, We could change it to `setup_queues`, though I don't think even changing the name of "private" methods is a good idea. It could simply be an alias to `_setup_queues` or vice ver

[issue6407] multiprocessing Pool should allow custom task queue

Antoine Pitrou added the comment: As the name implies, _setup_queues is a private method. It feels a bit weird to recommend overriding it in a subclass. -- nosy: +pitrou stage: test needed -> needs patch versions: +Python 3.3 -Python 3.2 ___ Python

[issue6407] multiprocessing Pool should allow custom task queue

Changes by Petri Lehtinen : -- nosy: +petri.lehtinen ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail

[issue6407] multiprocessing Pool should allow custom task queue

Sterling Windmill added the comment: I would also like to see this functionality as I'm currently using more memory than I'd like when using multiprocessing.Pool -- nosy: +swindmill ___ Python tracker

[issue6407] multiprocessing Pool should allow custom task queue

Matthew Leon Grinshpun added the comment: I should be able to do this in November. For the moment I'm a bit busy. -- ___ Python tracker ___ __

[issue6407] multiprocessing Pool should allow custom task queue

Ask Solem added the comment: Matthew, would you be willing to write tests + documentation for this? -- ___ Python tracker ___ ___ Pyth

[issue6407] multiprocessing Pool should allow custom task queue

Jesse Noller added the comment: > are there really any test/doc changes needed for this? Yes. At bare minimum we need unit tests for the new possible usage of the API; I'd like docs and an example which show the new usage as well. -- ___ Python trac

[issue6407] multiprocessing Pool should allow custom task queue

Changes by Ask Solem : -- stage: needs patch -> unit test needed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue6407] multiprocessing Pool should allow custom task queue

Ask Solem added the comment: are there really any test/doc changes needed for this? -- ___ Python tracker ___ ___ Python-bugs-list mai

[issue6407] multiprocessing Pool should allow custom task queue

Changes by Mark Lawrence : -- stage: -> needs patch versions: +Python 3.2 -Python 3.1 ___ Python tracker ___ ___ Python-bugs-list mail

[issue6407] multiprocessing Pool should allow custom task queue

Changes by Ask Solem : -- nosy: +asksol ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/m

[issue6407] multiprocessing Pool should allow custom task queue

Jesse Noller added the comment: Thanks Matthew - I agree, it's simple. But fixing the code is 1/3 of the patch. I'll need to add tests/update the ones there as well as modify the .rst documentation. -- ___ Python tracker

[issue6407] multiprocessing Pool should allow custom task queue

Matthew Leon Grinshpun added the comment: I have attached a patch. All I did was shift the one line from __init__ to _setup_queues. That's it. You could take advantage of the change in the following way: class BlockingPool(pool.Pool): def _setup_queues(self): pool.Pool._setup_queu

[issue6407] multiprocessing Pool should allow custom task queue

Jesse Noller added the comment: I'm not against this; new features will only be committed to the 2.7 and 3.1.x branches however. Something to help speed this up would be an actual patch with docs/tests for the module made against python trunk. That way I could review and simply submit. ---

[issue6407] multiprocessing Pool should allow custom task queue

Changes by Benjamin Peterson : -- assignee: -> jnoller nosy: +jnoller ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubsc

[issue6407] multiprocessing Pool should allow custom task queue

New submission from Matthew Leon Grinshpun : Multiprocessing's Pool class __init__ method is written in a way that makes it very difficult for a subclass to modify self._taskqueue. There are very good reasons for wanting to do this - ie, making the taskqueue block when it reaches a certain size.