[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Roundup Robot added the comment: New changeset 72a5ac909c7a by Richard Oudkerk in branch 'default': Issue #18999: Make multiprocessing use context objects. http://hg.python.org/cpython/rev/72a5ac909c7a -- nosy: +python-dev ___ Python tracker

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Lars Buitinck added the comment: Ok, great. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyt

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Richard Oudkerk added the comment: > I haven't read all of your patch yet, but does this mean a forkserver > will be started regardless of whether it is later used? No, it is started on demand. But since it is started using _posixsbuprocess.fork_exec(), nothing is inherited from the main proc

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Lars Buitinck added the comment: > BTW, the context objects are singletons. I haven't read all of your patch yet, but does this mean a forkserver will be started regardless of whether it is later used? That would be a good thing, since starting the fork server after reading in large data sets

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Richard Oudkerk added the comment: Attached is a patch which allows the use of separate contexts. For example try: ctx = multiprocessing.get_context('forkserver') except ValueError: ctx = multiprocessing.get_context('spawn') q = ctx.Queue() p = ctx.Process(targe

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Richard Oudkerk added the comment: BTW, the context objects are singletons. I could not see a sensible way to make ctx.Process be a picklable class (rather than a method) if there can be multiple instances of a context type. This means that the helper processes survive until the program close

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Richard Oudkerk added the comment: I'll review the patch. (According to http://www.python.org/dev/peps/pep-0429/ feature freeze is expected in late November, so there is not too much of rush.) -- ___ Python tracker

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Lars Buitinck added the comment: Ok. Do you (or jnoller?) have time to review my proposed patch, at least before 3.4 is released? I didn't see it in the release schedule, so it's probably not planned soon, but I wouldn't want the API to change *again* in 3.5. -- __

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Richard Oudkerk added the comment: There are lots of things that behave differently depending on the currently set start method: Lock(), Semaphore(), Queue(), Value(), ... It is not just when creating a Process or Pool that you need to know the start method. Passing a context or start_method

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Olivier Grisel added the comment: Richard Oudkerk: thanks for the clarification, that makes sense. I don't have the time either in the coming month, maybe later. -- ___ Python tracker _

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Olivier Grisel added the comment: The process pool executor [1] from the concurrent futures API would be suitable to explicitly start and stop the helper process for the `forkserver` mode. [1] http://docs.python.org/3.4/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor Th

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Richard Oudkerk added the comment: By "context" I did not really mean a context manager. I just meant an object (possibly a singleton or module) which implements the same interface as multiprocessing. (However, it may be a good idea to also make it a context manager whose __enter__() method

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Lars Buitinck added the comment: I don't really see the benefit of a context manager over an argument. It's a power user feature anyway, and context managers (at least to me) signal cleanup actions, rather than construction options. -- ___ Python tr

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Changes by Lars Buitinck : -- nosy: +jnoller ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Olivier Grisel added the comment: > Maybe it would be better to have separate contexts for each start method. > That way joblib could use the forkserver context without interfering with the > rest of the user's program. Yes in general it would be great if libraries could customize the multip

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Richard Oudkerk added the comment: > In my patched version, the private popen.get_start_method gets a kwarg > set_if_needed=True. popen.Popen calls that as before, so its behavior > should not change, while the public get_start_method sets the kwarg to > False. My mistake. -- __

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Lars Buitinck added the comment: In my patched version, the private popen.get_start_method gets a kwarg set_if_needed=True. popen.Popen calls that as before, so its behavior should not change, while the public get_start_method sets the kwarg to False. I realise now that this has the side effec

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Lars Buitinck added the comment: Cleaned up the patch. -- Added file: http://bugs.python.org/file31722/mp_getset_start_method.patch ___ Python tracker ___ ___

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Changes by Lars Buitinck : Removed file: http://bugs.python.org/file31721/mp_getset_start_method.patch ___ Python tracker ___ ___ Python-bugs-

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Richard Oudkerk added the comment: With your patch, I think if you call get_start_method() without later calling set_start_method() then the helper process(es) will never be started. With the current code, popen.Popen() automatically starts the helper processes if they have not already been st

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Olivier Grisel added the comment: Related question: is there any good reason that would prevent to pass a custom `start_method` kwarg to the `Pool` constructor to make it use an alternative `Popen` instance (that is an instance different from the `multiprocessing._Popen` singleton)? This wou

[issue18999] Robustness issues in multiprocessing.{get, set}_start_method

Changes by Lars Buitinck : -- title: Allow multiple calls to multiprocessing.set_start_method -> Robustness issues in multiprocessing.{get,set}_start_method ___ Python tracker _