[issue21998] asyncio: support fork

STINNER Victor added the comment: There is no activity for 2 years. Asyncio is mature now. It seems like users learnt how to work around this issue, or don't use fork() with asyncio. I close the issue. -- resolution: -> out of date stage: -> resolved status: open -> closed ___

[issue21998] asyncio: support fork

Yury Selivanov added the comment: I'll revisit this later. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue21998] asyncio: support fork

STINNER Victor added the comment: > I'm torn between 2 & 3. Guido, Victor, Martin, what do you think? Give up, document that fork() is not supported and close the issue :-) IMHO it's not worth it. -- ___ Python tracker

[issue21998] asyncio: support fork

Zac Medico added the comment: I'm not sure about possible use cases that might conflict with this approach, but using a separate event loop for each pid seems very reasonable to me, as follows: _default_policy = asyncio.get_event_loop_policy() _pid_loop = {} class MultiprocessingPolicy(async

[issue21998] asyncio: support fork

Changes by Guido van Rossum : -- nosy: -gvanrossum ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue21998] asyncio: support fork

Yury Selivanov added the comment: > I'm not sure why it would be debug-only. You usually don't fork() often, and > you don't have many event loops around, so the feature sounds cheap. I think you're right. If it's low or zero overhead we can have the check always enabled. -- ___

[issue21998] asyncio: support fork

Antoine Pitrou added the comment: I'm not sure why it would be debug-only. You usually don't fork() often, and you don't have many event loops around, so the feature sounds cheap. In any case, I'm not directly affected by this issue, I'm merely suggesting options. -- ___

[issue21998] asyncio: support fork

Yury Selivanov added the comment: > A compromise for the short term would be to detect fork in debug mode and raise an exception, and explain how to fix such bug. What do you think? I'd prefer to fix it properly in 3.7. -- ___ Python tracker

[issue21998] asyncio: support fork

STINNER Victor added the comment: A compromise for the short term would be to detect fork in debug mode and raise an exception, and explain how to fix such bug. What do you think? -- ___ Python tracker

[issue21998] asyncio: support fork

Yury Selivanov added the comment: > Possible answer: have a global WeakSet of event loops. In the child fork > handler, iterate over all event loops and "break" those that have already > been started. We can do this but only in debug mode. -- ___

[issue21998] asyncio: support fork

Antoine Pitrou added the comment: Possible answer: have a global WeakSet of event loops. In the child fork handler, iterate over all event loops and "break" those that have already been started. -- ___ Python tracker

[issue21998] asyncio: support fork

STINNER Victor added the comment: > The most reasonable IMHO would be for it to mark the event loop "broken" (or > closed?) in the child, to forbid any further use. Hum, the problem is that Python cannot guess if the event loop will be used in the child or the parent process :-/ The problem onl

[issue21998] asyncio: support fork

Antoine Pitrou added the comment: > Python 3.7 got as new os.register_at_fork() function. I don't know if it > could help: The most reasonable IMHO would be for it to mark the event loop "broken" (or closed?) in the child, to forbid any further use. By the way, on Python 3 (which is pretty mu

[issue21998] asyncio: support fork

STINNER Victor added the comment: Python 3.7 got as new os.register_at_fork() function. I don't know if it could help: https://docs.python.org/dev/library/os.html#os.register_at_fork Can we close this issue? Sorry, I lost track of this issue and I see no activity since the end of 2015...

[issue21998] asyncio: support fork

Adam Bishop added the comment: A note about this issue should really be added to the documentation - on OS X, it fails with the rather non-sensical "OSError: [Errno 9] Bad file descriptor", making this very hard to debug. I don't have any specific requirement for fork support in asyncio as it'

[issue21998] asyncio: support fork

Changes by Christian H : -- nosy: +Christian H ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyt

[issue21998] asyncio: support fork

Changes by Larry Hastings : -- nosy: -larry ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue21998] asyncio: support fork

Larry Hastings added the comment: I've remarked it as "normal" priority and moved it to 3.6. Not my problem anymore! :D -- priority: deferred blocker -> normal versions: +Python 3.6 -Python 3.4, Python 3.5 ___ Python tracker

[issue21998] asyncio: support fork

STINNER Victor added the comment: > Surely this is too late for 3.5? I'm not 100% convinced that asyncio must support fork, so it's too late :-) Anyway, we don't care, asyncio will be under provisional status for one more cycle (3.5) :-p -- ___ Pyt

[issue21998] asyncio: support fork

Larry Hastings added the comment: Surely this is too late for 3.5? -- nosy: +larry ___ Python tracker ___ ___ Python-bugs-list mailing

[issue21998] asyncio: support fork

Guido van Rossum added the comment: I don't actually know if the 5th option is possible. My strong requirement is that no matter what the child process does, the parent should still be able to continue using the loop. IMO it's better to leak a FD in the child than to close a resource owned by

[issue21998] asyncio: support fork

Martin Richard added the comment: 015-05-26 20:40 GMT+02:00 Yury Selivanov : > > Yury Selivanov added the comment: > The only solution to safely fork a process is to fix loop.close() to > check if it's called from a forked process and to close the loop in > a safe way (to avoid breaking the mast

[issue21998] asyncio: support fork

Yury Selivanov added the comment: > I would therefore, in the child after a fork, close the loop without > breaking the selector state (closing without unregister()'ing fds), unset > the default loop so get_event_loop() would create a new loop, then raise > RuntimeError. > > I can elaborate o

[issue21998] asyncio: support fork

Martin Richard added the comment: Hi, My patch was a variation of haypo's patch. The goal was to duplicate the loop and its internal objects (loop and self pipes) without changing much to its state from the outside (keeping callbacks and active tasks). I wanted to be conservative with this patch

[issue21998] asyncio: support fork

Yury Selivanov added the comment: > That's really the problem of the code that calls fork(), not directly of > the event loop. There are some very solid patterns around that (I've > written several in the distant past, and Unix hasn't changed that much :-). Alright ;) I'll draft a patch sometim

[issue21998] asyncio: support fork

Guido van Rossum added the comment: That's really the problem of the code that calls fork(), not directly of the event loop. There are some very solid patterns around that (I've written several in the distant past, and Unix hasn't changed that much :-). On Tue, May 26, 2015 at 11:01 AM, Yury Sel

[issue21998] asyncio: support fork

Yury Selivanov added the comment: > I don't understand. If the fork fails nothing changes right? I guess I'm > missing some context or use case. Maybe I'm wrong about this. My line of thoughts is: a failed fork() call is a bug in the program. Now, the master process will continue operating a

[issue21998] asyncio: support fork

Guido van Rossum added the comment: I don't understand. If the fork fails nothing changes right? I guess I'm missing some context or use case. -- ___ Python tracker ___

[issue21998] asyncio: support fork

Yury Selivanov added the comment: > I was thinking only in the child. The parent should be able to continue to > use the loop as if the fork didn't happen, right? Yes, everything should be fine. I'll rephrase my question: do you think there is a way (and need) to at least throw a warning in t

[issue21998] asyncio: support fork

Changes by Christian Heimes : -- nosy: +christian.heimes ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:

[issue21998] asyncio: support fork

Guido van Rossum added the comment: I was thinking only in the child. The parent should be able to continue to use the loop as if the fork didn't happen, right? -- ___ Python tracker __

[issue21998] asyncio: support fork

Yury Selivanov added the comment: > I think only (3) is reasonable -- raise RuntimeError. Just to be clear -- do we want to raise a RuntimeError in the parent, in the child, or both processes? -- ___ Python tracker

[issue21998] asyncio: support fork

Guido van Rossum added the comment: I think only (3) is reasonable -- raise RuntimeError. There are too many use cases to consider and the behavior of the selectors seems to vary as well. Apps should ideally not fork with an event loop open; the only reasonable thing to do after a fork with an

[issue21998] asyncio: support fork

Yury Selivanov added the comment: > How do other event loops handle fork? Twisted, Tornado, libuv, libev, libevent, etc. It looks like using fork() while an event loop is running isn't recommended in any of the above. If I understand the code correctly, libev & gevent reinitialize loops in th

[issue21998] asyncio: support fork

Changes by Yury Selivanov : -- priority: normal -> deferred blocker ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue21998] asyncio: support fork

STINNER Victor added the comment: How do other event loops handle fork? Twisted, Tornado, libuv, libev, libevent, etc. -- ___ Python tracker ___ _

[issue21998] asyncio: support fork

Martin Richard added the comment: The goal of the patch is to create a duplicate selector (a new epoll() structure with the same watched fds as the original epoll). It allows to remove fds watched in the child's loop without impacting the parent process. Actually, it's true that with the curre

[issue21998] asyncio: support fork

STINNER Victor added the comment: 2015-02-17 20:16 GMT+01:00 Charles-François Natali : > What would you do with the selector after fork(): register the FDs in > a new epoll, remove them? See the patch: +def _at_fork(self): +# don't unregister file descriptors: epoll is still

[issue21998] asyncio: support fork

Charles-François Natali added the comment: > @neologix: Would you be ok to add a *private* _at_fork() method to selectors > classes in Python 3.4 to fix this issue? Not really: after fork(), you're hosed anyway: """ Q6 Will closing a file descriptor cause it to be removed from all epol

[issue21998] asyncio: support fork

Martin Richard added the comment: In that case, I suggest a small addition to your patch that would do the trick: in unix_events.py: +def _at_fork(self): +super()._at_fork() +self._selector._at_fork() +self._close_self_pipe() +self._make_self_pipe() + becomes

[issue21998] asyncio: support fork

STINNER Victor added the comment: > It will (obviously) not work with python 3.4 since self._selector won't have > an _at_fork() method. asyncio doc contains: "The asyncio package has been included in the standard library on a provisional basis. Backwards incompatible changes (up to and includ

[issue21998] asyncio: support fork

Martin Richard added the comment: I read the patch, it looks good to me for python 3.5. It will (obviously) not work with python 3.4 since self._selector won't have an _at_fork() method. I ran the tests on my project with python 3.5a1 and the patch, it seems to work as expected: ie. when I clo

[issue21998] asyncio: support fork

STINNER Victor added the comment: Can someone review at_fork-2.patch? Martin: Can you please try my patch? -- ___ Python tracker ___

[issue21998] asyncio: support fork

Changes by STINNER Victor : -- title: asyncio: a new self-pipe should be created in the child process after fork -> asyncio: support fork ___ Python tracker ___