Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
On Sat, Aug 27, 2011 at 2:59 AM, Ask Solem a...@celeryproject.org wrote: On 26 Aug 2011, at 16:53, Antoine Pitrou wrote: Hi, I think that deprecating the use of threads w/ multiprocessing - or at least crippling it is the wrong answer. Multiprocessing needs the helper threads it uses internally to manage queues, etc. Removing that ability would require a near-total rewrite, which is just a non-starter. I agree that this wouldn't actually benefit anyone. Besides, I don't think it's even possible to avoid threads in multiprocessing, given the various constraints. We would have to force the user to run their main thread in an event loop, and that would be twisted (tm). I would focus on the atfork() patch more directly, ignoring multiprocessing in the discussion, and focusing on the merits of gps' initial proposal and patch. I think this could also be combined with Charles-François' patch. Regards Have to agree with Jesse and Antoine here. Celery (celeryproject.org) uses multiprocessing, is wildly used in production, and is regarded as stable software that have been known to run for months at a time only to be restarted for software upgrades. I have been investigating an issue for some time, that I'm pretty sure is caused by this. It occurs only rarely, so rarely I have not had any actual bug reports about it, it's just something I have experienced during extensive testing. The tone of the discussion on the bug tracker makes me think that I have been very lucky :-) Using the fork+exec approach seems like a much more realistic solution than rewriting multiprocessing.Pool and Manager to not use threads. In fact this is something I have been considering as a fix for the suspected issue for for some time. It does have implications that are annoying for sure, but we are already used to this on the Windows platform (it could help portability even). +3 (agreed to Jesse, Antoine and Ask here). The http://bugs.python.org/issue8713 described non-fork implementation that always uses subprocesses rather than plain forked processes is the right way forward for multiprocessing. -gps ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
+3 (agreed to Jesse, Antoine and Ask here). The http://bugs.python.org/issue8713 described non-fork implementation that always uses subprocesses rather than plain forked processes is the right way forward for multiprocessing. I see two drawbacks: - it will be slower, since the interpreter startup time is non-negligible (well, normally you shouldn't spawn a new process for every item, but it should be noted) - it'll consume more memory, since we lose the COW advantage (even though it's already limited by the fact that even treating a variable read-only can trigger an incref, as was noted in a previous thread) cf ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
2011/8/29 Charles-François Natali neolo...@free.fr: +3 (agreed to Jesse, Antoine and Ask here). The http://bugs.python.org/issue8713 described non-fork implementation that always uses subprocesses rather than plain forked processes is the right way forward for multiprocessing. I see two drawbacks: - it will be slower, since the interpreter startup time is non-negligible (well, normally you shouldn't spawn a new process for every item, but it should be noted) Yes; but spawning and forking are both slow to begin with - it's documented (I hope heavily enough) that you should spawn multiprocessing children early, and keep them around instead of constantly creating/destroying them. - it'll consume more memory, since we lose the COW advantage (even though it's already limited by the fact that even treating a variable read-only can trigger an incref, as was noted in a previous thread) cf Yes, it would consume slightly more memory; but the benefits - making it consistent across *all* platforms with the *same* restrictions gets us closer to the principle of least surprise. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
On Mon, 29 Aug 2011 13:03:53 -0400 Jesse Noller jnol...@gmail.com wrote: 2011/8/29 Charles-François Natali neolo...@free.fr: +3 (agreed to Jesse, Antoine and Ask here). The http://bugs.python.org/issue8713 described non-fork implementation that always uses subprocesses rather than plain forked processes is the right way forward for multiprocessing. I see two drawbacks: - it will be slower, since the interpreter startup time is non-negligible (well, normally you shouldn't spawn a new process for every item, but it should be noted) Yes; but spawning and forking are both slow to begin with - it's documented (I hope heavily enough) that you should spawn multiprocessing children early, and keep them around instead of constantly creating/destroying them. I think fork() is quite fast on modern systems (e.g. Linux). exec() is certainly slow, though. The third drawback is that you are limited to picklable objects when specifying the arguments for your child process. This can be annoying if, for example, you wanted to pass an OS resource. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
On Mon, Aug 29, 2011 at 1:16 PM, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 29 Aug 2011 13:03:53 -0400 Jesse Noller jnol...@gmail.com wrote: 2011/8/29 Charles-François Natali neolo...@free.fr: +3 (agreed to Jesse, Antoine and Ask here). The http://bugs.python.org/issue8713 described non-fork implementation that always uses subprocesses rather than plain forked processes is the right way forward for multiprocessing. I see two drawbacks: - it will be slower, since the interpreter startup time is non-negligible (well, normally you shouldn't spawn a new process for every item, but it should be noted) Yes; but spawning and forking are both slow to begin with - it's documented (I hope heavily enough) that you should spawn multiprocessing children early, and keep them around instead of constantly creating/destroying them. I think fork() is quite fast on modern systems (e.g. Linux). exec() is certainly slow, though. The third drawback is that you are limited to picklable objects when specifying the arguments for your child process. This can be annoying if, for example, you wanted to pass an OS resource. Regards Antoine. Yes, it is annoying; but again - this makes it more consistent with the windows implementation. I'd rather that restriction than the sanitization of the ability to use threading and multiprocessing alongside one another. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
Le lundi 29 août 2011 à 13:23 -0400, Jesse Noller a écrit : Yes, it is annoying; but again - this makes it more consistent with the windows implementation. I'd rather that restriction than the sanitization of the ability to use threading and multiprocessing alongside one another. That sanitization is generally useful, though. For example if you want to use any I/O after a fork(). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
On Mon, Aug 29, 2011 at 1:22 PM, Antoine Pitrou solip...@pitrou.net wrote: Le lundi 29 août 2011 à 13:23 -0400, Jesse Noller a écrit : Yes, it is annoying; but again - this makes it more consistent with the windows implementation. I'd rather that restriction than the sanitization of the ability to use threading and multiprocessing alongside one another. That sanitization is generally useful, though. For example if you want to use any I/O after a fork(). Oh! I don't disagree; I'm just against the removal of the ability to mix multiprocessing and threads; which it does internally and others do in every day code. The proposed removal of that functionality - using the two together - would leave users in the dust, and not needed if we patch http://bugs.python.org/issue8713 - which at it's core is just an addition flag. We could document the risk(s) of using the fork() mechanism which has to remain the default for some time. The point is, is that the solution to http://bugs.python.org/issue6721 should not be intertwined or cause a severe change in the multiprocessing module (e.g. rewriting from scratch), etc. I'm not arguing that both bugs should not be fixed. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
On Mon, Aug 29, 2011 at 8:16 PM, Antoine Pitrou solip...@pitrou.net wrote: On Mon, 29 Aug 2011 13:03:53 -0400 Jesse Noller jnol...@gmail.com wrote: Yes; but spawning and forking are both slow to begin with - it's documented (I hope heavily enough) that you should spawn multiprocessing children early, and keep them around instead of constantly creating/destroying them. I think fork() is quite fast on modern systems (e.g. Linux). exec() is certainly slow, though. On my system, the time it takes worker code to start is: 40 usec with thread.start_new_thread 240 usec with threading.Thread().start 450 usec with os.fork 1 ms with multiprocessing.Process.start 25 ms with subprocess.Popen to start a trivial script. so os.fork has similar latency to threading.Thread().start, while spawning is 100 times slower. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
On Mon, Aug 29, 2011 at 8:42 PM, Jesse Noller jnol...@gmail.com wrote: On Mon, Aug 29, 2011 at 1:22 PM, Antoine Pitrou solip...@pitrou.net wrote: That sanitization is generally useful, though. For example if you want to use any I/O after a fork(). Oh! I don't disagree; I'm just against the removal of the ability to mix multiprocessing and threads; which it does internally and others do in every day code. I am not familiar with the python-dev definition for deprecation, but when I used the word in the bug discussion I meant to advertize to users that they should not mix threading and forking since that mix is and will remain broken by design; I did not mean removal or crippling of functionality. “When I use a word,” Humpty Dumpty said, in rather a scornful tone, “it means just what I choose it to mean—neither more nor less.” - Through the Looking-Glass (btw, my tone is not scornful) And there is no way around it - the mix in general is broken, with an atfork mechanism or without it. People can choose to keep doing it in their every day code at their own risk, be it significantly high or insignificantly low. But the documentation should explain the problem clearly. As for the internal use of threads in the multiprocessing module I proposed a potential way to sanitize those particular worker threads: http://bugs.python.org/issue6721#msg140402 If it makes sense and entails changes to internal multiprocessing worker threads, those changes could be applied as bug fixes to Python 2.x and previous Python 3.x releases. This does not contradict adding now the feature to spawn, and to make it the only possibility in the future. I agree that this is the saner approach but it is a new feature not a bug fix. Nir ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
On 8/29/2011 3:41 PM, Nir Aides wrote: I am not familiar with the python-dev definition for deprecation, but Possible to planned eventual removal when I used the word in the bug discussion I meant to advertize to users that they should not mix threading and forking since that mix is and will remain broken by design; I did not mean removal or crippling of functionality. This would be a note or warning in the doc. You can suggest what and where to add something on an existing issue or a new one. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
On 26 Aug 2011, at 16:53, Antoine Pitrou wrote: Hi, I think that deprecating the use of threads w/ multiprocessing - or at least crippling it is the wrong answer. Multiprocessing needs the helper threads it uses internally to manage queues, etc. Removing that ability would require a near-total rewrite, which is just a non-starter. I agree that this wouldn't actually benefit anyone. Besides, I don't think it's even possible to avoid threads in multiprocessing, given the various constraints. We would have to force the user to run their main thread in an event loop, and that would be twisted (tm). I would focus on the atfork() patch more directly, ignoring multiprocessing in the discussion, and focusing on the merits of gps' initial proposal and patch. I think this could also be combined with Charles-François' patch. Regards Have to agree with Jesse and Antoine here. Celery (celeryproject.org) uses multiprocessing, is wildly used in production, and is regarded as stable software that have been known to run for months at a time only to be restarted for software upgrades. I have been investigating an issue for some time, that I'm pretty sure is caused by this. It occurs only rarely, so rarely I have not had any actual bug reports about it, it's just something I have experienced during extensive testing. The tone of the discussion on the bug tracker makes me think that I have been very lucky :-) Using the fork+exec approach seems like a much more realistic solution than rewriting multiprocessing.Pool and Manager to not use threads. In fact this is something I have been considering as a fix for the suspected issue for for some time. It does have implications that are annoying for sure, but we are already used to this on the Windows platform (it could help portability even). -- Ask Solem twitter.com/asksol | +44 (0)7713357179 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
Another face of the discussion is about whether to deprecate the mixing of the threading and processing modules and what to do about the multiprocessing module which is implemented with worker threads. On Tue, Aug 23, 2011 at 11:29 PM, Antoine Pitrou solip...@pitrou.netwrote: Le mardi 23 août 2011 à 22:07 +0200, Charles-François Natali a écrit : 2011/8/23 Antoine Pitrou solip...@pitrou.net: Well, I would consider the I/O locks the most glaring problem. Right now, your program can freeze if you happen to do a fork() while e.g. the stderr lock is taken by another thread (which is quite common when debugging). Indeed. To solve this, a similar mechanism could be used: after fork(), in the child process: - just reset each I/O lock (destroy/re-create the lock) if we can guarantee that the file object is in a consistent state (i.e. that all the invariants hold). That's the approach I used in my initial patch. For I/O locks I think that would work. There could also be a process-wide fork lock to serialize locks and other operations, if we want 100% guaranteed consistency of I/O objects across forks. - call a fileobject method which resets the I/O lock and sets the file object to a consistent state (in other word, an atfork handler) I fear that the complication with atfork handlers is that you have to manage their lifecycle as well (i.e., when an IO object is destroyed, you have to unregister the handler). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/nir%40winpdb.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
On Fri, Aug 26, 2011 at 3:18 AM, Nir Aides n...@winpdb.org wrote: Another face of the discussion is about whether to deprecate the mixing of the threading and processing modules and what to do about the multiprocessing module which is implemented with worker threads. There's a bug open - http://bugs.python.org/issue8713 which would offer non windows users the ability to avoid using fork() entirely, which would sidestep the problem outlined in the atfork() bug. Under windows, which has no fork() mechanism, we create a subprocess and then use pipes for intercommunication: nothing is inherited from the parent process except the state passed into the child. I think that deprecating the use of threads w/ multiprocessing - or at least crippling it is the wrong answer. Multiprocessing needs the helper threads it uses internally to manage queues, etc. Removing that ability would require a near-total rewrite, which is just a non-starter. I'd rather examine bug 8713 more closely, and offer this option for all users in 3.x and document the existing issues outlined in http://bugs.python.org/issue6721 for 2.x - the proposals in that bug are IMHO, out of bounds for a 2.x release. In essence; the issue here is multiprocessing's use of fork on unix without the following exec - which is what the windows implementation essentially does using subprocess. Adding the option to *not* fork changes the fundamental behavior on unix systems - but I fundamentally feel that it's a saner, and more consistent behavior for the module as a whole. So, I'd ask that we not talk about tearing out the ability to use MP and threads, or threads with MP - that would be crippling, and there's existing code in the wild (including multiprocessing itself) that uses this mix without issue - it's stripping out functionality for what is a surprising and painful edge case that rarely directly affects users. I would focus on the atfork() patch more directly, ignoring multiprocessing in the discussion, and focusing on the merits of gps' initial proposal and patch. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
Hi, I think that deprecating the use of threads w/ multiprocessing - or at least crippling it is the wrong answer. Multiprocessing needs the helper threads it uses internally to manage queues, etc. Removing that ability would require a near-total rewrite, which is just a non-starter. I agree that this wouldn't actually benefit anyone. Besides, I don't think it's even possible to avoid threads in multiprocessing, given the various constraints. We would have to force the user to run their main thread in an event loop, and that would be twisted (tm). I would focus on the atfork() patch more directly, ignoring multiprocessing in the discussion, and focusing on the merits of gps' initial proposal and patch. I think this could also be combined with Charles-François' patch. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
Hi all, Please consider this invitation to stick your head into an interesting problem: http://bugs.python.org/issue6721 Nir ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
2011/8/23, Nir Aides n...@winpdb.org: Hi all, Hello Nir, Please consider this invitation to stick your head into an interesting problem: http://bugs.python.org/issue6721 Just for the record, I'm now in favor of the atfork mechanism. It won't solve the problem for I/O locks, but it'll at least make room for a clean and cross-library way to setup atfork handlers. I just skimmed over it, but it seemed Gregory's atfork module could be a good starting point. cf ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
On Tue, 23 Aug 2011 20:43:25 +0200 Charles-François Natali cf.nat...@gmail.com wrote: Please consider this invitation to stick your head into an interesting problem: http://bugs.python.org/issue6721 Just for the record, I'm now in favor of the atfork mechanism. It won't solve the problem for I/O locks, but it'll at least make room for a clean and cross-library way to setup atfork handlers. I just skimmed over it, but it seemed Gregory's atfork module could be a good starting point. Well, I would consider the I/O locks the most glaring problem. Right now, your program can freeze if you happen to do a fork() while e.g. the stderr lock is taken by another thread (which is quite common when debugging). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
2011/8/23 Antoine Pitrou solip...@pitrou.net: Well, I would consider the I/O locks the most glaring problem. Right now, your program can freeze if you happen to do a fork() while e.g. the stderr lock is taken by another thread (which is quite common when debugging). Indeed. To solve this, a similar mechanism could be used: after fork(), in the child process: - just reset each I/O lock (destroy/re-create the lock) if we can guarantee that the file object is in a consistent state (i.e. that all the invariants hold). That's the approach I used in my initial patch. - call a fileobject method which resets the I/O lock and sets the file object to a consistent state (in other word, an atfork handler) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue 6721 Locks in python standard library should be sanitized on fork
Le mardi 23 août 2011 à 22:07 +0200, Charles-François Natali a écrit : 2011/8/23 Antoine Pitrou solip...@pitrou.net: Well, I would consider the I/O locks the most glaring problem. Right now, your program can freeze if you happen to do a fork() while e.g. the stderr lock is taken by another thread (which is quite common when debugging). Indeed. To solve this, a similar mechanism could be used: after fork(), in the child process: - just reset each I/O lock (destroy/re-create the lock) if we can guarantee that the file object is in a consistent state (i.e. that all the invariants hold). That's the approach I used in my initial patch. For I/O locks I think that would work. There could also be a process-wide fork lock to serialize locks and other operations, if we want 100% guaranteed consistency of I/O objects across forks. - call a fileobject method which resets the I/O lock and sets the file object to a consistent state (in other word, an atfork handler) I fear that the complication with atfork handlers is that you have to manage their lifecycle as well (i.e., when an IO object is destroyed, you have to unregister the handler). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
