[issue14404] multiprocessing with maxtasksperchild: bug in control logic?
Changes by Jesse Noller : -- nosy: +asksol, jnoller ___ Python tracker <http://bugs.python.org/issue14404> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13812] multiprocessing package doesn't flush stderr on child exception
Jesse Noller added the comment: There's already a bug / pending patch for this behavior here: http://bugs.python.org/issue8713 No need to take it to -ideas. -- ___ Python tracker <http://bugs.python.org/issue13812> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Jesse Noller added the comment: On Wednesday, December 21, 2011 at 10:04 AM, Charles-François Natali wrote: While I would tend to agree with you in theory - I don't think we should make it the default - at least not without a LOT of lead time. There's a surprising amount of code relying on the current behavior that I think the best course is to enable this option, and change the docs to steer users in this direction. For users jumping from 2.x into 3.x, I think the less surprises they have the better, and changing the default behavior of the stdlib module in this was would qualify as surprising. -- ___ Python tracker <http://bugs.python.org/issue8713> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13139] multiprocessing.map skips finally blocks
Jesse Noller added the comment: Antoine is correct, as he usually is. This is more of a documentation issue than bug. -- ___ Python tracker <http://bugs.python.org/issue13139> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10348] multiprocessing: use SysV semaphores on FreeBSD
Jesse Noller added the comment: Charles and Antoine's votes match my own, therefore closing the bug wont fix -- resolution: -> wont fix status: open -> closed ___ Python tracker <http://bugs.python.org/issue10348> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Jesse Noller added the comment: No one is currently working on a patch AFAIK -- ___ Python tracker <http://bugs.python.org/issue8713> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4106] multiprocessing occasionally spits out exception during shutdown
Jesse Noller added the comment: On Wed, Aug 24, 2011 at 3:01 PM, Antoine Pitrou wrote: > > Antoine Pitrou added the comment: > > I can't seem to reproduce this under 3.3. Should it be closed? I don't think so; it's still applicable to 2.x, and a fix should go into 2.7 ideally. http://bugs.python.org/issue9207 is the source of the issue AFAIR -- ___ Python tracker <http://bugs.python.org/issue4106> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11657] multiprocessing_{send,recv}fd fail with fds > 256
Jesse Noller added the comment: Charles; you have +commit, it seems. I would welcome the patch and test (just as long as the aforementioned reliance on /etc/fstab was removed). -- ___ Python tracker <http://bugs.python.org/issue11657> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11657] multiprocessing_{send,recv}fd fail with fds > 256
Jesse Noller added the comment: Yes, Charles - the test is not only welcome, but needed - it just can't rely on reading /etc/fstab ;) -- ___ Python tracker <http://bugs.python.org/issue11657> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6056] socket.setdefaulttimeout affecting multiprocessing Manager
Jesse Noller added the comment: I agree derek, I think that would be a fine addition, however we lack a patch and I don't have the current bandwidth to add it. -- ___ Python tracker <http://bugs.python.org/issue6056> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11969] Can't launch multiproccessing.Process on methods
Jesse Noller added the comment: Do things like this in the REPL are known not to work. Things are not protected in the if __name__ block so we can import the script properly to run it. -- ___ Python tracker <http://bugs.python.org/issue11969> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10632] multiprocessing generates a fatal error
Jesse Noller added the comment: Dupe of issue10517 -- resolution: -> duplicate status: open -> closed ___ Python tracker <http://bugs.python.org/issue10632> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10332] Multiprocessing maxtasksperchild results in hang
Jesse Noller added the comment: > Note: I noticed that there are some thread-unsafe operations (the cache that > can be modified from different threads, and thread states are modified also > from different threads). While this isn't an issue with the current cPython > implementation (GIL), I wonder if this should be fixed. > Yes. We should fix those. -- ___ Python tracker <http://bugs.python.org/issue10332> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11750] Mutualize win32 functions
Jesse Noller added the comment: Agreed; I'm not personally the windows expert that should handle that consolidation though. -- ___ Python tracker <http://bugs.python.org/issue11750> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11743] Rewrite PipeConnection and Connection in pure Python
Jesse Noller added the comment: Nothing jumps out at me at initial review; I've asked other contributors/interested parties to take a look too. Thanks a ton Antoine for doing this work -- ___ Python tracker <http://bugs.python.org/issue11743> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Jesse Noller added the comment: On Thu, Mar 31, 2011 at 9:05 AM, Antoine Pitrou wrote: > > Antoine Pitrou added the comment: > > Possible plan for POSIX, where a connection uses a pipe() or socketpair(): > exploit the fact that an endpoint becomes ready for reading (indicating EOF) > when the other endpoint is closed: > >>>> r, w = os.pipe() >>>> select.select([r], [], [r], 0) > ([], [], []) >>>> os.close(w) >>>> select.select([r], [], [r], 0) > ([4], [], []) > >>>> a, b = socket.socketpair() >>>> select.select([b], [], [b], 0) > ([], [], []) >>>> a.close() >>>> select.select([b], [], [b], 0) > ([], [], []) > > So, each Process could have a sentinel fd in the parent process, which > becomes ready when the process exits. These sentinel fds can be used in the > various select() calls underlying Queue.get(). > > (I don't understand why _multiprocessing/socket_connection.c in written in C. > Rewriting it in Python would make improvements much easier) I concur with you. The only reason it is in C is legacy; I don't see an immediate reason why it should continue to be that way -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Jesse Noller added the comment: On Thu, Mar 31, 2011 at 8:25 AM, Antoine Pitrou wrote: > > Antoine Pitrou added the comment: > > Speaking of which, I wonder why we have both multiprocessing.Pool and > concurrent.futures.ProcessPoolExecutor. They seem to fill the exact same > needs, with different APIs... > Brian and I need to work on the consolidation we intend(ed) to occur as people got comfortable with the APIs. My eventual goal is to remove anything but the basic multiprocessing.Process/Queue stuff out of MP and into concurrent.* and support threading backends for it. -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11675] multiprocessing Arrays not automatically zeroed.
Jesse Noller added the comment: I concur with Antoine. I think you're good to go Mark. -- ___ Python tracker <http://bugs.python.org/issue11675> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11569] multiprocessing/darwin: sysctl(8) mislocation
Jesse Noller added the comment: Ronald - if you're OK w/ patch 2, please commit! -- ___ Python tracker <http://bugs.python.org/issue11569> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11358] Please replace the use of pickle in multiprocessing with json.
Jesse Noller added the comment: I concur with RDM. We need complex data structures, and switching to JSON represents a non zero amount of work, isn't as fast and pickle works well. If you want to use JSON as a transport, I would do custom subclassing. -- ___ Python tracker <http://bugs.python.org/issue11358> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3244] multipart/form-data encoding
Jesse Noller added the comment: Yeah, despite what the RFC says, the most common usage is in web clients, and stuffing it in the email module won't be obvious to 95% of the population I think, unless that's where the implementation lives, but we can add a doc stub in the http docs pointing to it and why. -- ___ Python tracker <http://bugs.python.org/issue3244> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10845] test_multiprocessing failure under Windows
Jesse Noller added the comment: I'm fine with that tweak antoine -- ___ Python tracker <http://bugs.python.org/issue10845> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4106] multiprocessing occasionally spits out exception during shutdown
Jesse Noller added the comment: On Tue, Jan 18, 2011 at 6:23 PM, Brian Thorne wrote: > > Brian Thorne added the comment: > > With the example script attached I see the exception every time. On Ubuntu > 10.10 with Python 2.6 > > Since the offending line in multiprocesing/queues.py (233) is a debug > statement, just commenting it out seems to stop this exception. > > Looking at the util file shows the logging functions to be all of the form: > > if _logger: > _logger.log(... > > Could it be possible that after the check the _logger global (or the debug > function) is destroyed by the exit handler? Can we convince them to stick > around until such a time that they cannot be called? > > Adding a small delay before joining also seems to work, but is ugly. Why > should another Process *have* to have a minimum amount of work to not throw > an exception? See http://bugs.python.org/issue9207 - but yes, the problem is that the VM is nuking our imported modules before all the processes are shutdown. -- ___ Python tracker <http://bugs.python.org/issue4106> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10348] multiprocessing: use SysV semaphores on FreeBSD
Jesse Noller added the comment: Adding, or moving, to SYSV semaphores is very low on the list of things to do. If someone were to provide a patch, I'm sure we could consider it. -- ___ Python tracker <http://bugs.python.org/issue10348> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5725] process SysV-Semaphore support
Changes by Jesse Noller : -- nosy: +jnoller ___ Python tracker <http://bugs.python.org/issue5725> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10527] multiprocessing.Pipe problem: "handle out of range in select()"
Changes by Jesse Noller : -- nosy: +asksol ___ Python tracker <http://bugs.python.org/issue10527> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9162] License for multiprocessing files
Jesse Noller added the comment: We have to add the BSD header and maintain the copyright clause on all of the multiprocessing files. Apologies for the delay -- ___ Python tracker <http://bugs.python.org/issue9162> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8028] self.terminate() from a multiprocessing.Process raises AttributeError exception
Jesse Noller added the comment: Can you please expand on "deeply different"? -- ___ Python tracker <http://bugs.python.org/issue8028> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8028] self.terminate() from a multiprocessing.Process raises AttributeError exception
Jesse Noller added the comment: Fine w/ committing this Ask as-is ask. You are correct in the original intent of the code. -- ___ Python tracker <http://bugs.python.org/issue8028> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7707] multiprocess.Queue operations during import can lead to deadlocks
Jesse Noller added the comment: Fine w/ committing this Ask. -- ___ Python tracker <http://bugs.python.org/issue7707> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9244] multiprocessing.pool: Worker crashes if result can't be encoded
Jesse Noller added the comment: Fine w/ committing this Ask. -- ___ Python tracker <http://bugs.python.org/issue9244> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6645] multiprocessing build fails on AIX - /dev/urandom (or equivalent) not found
Jesse Noller added the comment: Closing per Sridhar -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue6645> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9935] Faster pickling of instances
Jesse Noller added the comment: On Mon, Oct 25, 2010 at 10:17 AM, Antoine Pitrou wrote: > Well, I'm not asking anyone to rewrite the entire multiprocessing test suite; > and, besides, I've provided a patch myself to improve it in that respect ;) > (in issue10173) I just saw that one - I'll poke at that next > Of course, it also means the present pickle patch is imperfect, though the > result of __reduce__ in this case looks more like a side-effect of an > implementation detail than documented behaviour (since the result isn't > usable anyway). I may try to come up with a better patch before the 3.2 beta > but it's not sure I will find enough time/motivation. Okie doke. -- ___ Python tracker <http://bugs.python.org/issue9935> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6269] threading documentation makes no mention of the GIL
Changes by Jesse Noller : -- assignee: jnoller -> nobody keywords: +easy nosy: +nobody priority: normal -> low ___ Python tracker <http://bugs.python.org/issue6269> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6645] multiprocessing build fails on AIX - /dev/urandom (or equivalent) not found
Jesse Noller added the comment: Sridhar can you confirm if this is still a problem on 3.2? -- ___ Python tracker <http://bugs.python.org/issue6645> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9935] Faster pickling of instances
Jesse Noller added the comment: I doubt I, or Ask will have the time to rewrite the entire multiprocessing test suite right now to work around the change Antoine. -- ___ Python tracker <http://bugs.python.org/issue9935> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10015] Creating a multiproccess.pool.ThreadPool from a child thread blows up.
Jesse Noller added the comment: I can not, for the life of me, remember why ThreadPool is there, except as a fallback. It's also not part of the documented interface as well. Additionally, in Python 3 we now have futures. -- ___ Python tracker <http://bugs.python.org/issue10015> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9897] multiprocessing problems
Jesse Noller added the comment: hume; filing this again doesn't help. I closed issue 9851 as a duplicate of this for you. -- ___ Python tracker <http://bugs.python.org/issue9897> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9851] multiprocessing socket timeout will break client
Jesse Noller added the comment: Duplicate; 9897 -- resolution: -> duplicate status: open -> closed ___ Python tracker <http://bugs.python.org/issue9851> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9162] License for multiprocessing files
Jesse Noller added the comment: On Thu, Sep 16, 2010 at 8:15 AM, Michael Fladischer wrote: > > Michael Fladischer added the comment: > > Is there any timeline on when this will be fixed? It's currently blocking > work on a python-related package for Debian. > I'll ping the PSF board for the legal recommendations again. -- ___ Python tracker <http://bugs.python.org/issue9162> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9851] multiprocessing socket timeout will break client
Jesse Noller added the comment: Calling it stupid doesn't incentivize me to help you, or fix it. -- ___ Python tracker <http://bugs.python.org/issue9851> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6407] multiprocessing Pool should allow custom task queue
Jesse Noller added the comment: > are there really any test/doc changes needed for this? Yes. At bare minimum we need unit tests for the new possible usage of the API; I'd like docs and an example which show the new usage as well. -- ___ Python tracker <http://bugs.python.org/issue6407> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8296] multiprocessing.Pool hangs when issuing KeyboardInterrupt
Jesse Noller added the comment: > If we keep that behavior, the real problem here is that the > result handler hangs if the process that reserved a job is gone, which is > going to be handled > by #9205. Should we mark it as a duplicate? I would tend to agree with your assessment; we're better served just gracefully handling everything per 9205 -- ___ Python tracker <http://bugs.python.org/issue8296> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9663] importlib should exclusively open bytecode files
Jesse Noller added the comment: Nice. -- nosy: +jnoller ___ Python tracker <http://bugs.python.org/issue9663> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9592] Limitations in objects returned by multiprocessing Pool
Jesse Noller added the comment: Thanks Freek - we're actually discussing some stuff like this in issue9205 as well -- ___ Python tracker <http://bugs.python.org/issue9592> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9586] "warning: comparison between pointer and integer" in multiprocessing build on Tiger
Jesse Noller added the comment: looks fine mark -- ___ Python tracker <http://bugs.python.org/issue9586> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8428] buildbot: test_multiprocessing timeout (test_notify_all? test_pool_worker_lifetime?)
Jesse Noller added the comment: Is this intermittent, or consistently failing? Updating it with more buildbot failures doesn't help. -- ___ Python tracker <http://bugs.python.org/issue8428> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9592] Limitations in objects returned by multiprocessing Pool
Jesse Noller added the comment: Florent - Are you running the script from Freek on the buildbots, or are you just updating this bugs with other run failures? I'm having a really hard time separating things. -- ___ Python tracker <http://bugs.python.org/issue9592> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9586] "warning: comparison between pointer and integer" in multiprocessing build on Tiger
Jesse Noller added the comment: I agree with Antoine -- ___ Python tracker <http://bugs.python.org/issue9586> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8426] multiprocessing.Queue fails to get() very large objects
Jesse Noller added the comment: I don't know that it's unreasonable to send that much data, but it would certainly be slow, and I would not recommend it. Therefore, this is still on the list for when I have time -- ___ Python tracker <http://bugs.python.org/issue8426> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8184] multiprocessing.managers will not fail if listening ocket already in use
Jesse Noller added the comment: Mark - did you observe the behavior in the Op? He's not stating the the code snippet runs fine, but that the second run at the same time on windows to cause a conflict. We need to show that either running it twice, at the same time against the same socket fails on windows as it should or we still have a bug -- status: pending -> open ___ Python tracker <http://bugs.python.org/issue8184> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9513] test_multiprocessing skipped on Windows
Jesse Noller added the comment: The relative imports have been more of a pain then they've helped. I'm fine with nuking them so long as the test suite passes. -- ___ Python tracker <http://bugs.python.org/issue9513> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6720] multiprocessing logging
Jesse Noller added the comment: The backport of multiprocessing is currently stale; and there are a few bugs in the tracker assigned to christian or myself in regards to it. If it's not too much trouble, I'd leave this one alone until the exact future of the backport can be resolved. -- ___ Python tracker <http://bugs.python.org/issue6720> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions
Jesse Noller added the comment: @ray - you probably don't have the "dcon" binary on your path. bug.py calls a subprocess call. -- ___ Python tracker <http://bugs.python.org/issue9400> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5001] Remove assertion-based checking in multiprocessing
Jesse Noller added the comment: @Mark Yeah - I'm the current multiprocessing maintainer. If I fix it, I'll just commit it :) I filed this as a to do against myself. -- ___ Python tracker <http://bugs.python.org/issue5001> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Jesse Noller added the comment: (sorry, I thought I had replied to your comment when I hadn't!) I think we can get away with a new optional kwarg. -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Jesse Noller added the comment: You two are bigger users of this then I currently am (the curse/blessing of switching jobs), which is why I've let you hash it out. Let me point out: my goal is to deal with errors in a way which does not cause a total crash, a lockup, or hanging processes. Whether or not we lose jobs is another thing entirely, and something I'm torn on. I also need to mention - I think we can add fixes to the behavior to 2.7 - we can not, however, change the API. If we change the API, this fix will be only on Python 3.2 which is not what I suspect either of you want. -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Jesse Noller added the comment: Passing the references seems to be a losing game; for _handle_workers - we only need 1 function (debug) - for others (say _join_exited_workers), we need references to reversed/range/len. A possible alternative is to make those threads non-daemon threads; but I'd have to test that. -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9244] multiprocessing.pool: Worker crashes if result can't be encoded
Changes by Jesse Noller : -- assignee: -> jnoller nosy: +gdb ___ Python tracker <http://bugs.python.org/issue9244> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Jesse Noller added the comment: Greg - I asked Ask to take a look - his celery package is a huge consumer of multiprocessing, and so I tend to run things past him as well. That said - to both of you - the fundamental problem the shutdown patch is trying to scratch is located in issue 9207 - greg's termination patch just exposes the problem in 9207 a lot more. Focusing specifically on the shutdown patch; our issue is that during interpreter shutdown, sys.modules is iterated, and entries are set to None - for threads which "live on" well into that cycle can end up losing imported functions/modules/etc. The multiple daemon threads in the Pool code are exposing this as code which executed imported functions (such as the debug() statement in handle_workers) which will fire after the pool has exited and the interpreter is shut down. We can work around the shutdown issue (really, bug 9207) by ignoring the exception such as shutdown.patch does, or passing in references/adding references to the functions those methods need. Or (as Brett suggested) converting them to class methods and adding references to the class. Or passing them in via the signature like this _handle_workers(arg, _debug=debug), etc. -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9207] multiprocessing occasionally spits out exception during shutdown (_handle_workers)
Jesse Noller added the comment: Thank you for doing that footwork Greg, it means a lot to me. I'm leaning towards the patch to swallow the errors - I just wanted to ponder it just a tiny bit longer before I pull the trigger. -- ___ Python tracker <http://bugs.python.org/issue9207> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Jesse Noller added the comment: A+ for creativity; I wouldn't have thought of that ;) -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Jesse Noller added the comment: Ugh. I'm going to have to think about the cleanest way of handling this case of functions vanishing from us since this is going to be more widespread inside the code. Suggestions welcome. -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Changes by Jesse Noller : -- assignee: -> jnoller ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9205] Parent process hanging in multiprocessing if children terminate unexpectedly
Jesse Noller added the comment: thanks greg; I'm going to take a look and think about this. I'd like to resolve bug 9207 first though -- ___ Python tracker <http://bugs.python.org/issue9205> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9207] multiprocessing occasionally spits out exception during shutdown (_handle_workers)
Jesse Noller added the comment: Talking with Brett; the fix should be as simple as keeping a reference to the debug function which we have in the imports. During interpreter shutdown, the sys.modules is iterated and each module replaced with None. Since the _handle_workers thread persists slightly past the point of the parent (and can, it's a daemon thread) debug is vanishing on us. We can go with switching this to a classmethod, and keeping a reference on the class, passing debug directly into the _handle_workers thread (testing this last night fixed it 100% of the time) -- ___ Python tracker <http://bugs.python.org/issue9207> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9207] multiprocessing occasionally spits out exception during shutdown (_handle_workers)
Jesse Noller added the comment: Greg - yeah. it's the same problem. -- ___ Python tracker <http://bugs.python.org/issue9207> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9162] License for multiprocessing files
Jesse Noller added the comment: Egad No! I gotcha. I'll find out what the deal with the contrib. agreement is. Pretty sure we had to wait on it when we brought it in. -- ___ Python tracker <http://bugs.python.org/issue9162> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9162] License for multiprocessing files
Jesse Noller added the comment: Hi Stefan - I'm familiar with the license/etc of the original module. This bug isn't urgent; I shot a quick email to the PSF's secretary Pat to confirm we have a contributor agreement however. -- ___ Python tracker <http://bugs.python.org/issue9162> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9207] multiprocessing occasionally spits out exception during shutdown (_handle_workers)
Jesse Noller added the comment: I'm not sure if there would still be the possibility; the thing which worries me is the debug() function vanishing on us - something not good is happening on interpreter shutdown. -- ___ Python tracker <http://bugs.python.org/issue9207> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9207] multiprocessing occasionally spits out exception during shutdown (_handle_workers)
Jesse Noller added the comment: Greg, can you comment out line 272 in Lib/multiprocessing/pool.py and tell me if you can reproduce? -- ___ Python tracker <http://bugs.python.org/issue9207> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9207] multiprocessing occasionally spits out exception during shutdown (_handle_workers)
Jesse Noller added the comment: Correction; it can and does happen on OS/X. So, this is not a platform specific bug. -- ___ Python tracker <http://bugs.python.org/issue9207> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9207] multiprocessing occasionally spits out exception during shutdown (_handle_workers)
Jesse Noller added the comment: It does not seem to appear on OS/X 10.6.4 - so the only question is does this show up on Ubuntu 32bit -- ___ Python tracker <http://bugs.python.org/issue9207> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9207] multiprocessing occasionally spits out exception during shutdown (_handle_workers)
Jesse Noller added the comment: I can confirm with a clean ubuntu 64 install, with a clean checkout of release27 that it explodes with that exception, while the stock 2.6.5 does not. -- ___ Python tracker <http://bugs.python.org/issue9207> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9162] License for multiprocessing files
Jesse Noller added the comment: Yes; the copyright has to stay; but the license data can leave afaik. -- ___ Python tracker <http://bugs.python.org/issue9162> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9207] multiprocessing occasionally spits out exception during shutdown (_handle_workers)
Jesse Noller added the comment: Alright, I'm fighting ubuntu 64 bit in my vmware install right now, I'll see if I can get it up and running. -- ___ Python tracker <http://bugs.python.org/issue9207> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9207] multiprocessing occasionally spits out exception during shutdown (_handle_workers)
Jesse Noller added the comment: Wait - so, you are pulling svn trunk, compiling and running your test with the built python executable? I'm not following the "multiprocessing-from-trunk" distinction unless you're picking the module out of the tree / compiling it and then moving it into some other install. I might be being overly dense. You're running your test with cd src/tree/ && ./python - right? Also, what, if any, compile flags are you passing to the python build? -- ___ Python tracker <http://bugs.python.org/issue9207> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9207] multiprocessing occasionally spits out exception during shutdown (_handle_workers)
Jesse Noller added the comment: Oh, you mean the backport from google code? The person who stepped up to maintain that has not refreshed that in some time. I need to decide what to do with it long term. I'm pretty sure it's badly out of date. -- ___ Python tracker <http://bugs.python.org/issue9207> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9207] multiprocessing occasionally spits out exception during shutdown (_handle_workers)
Changes by Jesse Noller : -- title: multiprocessing occasionally spits out exception during shutdown -> multiprocessing occasionally spits out exception during shutdown (_handle_workers) ___ Python tracker <http://bugs.python.org/issue9207> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9207] multiprocessing occasionally spits out exception during shutdown
Jesse Noller added the comment: Thanks greg; so this affects 2.6 as well (not using the backport at all) -- assignee: -> jnoller nosy: +jnoller ___ Python tracker <http://bugs.python.org/issue9207> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4106] multiprocessing occasionally spits out exception during shutdown
Jesse Noller added the comment: Greg - this is actually a different exception then the original bug report; could you please file a new issue with the information you've provided? I'm going to need to find a 64bit ubuntu box as I don't have one right now. -- ___ Python tracker <http://bugs.python.org/issue4106> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4106] multiprocessing occasionally spits out exception during shutdown
Jesse Noller added the comment: Greg - what platform? -- ___ Python tracker <http://bugs.python.org/issue4106> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9144] Import error for multiprocessing in 2.7rc2 on Windows
Jesse Noller added the comment: Pushed it in r82489 - worked for me on Linux and OS/X. Please let me know if anything else comes up. -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue9144> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9144] Import error for multiprocessing in 2.7rc2 on Windows
Jesse Noller added the comment: the patch looks good to me - unless someone beats me to it, I'm going to commit it shortly to fix 2.7 -- ___ Python tracker <http://bugs.python.org/issue9144> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8995] Performance issue with multiprocessing queue (3.1 VS 2.6)
Jesse Noller added the comment: No - I don't know of anything which would trigger this in 3.1 off the top of my head. The performance degradation is pretty worrisome -- ___ Python tracker <http://bugs.python.org/issue8995> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Jesse Noller added the comment: I pretty much agree with (b) an argument - your gut instinct is correct - there's a long standing thread in python-dev which pretty much solidified my thinking about whether or not we need this (we do). Any patch has to be backwards compatible by the way, it can not alter the current default behavior, also, it has to target python3 as 2.7 is nearing final, and this is a behavioral change. -- ___ Python tracker <http://bugs.python.org/issue8713> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8713] multiprocessing needs option to eschew fork() under Linux
Jesse Noller added the comment: This is on my wish list; but I have not had time to do it. Patch welcome. -- ___ Python tracker <http://bugs.python.org/issue8713> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8333] test_multiprocessing: pickling failures
Jesse Noller added the comment: When did this problem start? -- ___ Python tracker <http://bugs.python.org/issue8333> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8237] multiprocessing.Queue() blocks program
Jesse Noller added the comment: multiprocessing.Queue.Put() acts the same as Queue.put() - if the queue is full, the put call "hangs" until the queue is no longer full. The process will not exit, as the Queue is full, and it's waiting in put. This works as designed, unless I'm missing something painfully obvious, which is entirely possible. -- resolution: -> invalid status: open -> closed ___ Python tracker <http://bugs.python.org/issue8237> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6963] Add worker process lifetime to multiprocessing.Pool - patch included
Jesse Noller added the comment: Committed to trunk in r77794 Merged to Py3k in r77795 -- resolution: -> fixed status: open -> closed ___ Python tracker <http://bugs.python.org/issue6963> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6963] Add worker process lifetime to multiprocessing.Pool - patch included
Jesse Noller added the comment: Attached is the ported patch for Python 2.7/trunk. Please review it to make sure I didn't completely flub anything. I noticed you had forgotten the maxtasksperchild argument in the unit test, so I added that. I also expanded the docs a little but, paraphrasing your original post w.r.t to the justification. Let me know what you think, or if you have issues. Once you give the go ahead, I will commit to trunk and then merge to Py3k. Once again, thanks for the patch -- Added file: http://bugs.python.org/file16004/mp_patch_6963_trunk.patch ___ Python tracker <http://bugs.python.org/issue6963> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7774] sys.executable: wrong location if zeroth command argument is modified.
Jesse Noller added the comment: I'm not the subprocess owner Tarek :( -- assignee: jnoller -> ___ Python tracker <http://bugs.python.org/issue7774> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6963] Add worker process lifetime to multiprocessing.Pool - patch included
Jesse Noller added the comment: I'm working on this now; I'm going to need to port the patch to trunk before moving forward with it. Shouldn't take me long. -- ___ Python tracker <http://bugs.python.org/issue6963> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7753] newgil backport
Changes by Jesse Noller : -- nosy: +jnoller ___ Python tracker <http://bugs.python.org/issue7753> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6963] Add worker process lifetime to multiprocessing.Pool - patch included
Jesse Noller added the comment: I'm fine with the functionality; I'm going to test it out and look at committing it by mid-week next week. I apologize, I've been pretty maxed out. -- ___ Python tracker <http://bugs.python.org/issue6963> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7498] test_multiprocessing test_rapid_restart fails if port 9999 already in use
Changes by Jesse Noller : -- assignee: -> jnoller ___ Python tracker <http://bugs.python.org/issue7498> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6615] multiprocessing logging support test
Jesse Noller added the comment: Yeah, I should have checked the tearDown stuff in the logging test suite -- ___ Python tracker <http://bugs.python.org/issue6615> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7383] test_multiprocessing leaks
Jesse Noller added the comment: I've backed it out on trunk and py3k. I've reopened the original bug to debug the refleak caused by the test. -- resolution: -> fixed status: open -> closed ___ Python tracker <http://bugs.python.org/issue7383> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6615] multiprocessing logging support test
Jesse Noller added the comment: I've commented out the test (therefore, reopening this) the test introduces a pretty bad refleak problem. Need to debug. -- resolution: fixed -> accepted status: closed -> open ___ Python tracker <http://bugs.python.org/issue6615> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7383] test_multiprocessing leaks
Jesse Noller added the comment: Well, that's alarming. I'll back that out for now - I'm fairly disturbed it introduced leaks that bad. -- ___ Python tracker <http://bugs.python.org/issue7383> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
