[issue7123] Multiprocess Process does not always exit when run from a thread.
Charles-François Natali neolo...@free.fr added the comment: I am pretty sure this is another instance of issue 6721. Indeed. Closing as duplicate. -- status: open - closed superseder: - Locks in python standard library should be sanitized on fork ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +neologix ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Tomaž Šolc tomaz.s...@tablix.org added the comment: I am pretty sure this is another instance of issue 6721. multiprocessing is forking a new process and if that happens while the other thread is holding the lock for stdout, the process will deadlock because the lock state is replicated in the subprocess while the thread that is supposed unlock it isn't (even if you remove all prints, the deadlock will happen on a stdout flush). Attaching a gdb to a hung subprocess confirms that it is waiting for a lock in the bufferedio module. I've reproduced this on Python trunk running on Linux (using pthreads) and can confirm that the patch attached to issue 6721 fixes it. I'm also attaching a shorter test case that omits unused code from the original example. -- nosy: +avian Added file: http://bugs.python.org/file22465/testing.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Changes by Ask Solem a...@opera.com: -- nosy: +asksol ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Peter Saunders p...@fodder.org.uk added the comment: Tested on Linux, and repeated the issue - just to confirm this isn't a Solaris specific issue. Tested using dummy_threading - bug does not appear when that is used. Added _verbose=True to threads, and log_to_stderr(SUBDEBUG) set for Process. Attached is the output of the script with those set. Other odd things to notice, and some point, more than 2 children sometimes exist.. isalive PID: 7646 : True isalive PID: 7652 : True isalive PID: 7653 : True isalive PID: 7646 : True isalive PID: 7652 : True isalive PID: 7653 : True isalive PID: 7646 : True isalive PID: 7653 : True The code starts 2 threads, which start 2 processes. Both threads are joined before new ones are started.. So, in theroy, only 2 should be able to exist at once.. ? -- Added file: http://bugs.python.org/file15132/results1.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
New submission from Peter Saunders p...@fodder.org.uk: I have an example code that works fine on Python 2.6.3, but when run in Python 3.1.1 - after a very short period of time, will go wrong. Summary: We have a queue, and when the queue has something in it (a list), we start a thread to deal with that entry. The thread then runs a Process for every entry in the list we popped off the queue. The Process target takes a Pipe, and it sends True back down the pipe, and then exits. However, this Process never goes defunct, and thus the Thread that started the Process never reaps it. When doing a truss on the Process, it sits there in: lwp_park(0x, 0) (sleeping...) -- files: testing.py messages: 93978 nosy: p...@fodder.org.uk severity: normal status: open title: Multiprocess Process does not always exit when run from a thread. versions: Python 3.1 Added file: http://bugs.python.org/file15124/testing.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Peter Saunders p...@fodder.org.uk added the comment: Further information: it doesn't fail everytime in Python 3.1 - usually 1 in 4, or 1 in 5 times. It never fails with Python 2.6.3 Example output from the script when its failing (python 3.1): Starting data1 Starting data2 Started subproc: PID: 20209 : args: data1 poll Started subproc: PID: 20210 : args: data2 poll Child has sent:True poll Child has sent:True poll isalive: False isalive: True Reaping PID: 20210 isalive: True poll Finishing: data2 isalive: True poll isalive: True poll isalive: True poll isalive: True poll repeat isalive and poll lines until: isalive: True poll isalive: True terminating Child had to be killed due to a timeout Finishing: data1 True Working output (consistantly works on Python 2.6.3): Starting data1 Starting data2 Started subproc: PID: 20252 : args: data1 poll Started subproc: PID: 20253 : args: data2 poll Child has sent:True Child has sent:True poll Child has sent:True poll isalive: False isalive: False isalive: False Reaping PID: 20252 Reaping PID: 20253 Finishing: data1 Finishing: data2 True -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Changes by Antoine Pitrou pit...@free.fr: -- assignee: - jnoller components: +Library (Lib) nosy: +jnoller priority: - normal type: - behavior versions: +Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Antoine Pitrou pit...@free.fr added the comment: Acccording to http://blogs.sun.com/chrisg/entry/lwp_park_and_lwp_unpark, the lwp_park() call could point to a mutex which is waiting to be acquired. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Peter Saunders p...@fodder.org.uk added the comment: Well, if it helps, here is the output of the dtrace script from starting of a loop with the failure, and stopping during the failure. -- Added file: http://bugs.python.org/file15126/dtrace.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Antoine Pitrou pit...@free.fr added the comment: Does it not happen if you call your checkAlive() function directly from your main() function? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Antoine Pitrou pit...@free.fr added the comment: As for the dtrace output: I'm not a Solaris expert unfortunately, I was just trying to suggest a possible direction for diagnosing this problem. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Peter Saunders p...@fodder.org.uk added the comment: If you mean, in main() instead of doing: while True: q.put([data1, data2]) t = Process(target=popJobs, args=(q, )) t.start() t.join() and doing: while True: q.put([data1, data2]) popJobs(q) instead. Then, the bug does indeed occur the same way. It did take more iterations before it occured however. If however, you meant: while True: fail=failureObject() for x in [data1, data2]: checkAlive(fail, x) print(fail.status()) Then the bug never occurs. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Peter Saunders p...@fodder.org.uk added the comment: Further oddness: When running the script (i've reduced its size further now, see attached): I get the following output: Reaping PID: 23215 True Started subproc: PID: 23216 : args: data1 Started subproc: PID: 23216 : args: data1 Started subproc: PID: 23217 : args: data2 poll isalive: True isalive: True poll Notice the Started subprod of the SAME PID and the Same args twice, yet this print only occurs once in the code, and I can't see how this should happen? -- Added file: http://bugs.python.org/file15127/testing.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Antoine Pitrou pit...@free.fr added the comment: Notice the Started subprod of the SAME PID and the Same args twice, yet this print only occurs once in the code, and I can't see how this should happen? This is a thread-safety issue in sys.stdout/stderr, it will be fixed in 3.1.2 (see issue6750). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7123] Multiprocess Process does not always exit when run from a thread.
Peter Saunders p...@fodder.org.uk added the comment: Sorry for the spam on the updates :) - but, its the same thread printing this out too. I changed the print line to: curThread = threading.current_thread() print(Started subproc: PID: %d : args: %s Thread ID: %s %(newJob.pid, str(args), str(curThread.ident))) and got the output: Reaping PID: 23707 True Started subproc: PID: 23709 : args: data1 Thread ID: 12 Started subproc: PID: 23709 : args: data1 Thread ID: 12 poll isalive: True Started subproc: PID: 23710 : args: data2 Thread ID: 13 poll isalive: True Again, I can't see how this could pring it out twice? Could this be related to the problem we are having? Altgough I do seem to get the line twice on 2.6.3 as well - but nothing like as often.. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
