Changes by Nir Aides n...@winpdb.org:
--
title: malloc - threading module can deadlock after fork
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue874900
___
Changes by Bobby Impollonia bob...@gmail.com:
--
nosy: +bobbyi
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue874900
___
___
Python-bugs-list
Changes by Jesús Cea Avión j...@jcea.es:
--
nosy: +jcea
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue874900
___
___
Python-bugs-list mailing list
Gregory P. Smith [EMAIL PROTECTED] added the comment:
forkthread2.patch looks good to me. to be consistent shouldn't we also
apply that fix to 2.6?
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Le samedi 06 septembre 2008 à 22:27 +, Gregory P. Smith a écrit :
Gregory P. Smith [EMAIL PROTECTED] added the comment:
forkthread2.patch looks good to me. to be consistent shouldn't we also
apply that fix to 2.6?
Only the
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Committed in r66274, r66275. Thanks!
--
resolution: - fixed
status: open - closed
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
Gregory P. Smith [EMAIL PROTECTED] added the comment:
I backported the last bit from r66275 to release25-maint in r66279.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Just checked that the latest patch works on Windows as well.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Ok, with this patch the test passes under py3k.
Apart from the trivial typo (_Thread__stopped - _stopped), I had to
change print(...) to os.write(1, b...\n) in the tests as otherwise
subprocess wouldn't receive any output from the third test
Benjamin Peterson [EMAIL PROTECTED] added the comment:
I feel like that patch sort of avoids the problem by changing the test.
The test is hanging for some reason, so we should try to fix that, not
the test. :) I wonder if it has something to do with the various
deadlocks we are discovering in
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Benjamin, if you don't change the test, the deadlock problem is still
solved, it's just that the third test fails because the subprocess
stdout is empty instead of containing the desired value. It is *not*
because the subprocess doesn't print
Benjamin Peterson [EMAIL PROTECTED] added the comment:
On Thu, Sep 4, 2008 at 6:08 PM, Antoine Pitrou [EMAIL PROTECTED] wrote:
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Benjamin, if you don't change the test, the deadlock problem is still
solved, it's just that the third test
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Instead of os.write(), it is actually sufficient to sys.stdout.flush()
at the end of the subprocess. Patch attached.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
Changes by Antoine Pitrou [EMAIL PROTECTED]:
Added file: http://bugs.python.org/file11379/forkthread2.patch
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
Benjamin Peterson [EMAIL PROTECTED] added the comment:
This is also causing hangs in 2.5.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
___
Gregory P. Smith [EMAIL PROTECTED] added the comment:
it passes on release25-maint for me on linux. i don't see it hanging in
any of the 2.5 buildbots. looks like your r66003 change was a decent
fix for windows judging by the buildbot.
(fwiw, that test you patched in 66003 should probably use
Antoine Pitrou [EMAIL PROTECTED] added the comment:
If you remove the print() call from joining_func(), does it stop
hanging? It may be due to differences between the io library in py3k and
the builtin file objects in 2.x.
(looking at the date of the commit disabling the test, it is not related
Gregory P. Smith [EMAIL PROTECTED] added the comment:
backported to release25-maint in r65789.
--
resolution: - fixed
status: open - closed
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
Benjamin Peterson [EMAIL PROTECTED] added the comment:
One of tests is hanging in 3.0.
--
resolution: fixed -
status: closed - open
versions: +Python 3.0 -Python 2.5
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
Gregory P. Smith [EMAIL PROTECTED] added the comment:
specifically, test_3_join_in_forked_from_thread hangs in py3k and is
currently disabled.
--
assignee: gregory.p.smith -
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Greg, I'm not sure your improvement patch is right, since some code may
be holding a reference to the former _MainThread instance and expecting
it to still be part of the active threads container.
On the other hand there are things in the
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Selon Gregory P. Smith [EMAIL PROTECTED]:
To answer Antoine Pitrou's question about using the old ident vs the new
_get_ident(). I don't know if the forked process will have the same
thread id.
Then wouldn't it be safer to use
Jesse Noller [EMAIL PROTECTED] added the comment:
To add to ben's comment, under py3k the third test hangs, if you pull
out the basic script code being executed in subprocess:
if 1:
import sys, os, time, threading
# a thread, which waits for the main program to terminate
Jesse Noller [EMAIL PROTECTED] added the comment:
Ben commented out the hanging test, lowering this from a release blocker
as the patch is on both trunk and 3k, and minus that third new test,
test_threading and test_multiprocessing are both passing
--
priority: release blocker -
Gregory P. Smith [EMAIL PROTECTED] added the comment:
Jesse: thanks for doing the py3k merge.
Antoine: yeah it might be safer to use _get_ident() but since the
len(_active) == 1 assert is not firing we're probably fine as is.
A change to this that I was considering making to this code has been
Jesse Noller [EMAIL PROTECTED] added the comment:
Greg/Antoine - do you have any problem with me applying the latest patch
as-is today?
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
Jesse Noller [EMAIL PROTECTED] added the comment:
Alas, I don't have a windows machine - I agree we should leave it open
though
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
Antoine Pitrou [EMAIL PROTECTED] added the comment:
It would be nice to test under Windows first, if you can.
Also, this bug entry should stay open until we discuss the remaining
details.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
Jesse Noller [EMAIL PROTECTED] added the comment:
I've applied Greg's patch in 65032 on trunk.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
___
Gregory P. Smith [EMAIL PROTECTED] added the comment:
I added a Misc/NEWS note for this in r65057.
This is a good candidate for backporting to release25-maint.
To answer Antoine Pitrou's question about using the old ident vs the new
_get_ident(). I don't know if the forked process will have
Benjamin Peterson [EMAIL PROTECTED] added the comment:
Can somebody also merge this into Py3k, please?
--
nosy: +benjamin.peterson
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
Changes by Jesse Noller [EMAIL PROTECTED]:
--
priority: high - release blocker
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
___
Jesse Noller [EMAIL PROTECTED] added the comment:
I've been testing greg's latest patch and it seems to work for me - I'm
not seeing any test_multiprocessing hangs.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
Antoine Pitrou [EMAIL PROTECTED] added the comment:
I still don't like the _after_fork() implementation. Its O(n) where n
== number of threads the parent process had.
It may be O(n) but the inner loop looks very cheap. Even with n == 1000
I'm not sure it would make a difference.
However,
Gregory P. Smith [EMAIL PROTECTED] added the comment:
I still don't like the _after_fork() implementation. Its O(n) where n
== number of threads the parent process had.
Very wasteful when the fork() was done in the most common case of being
followed by an exec and calling os._exit(). It won't
Changes by Gregory P. Smith [EMAIL PROTECTED]:
--
assignee: - gregory.p.smith
nosy: +gregory.p.smith
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
Gregory P. Smith [EMAIL PROTECTED] added the comment:
The existing fork-and-thread4.patch doesn't actually reset
_active_limbo_lock. Its missing a global _active_limbo_lock statement
in the threading._after_fork() function.
___
Python tracker [EMAIL
Gregory P. Smith [EMAIL PROTECTED] added the comment:
and a few more bugs. a new patch is attached. With this applied,
pitrou's fork_threading.py bug demonstration script finally does the
right thing.
test_threading, including the new deadlock tests, and
test_multiprocessing both pass.
Changes by Gregory P. Smith [EMAIL PROTECTED]:
Removed file: http://bugs.python.org/file10855/fork-and-thread.patch
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
Changes by Gregory P. Smith [EMAIL PROTECTED]:
Removed file: http://bugs.python.org/file10859/fork-and-thread2.patch
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
Changes by Gregory P. Smith [EMAIL PROTECTED]:
Removed file: http://bugs.python.org/file10869/fork-and-thread3.patch
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
Jesse Noller [EMAIL PROTECTED] added the comment:
I can confirm that this seems to clear up the test_multiprocessing hangs
on my mac, I ran make test in a loop almost all day yesterday without
hangs. I would really suggest we get this in, minus the new test_threading
tests for now.
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:
I agree. My attempt to describe the behaviour of fork+threads in a
platform-independent test is another issue.
Just one more thing: looking at Module/posixmodule.c, os.fork1() calls
PyOS_AfterFork() in both parent and child processes. Is
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:
I am leaving for the whole next week, so anyone, feel free to commit
(part of) my patch if it helps.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
Jesse Noller [EMAIL PROTECTED] added the comment:
FWIW: the threading tests still hang for me with the latest patch. I'm
confirming that the patch itself fixes the hanging MP tests though
___
Python tracker [EMAIL PROTECTED]
Jesse Noller [EMAIL PROTECTED] added the comment:
Amaury - your latest patch doesn't seem to apply cleanly to trunk, it's
choking on the Python/ceval.c changes
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:
Here is a third patch with latest trunk.
Did you apply it to a clean checkout?
Added file: http://bugs.python.org/file10869/fork-and-thread3.patch
___
Python tracker [EMAIL PROTECTED]
Jesse Noller [EMAIL PROTECTED] added the comment:
I had to flip on ignore whitespace to patch:
patch -p0 -l ~/Desktop/fork-and-thread3.patch
Worked. Unfortunately, test_threading is locking up during a make test.
Here's the verbose regrtest.py output:
woot:python-trunk jesse$ ./python.exe
Jesse Noller [EMAIL PROTECTED] added the comment:
Amaury, it looks like it's hanging in subprocess:
/Users/jesse/open_source/subversion/python-
trunk/Lib/test/test_threading.py(338)_run_and_join()
- p = subprocess.Popen([sys.executable, -c, script],
stdout=subprocess.PIPE)
(Pdb) step
Jesse Noller [EMAIL PROTECTED] added the comment:
Here's the good news, with the patch applied to trunk, I'm not seeing
hangs in the multiprocessing test suite. I'm running it on a tight loop
excluding the threading tests to confirm.
___
Python tracker
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:
Well I was a bit presumptuous that my thread+fork tests would pass on
all platforms out of the box.
We should disable the tests, or have someone with better Unix expertise
examine and correct them.
At least I feel that the actual
Adam Olsen [EMAIL PROTECTED] added the comment:
In general I suggest replacing the lock with a new lock, rather than
trying to release the existing one. Releasing *might* work in this
case, only because it's really a semaphore underneath, but it's still
easier to think about by just replacing.
Adam Olsen [EMAIL PROTECTED] added the comment:
Looking over some of the other platforms for thread_*.h, I'm sure
replacing the lock is the right thing.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:
A new patch:
- I replaced _active_limbo_lock.release() by
_active_limbo_lock=_allocate_lock()
- I replaced the successive deletions in _active by a re-creation with
only the current thread. There is no difference in the result, but I
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:
Attached patch releases the _active_limbo_lock after a fork(). It is not
a complete solution, since existing Thread objects don't correspond to
anything, but it corrects a problem in test_multiprocessing.
--
keywords: +patch
Changes by Jesse Noller [EMAIL PROTECTED]:
--
nosy: +jnoller
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
___
Python-bugs-list mailing list
Amaury Forgeot d'Arc [EMAIL PROTECTED] added the comment:
A slightly better patch, with tests.
Added file: http://bugs.python.org/file10859/fork-and-thread2.patch
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
Changes by Adam Olsen [EMAIL PROTECTED]:
--
nosy: +Rhamphoryncus
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
___
Python-bugs-list mailing
Antoine Pitrou added the comment:
It's not only the _active_limbo_lock. All global structures of the
threading module should be reinitialized (including the _MainThread
instance); for that purpose, reload() can be used. I attach an example
which exercises this problem. Normally the script should
Antoine Pitrou added the comment:
(or perhaps we should provide an API to hook into PyOS_AfterFork)
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
___
Changes by Christian Heimes:
--
priority: normal - high
type: - behavior
versions: +Python 2.6 -Python 2.3
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue874900
61 matches
Mail list logo
