[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Roundup Robot added the comment: New changeset 4e6aa98bb11c by Christian Heimes in branch '3.3': Issue #19227 / Issue #18747: Remove pthread_atfork() handler to remove OpenSSL re-seeding http://hg.python.org/cpython/rev/4e6aa98bb11c -- ___ Python tra

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Changes by Benjamin Peterson : -- resolution: -> fixed status: open -> closed ___ Python tracker ___ ___ Python-bugs-list mailing lis

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Roundup Robot added the comment: New changeset ad779da9e351 by Christian Heimes in branch '2.7': Issue #19227 / Issue #18747: Remove pthread_atfork() handler to remove OpenSSL re-seeding http://hg.python.org/cpython/rev/ad779da9e351 -- ___ Python tra

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Roundup Robot added the comment: New changeset 5942eea8cf41 by Christian Heimes in branch '3.3': Issue #19227 / Issue #18747: Remove pthread_atfork() handler to remove OpenSSL re-seeding http://hg.python.org/cpython/rev/5942eea8cf41 New changeset 705f2addd0f0 by Christian Heimes in branch 'defa

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Changes by Nadeem Vawda : -- nosy: +nadeem.vawda ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.p

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Charles-François Natali added the comment: Note that no matter where the atfork-hook is executed (prepare, parent, child), it will still be deadlock-prone in case of double-fork, since the hook will effectively be called after the first fork (and before the second one). And double-fork is common,

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Roundup Robot added the comment: New changeset 021ca321b26e by Georg Brandl in branch '3.3': Issue #19227: Try to fix deadlocks caused by re-seeding then OpenSSL http://hg.python.org/cpython/rev/021ca321b26e New changeset d3a13a7be9e9 by Georg Brandl in branch 'default': #19227: merge with 3.3 h

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Georg Brandl added the comment: OK, I'm going to apply atfork_prepare.patch for 3.3.3rc1. If there is any indication of a continuing problem, we'll revert the whole re-seeding code for the final. -- ___ Python tracker

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Changes by Arfrever Frehtes Taifersar Arahesis : -- nosy: +Arfrever ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Christian Heimes added the comment: OpenSSL kinda takes care of that. The PRNG mixes the PID into the seed so parent and child won't see the same output. The patch fixes a slightly more tricky issue: - parent initializes PRNG but never touches the PRNG again - parent forks lots of child proces

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Richard Oudkerk added the comment: Won't using a prepare handler mean that the parent and child processes will use the same seed until one or other of them forks again? -- ___ Python tracker __

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Christian Heimes added the comment: I wasn't talking about unlock_ssl_locks.patch but yet another patch. Anyway neither Python 2.7.6 nor 3.3.3 should be released with the current pthread_atfork() handler. We still don't understand the cause of the hanging buildbot but evidence points towards t

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Benjamin Peterson added the comment: Your patch adds a child process atfork handler, which definitely isn't executed in the parent -- ___ Python tracker ___

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Christian Heimes added the comment: I still don't get it. All arguments are about "don't execute async unsafe code after fork()". But the child process does *not* execute *any* code after fork. It's just the parent process that executes its handler. --

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Charles-François Natali added the comment: > Christian Heimes added the comment: > > I think it's more likely that my patch is triggering an existing bug. The > locking code for the SSL module and OpenSSL doesn't release locks on fork. I > have attached an experimental patch that unlocks all lo

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Christian Heimes added the comment: I think it's more likely that my patch is triggering an existing bug. The locking code for the SSL module and OpenSSL doesn't release locks on fork. I have attached an experimental patch that unlocks all locks in the client. Please try if it resolves the iss

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Charles-François Natali added the comment: > It looks as though fork() is indirectly calling something in > _ssl.cpython-34dm.so which is not > completing. The stack is strange: what is /lib/libncursesw.so.5 doing there? Also, we can see exit() is being called. And indeed, _fini() is some share

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Christian Heimes added the comment: Does 2.7 also hang? The pthread_atfork() handler is also installed in 2.7. Can you install the debugging symbols for libc and openssl on your machine? -- nosy: +benjamin.peterson, georg.brandl, larry priority: critical -> release blocker versions: +Py

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Richard Oudkerk added the comment: Actually, according to strace the call which blocks is futex(0xb7839454, FUTEX_WAIT_PRIVATE, 1, NULL -- ___ Python tracker ___ ___

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Richard Oudkerk added the comment: I finally have a gdb backtrace of a stuck child (started using os.fork() not multiprocessing): #1 0xb76194da in ?? () from /lib/libc.so.6 #2 0xb6d59755 in ?? () from /var/lib/buildslave/custom.murray-gentoo/build/build/lib.linux-i686-3.4-pydebug/_ssl.cpy

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

R. David Murray added the comment: Hmm. Looks like the name of the temp files used by the failing test (which is in bsddb, by the way) have changed, so my delete command didn't delete them. So now /tmp is definitely clean, and the next build on that bot should work (crosses fingers). --

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Richard Oudkerk added the comment: > I fixed the out of space last night. (Someday I'll get around to figuring > out which test it is that is leaving a bunch of data around when it fails, > but I haven't yet). It looks like on the Debug Gentoo buildbot configure and clean are failing. http

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

R. David Murray added the comment: I fixed the out of space last night. (Someday I'll get around to figuring out which test it is that is leaving a bunch of data around when it fails, but I haven't yet). I've installed strace and gdb on the bots, please send me your public key and I'll set up

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Richard Oudkerk added the comment: I can reproduce the problem on the Non-Debug Gentoo buildbot using only os.fork() and os.kill(pid, signal.SIGTERM). See http://hg.python.org/cpython/file/9853d3a20849/Lib/test/_test_multiprocessing.py#l339 To investigate further I think strace and/or gd

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Roundup Robot added the comment: New changeset 87f484679a39 by Richard Oudkerk in branch '3.3': Make test_terminate() succeed or fail quickly. http://hg.python.org/cpython/rev/87f484679a39 -- nosy: +python-dev ___ Python tracker

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Richard Oudkerk added the comment: > I'm already confused by the fact that the test is named > test_multiprocessing_spawn and the error is coming from a module named > popen_fork...) popen_spawn_posix.Popen is a subclass of popen_fork.Popen. -- ___

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Charles-François Natali added the comment: > Here is a more useful traceback: If the failures aren't linked to ENFILE, then you could use strace to find the process on which the test is doing a waitpid(), and then perform an strace and gdb on that process to see where it's stuck. And send it a f

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Charles-François Natali added the comment: > R. David Murray added the comment: OSError: [Errno 23] Too many open files in system: '/home/rdmurray/p34/Lib/tarfile.py' Well, the buildbot is hitting ENFILE. Try to run "lsof" as root, to see which process (more likely processes) have so many open

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

R. David Murray added the comment: That's great, but without a patch it doesn't help me debug *this* failure :) Any suggestions for getting more information out of that test (I'm not familiar with multiprocessing...I'm already confused by the fact that the test is named test_multiprocessing_sp

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

STINNER Victor added the comment: See also issue #12413 which proposes to dump also the Python traceback of child processes to analyze multiprocessing issues. -- nosy: +neologix ___ Python tracker

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

R. David Murray added the comment: Here is a more useful traceback: [394] test_multiprocessing_spawn Timeout (1:00:00)! Thread 0xb76a16c0: File "/home/rdmurray/p34/Lib/multiprocessing/popen_fork.py", line 30 in poll File "/home/rdmurray/p34/Lib/multiprocessing/popen_fork.py", line 54 in wait

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

R. David Murray added the comment: Running test_socket test_ssl test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_spaw under -F on the buildbot, I got the following failure during the second loop: [ 10] test_multiprocessing_spawn Traceback (most recent call last):

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python

[issue19227] test_multiprocessing_xxx hangs under Gentoo buildbots

New submission from Antoine Pitrou: test_multiprocessing has started to hang frequently on the Gentoo buildbots. It seems it has started happening with the "OpenSSL re-seeding" commits: http://hg.python.org/cpython/rev/8e1194c39beddb83337c0acb9e4c2922a02a36cf David, could you try to investigate