[issue35866] concurrent.futures deadlock

STINNER Victor added the comment: Great! I close the issue. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe

[issue35866] concurrent.futures deadlock

Change by STINNER Victor : -- resolution: -> fixed stage: -> resolved status: open -> closed ___ Python tracker ___ ___ Python-bug

[issue35866] concurrent.futures deadlock

Jakub Wilk added the comment: I can no longer reproduce the bug with Python from git. -- ___ Python tracker ___ ___ Python-bugs-lis

[issue35866] concurrent.futures deadlock

STINNER Victor added the comment: > #6 0x77bc7712 in __pthread_unwind () from /lib64/libpthread.so.0 > #7 0x77bbf7e7 in pthread_exit () from /lib64/libpthread.so.0 > #8 0x0051b2fc in PyThread_exit_thread () at > Python/thread_pthread.h:238 > #9 0x0055ed16 in

[issue35866] concurrent.futures deadlock

bentoi added the comment: FYI, I'm getting a similar deadlock in a child Python process which is stuck on locking a mutex from the dl library. See attached stack. I'm not using concurrent.futures however, the parent Python process is a test driver that uses threading.Thread and subprocess.P

[issue35866] concurrent.futures deadlock

Gregory P. Smith added the comment: > "the single-threaded ProcessPoolExecutor test program" I doubt it is single threaded, the .submit() method appears to spawn a thread internally. -- ___ Python tracker

[issue35866] concurrent.futures deadlock

cagney added the comment: So: #1 we've a bug: the single-threaded ProcessPoolExecutor test program should work 100% reliably - it does not #2 we've a cause: ProcessPoolExecutor is implemented internally using an unfortunate combination of fork and threads, this is causing the deadlock #3 w

[issue35866] concurrent.futures deadlock

Gregory P. Smith added the comment: concurrent.futures.ProcessPoolExecutor uses both multiprocessing and threading. multiprocessing defaults to using os.fork(). -- ___ Python tracker __

[issue35866] concurrent.futures deadlock

cagney added the comment: @gregory.p.smith, I'm puzzled by your references to POSIX and/or os.fork(). The code in question looks like: import concurrent.futures import sys def f(): import ctypes while True: with concurrent.futures.ProcessPoolExecutor() as executor: ftr = exe

[issue35866] concurrent.futures deadlock

cagney added the comment: We're discussing vanilla Python, for instance v3.7.0 is: git clone .../cpython cd cpython git checkout v3.7.0 ./configure --prefix=/home/python/v3.7.0 make -j && make -j install (my 3.6.x wasn't vanilla, but I clearly stated that) Like I also mentioned, lo

[issue35866] concurrent.futures deadlock

Gregory P. Smith added the comment: I am unable to get cf-deadlock.py to hang on my own builds of pure CPython 3.7.2+ d7cb2034bb or 3.6.8+ be77fb7a6e (versions i had in a local git clone). which specific python builds are seeing the hang using? Which specific platform/distro version? "3.7.

[issue35866] concurrent.futures deadlock

cagney added the comment: script to capture stack backtrace at time of fork, last backtrace printed will be for hang -- Added file: https://bugs.python.org/file48272/gdb.sh ___ Python tracker __

[issue35866] concurrent.futures deadlock

cagney added the comment: run ProcessPoolExecutor with one fixed child (over ride default of #cores) -- Added file: https://bugs.python.org/file48271/cf-deadlock-1.py ___ Python tracker _

[issue35866] concurrent.futures deadlock

cagney added the comment: Here's a possible stack taken during the fork(): Thread 1 "python3" hit Breakpoint 1, 0x77124734 in fork () from /lib64/libc.so.6 Thread 1814 (Thread 0x7fffe69d5700 (LWP 23574)): #0 0x77bc24e5 in __pthread_mutex_unlock_usercnt () from /lib64/libpth

[issue35866] concurrent.futures deadlock

Jakub Wilk added the comment: https://github.com/python/cpython/pull/12704 doesn't fix the bug for me. Reverting 3b699932e5ac3e76031bbb6d700fbea07492641d doesn't fix it either. -- ___ Python tracker

[issue35866] concurrent.futures deadlock

Gregory P. Smith added the comment: I'd appreciate it if someone with an application running into the issue could be tested with my PR from issue36533 (https://github.com/python/cpython/pull/12704) applied. -- ___ Python tracker

[issue35866] concurrent.futures deadlock

Gregory P. Smith added the comment: Please do not blindly revert that. See my PR in https://bugs.python.org/issue36533 which is specific to this "issue" with logging. -- versions: +Python 3.8 ___ Python tracker

[issue35866] concurrent.futures deadlock

cagney added the comment: (disclaimer: I'm mashing my high level backtraces in with @jwiki's low level backtraces) The Python backtrace shows the deadlocked process called 'f' which then 'called': import ctypes which, in turn 'called': from _ctypes import Union, Structure, Array and

[issue35866] concurrent.futures deadlock

STINNER Victor added the comment: A least 2 projects were broken by the logging change: libreswan and Anaconda. > I've filed a Fedora bug report that points to this one: > That's related to the libreswan project. Last year, there was an

[issue35866] concurrent.futures deadlock

STINNER Victor added the comment: Gregory: It seems like https://github.com/python/cpython/commit/3b699932e5ac3e76031bbb6d700fbea07492641d is causing deadlocks which is not a good thing. What do you think of reverting this change? -- nosy: +gregory.p.smith _

[issue35866] concurrent.futures deadlock

cagney added the comment: @hroncok see comment msg339370 Vanilla 3.7.0 (re-confirmed) didn't contain the change, nor did 3.6.8 (ok, that isn't vanilla) but both can hang using the test. It can take a while and, subjectively, it seems to depend on machine load. I've even struggled to get 3

[issue35866] concurrent.futures deadlock

Miro Hrončok added the comment: Reverting 3b699932e5ac3e76031bbb6d700fbea07492641d makes problem go away. -- nosy: +hroncok ___ Python tracker ___ _

[issue35866] concurrent.futures deadlock

cagney added the comment: Here's the children; yes there are somehow 4 children sitting around. Hopefully this is enough to figure out where things deadlock. 29970 8752 8752 29970 pts/6 8752 Sl+ 1000 1:00 | | \_ ./v3.7.3/bin/python3 cf-deadlock.py 8752 8975 8752 29970 pts/

[issue35866] concurrent.futures deadlock

cagney added the comment: More info from adding a faulthandler ... 15 def f(): 16 import ctypes 17 18 for i in range(0,50): 19 sys.stdout.write("\r%d" % i) 20 sys.stdout.flush() 21 signal.alarm(60*2) 22 for j in range(0,1000): 23

[issue35866] concurrent.futures deadlock

cagney added the comment: I'm seeing cf-deadlock-alarm.py hangs on vanilla python 3.7.[0123] with: Linux 5.0.5-100.fc28.x86_64 #1 SMP Wed Mar 27 22:16:29 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux glibc-2.27-37.fc28.x86_64 can anyone reproduce this? I also wonder if this is connected to bpo-67

[issue35866] concurrent.futures deadlock

cagney added the comment: I've attached a variation on cf-deadlock.py that, should nothing happen for 2 minutes, will kill itself. Useful with git bisect. -- Added file: https://bugs.python.org/file48242/cf-deadlock-alarm.py ___ Python tracker

[issue35866] concurrent.futures deadlock

STINNER Victor added the comment: Any update on this issue? I don't understand why the example hangs. -- nosy: +vstinner ___ Python tracker ___ ___

[issue35866] concurrent.futures deadlock

Hugh Redelmeier added the comment: I've filed a Fedora bug report that points to this one: -- ___ Python tracker ___ ___

[issue35866] concurrent.futures deadlock

Hugh Redelmeier added the comment: @jwilk: thanks for creating cf-deadlock.py I can replicate the test program hang on Fedora 29 with python3-3.7.2-4.fc29.x86_64 The test program hasn't yet hung on Fedora 29 with older packages, in particular python3-3.7.1-4.fc29.x86_64 My interest is due

[issue35866] concurrent.futures deadlock

Change by Jakub Wilk : Added file: https://bugs.python.org/file48224/gdb-bt-child.txt ___ Python tracker ___ ___ Python-bugs-list mailing li

[issue35866] concurrent.futures deadlock

Jakub Wilk added the comment: There are two processes running (parent and child) when the thing hangs. I'm attaching GDB backtraces for both. -- Added file: https://bugs.python.org/file48223/gdb-bt-parent.txt ___ Python tracker

[issue35866] concurrent.futures deadlock

Pablo Galindo Salgado added the comment: Could you use gdb/lldb to attach to the process hanging and give us a stack trace? -- ___ Python tracker ___

[issue35866] concurrent.futures deadlock

Pablo Galindo Salgado added the comment: This seem related to https://bugs.python.org/issue35809 -- nosy: +pablogsal ___ Python tracker ___ ___

[issue35866] concurrent.futures deadlock

Change by cagney : -- nosy: +cagney ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/m

[issue35866] concurrent.futures deadlock

Jakub Wilk added the comment: You're right that sys.stdout.flush() is missing in my code; but on Linux it doesn't make a big difference, because multiprocessing flushes stdout before fork()ing. And yes, it really hangs. -- ___ Python tracker

[issue35866] concurrent.futures deadlock

Josh Rosenberg added the comment: I've only got 3.7.1 Ubuntu bash on Windows (also amd64) immediately available, but I'm not seeing a hang, nor is there any obvious memory leak that might eventually lead to problems (memory regularly drops back to under 10 MB shared, 24 KB private working se

[issue35866] concurrent.futures deadlock

Change by SilentGhost : -- nosy: +bquinlan, pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://ma

[issue35866] concurrent.futures deadlock

New submission from Jakub Wilk : The attached test program hangs eventually (it may need a few thousand of iterations). Tested with Python v3.7.2 on Linux, amd64. -- components: Library (Lib) files: cf-deadlock.py messages: 334618 nosy: jwilk priority: normal severity: normal status: