[issue26360] Deadlock in thread.join on Python 2.7/macOS

[Zachary Ware]

Zachary Ware  added the comment:

As 2.7 is now EOL, I'm closing the issue.

--
nosy: +zach.ware
resolution:  -> out of date
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join on Python 2.7/macOS

[Kirill Smelkov]

Kirill Smelkov  added the comment:

> > Maybe issue38106 related.
>
> That looks plausible, but unfortunately I'm still able to reproduce the hang 
> with your PR (commit 9b135c02aa1edab4c99c915c43cd62d988f1f9c1, macOS 10.14.6).

Thanks for feedback. Then hereby bug is probably deadlock of another kind.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join on Python 2.7/macOS

[Kyle Stanley]

Kyle Stanley  added the comment:

> FWIW, I've confirmed again that the exact same script on the same machine 
> seems fine under Python 3.x. Given the imminent demise of Python 2, perhaps 
> this issue is just destined to be an unsolved historical oddity.

Since it doesn't seem to be occurring across every macOS platform, is specific 
to 2.7, and isn't proving to be an easy fix, it's probably not worthwhile to 
invest a significant amount of further time into the issue. Thanks for working 
on it though, at least we tried. (:

--
nosy: +aeros167

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join on Python 2.7/macOS

[Mark Dickinson]

Mark Dickinson  added the comment:

FWIW, I've confirmed again that the exact same script on the same machine seems 
fine under Python 3.x. Given the imminent demise of Python 2, perhaps this 
issue is just destined to be an unsolved historical oddity.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join on Python 2.7/macOS

[Mark Dickinson]

Change by Mark Dickinson :


--
title: Deadlock in thread.join on Python 2.7/Mac OS X 10.9, 10.10 -> Deadlock 
in thread.join on Python 2.7/macOS

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join on Python 2.7/Mac OS X 10.9, 10.10

[Mark Dickinson]

Mark Dickinson  added the comment:

> Maybe issue38106 related.

That looks plausible, but unfortunately I'm still able to reproduce the hang 
with your PR (commit 9b135c02aa1edab4c99c915c43cd62d988f1f9c1, macOS 10.14.6).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join on Python 2.7/Mac OS X 10.9, 10.10

[Kirill Smelkov]

Kirill Smelkov  added the comment:

Maybe issue38106 related.

--
nosy: +navytux

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join on Python 2.7/Mac OS X 10.9, 10.10

[Mark Dickinson]

Changes by Mark Dickinson :


--
title: Deadlock in thread.join on Python 2.7/Mac OS X 10.9 -> Deadlock in 
thread.join on Python 2.7/Mac OS X 10.9, 10.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join on Python 2.7/Mac OS X 10.9

[Mark Dickinson]

Mark Dickinson added the comment:

I've found OS X 10.10 and OS X 10.11 machines to test on. I'm able to reproduce 
on 10.10, but not on 10.11. This adds to the plausibility of the "Apple bug" 
explanation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join on Python 2.7/Mac OS X 10.9

[Mark Dickinson]

Mark Dickinson added the comment:

A minor update: I did manage to verify that the GIL is indeed not held by any 
threads at the moment of the hang, so it does look very much as though a 
`pthread_cond_signal` signal has gone astray somehow.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join on Python 2.7/Mac OS X 10.9

[Mark Dickinson]

Mark Dickinson added the comment:

Thanks, Ned. I tried with:

   ./configure --with-pydebug MACOSX_DEPLOYMENT_TARGET=10.9 && make

I'm still able to reproduce. I plan to find some time this weekend to dig into 
the source and see if I can narrow down the cause (or at least make the hang 
easier / quicker to reproduce).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join on Python 2.7/Mac OS X 10.9

[Ned Deily]

Ned Deily added the comment:

I'm not able to test it myself at the moment but you could also try ensuring 
your 2.7 build is with the most up-to-date ABI for your system:

./configure MACOSX_DEPLOYMENT_TARGET=10.9

2.7 builds do not set this automatically; 3.x builds do.

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join on Python 2.7/Mac OS X 10.9

[STINNER Victor]

STINNER Victor added the comment:

> Thanks for testing! It wouldn't entirely surprise me if this turned out to be 
> an Apple bug somewhere.

I'm ran threading_hang.py on Mac OS X 10.10 Yosemite (darwin 14.5) on Python 
2.7 compiled in release mode.

After 28 iterations (~4 hours, 9 minutes per iteration), the bug was still not 
reproduced.

It looks like my OS X version is more recent than yours. You should run your 
test on various OS X versions to try to identify the bug.

Since it's likely a race condition, I would not be surprised that the bug is 
very hard to reproduce and may exist on other platforms, but may be less likely 
on other platforms or even less likely on some computers for various reasons...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join on Python 2.7/Mac OS X 10.9

[Mark Dickinson]

Mark Dickinson added the comment:

Thanks for testing! It wouldn't entirely surprise me if this turned out to be 
an Apple bug somewhere.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join on Python 2.7/Mac OS X 10.9

[STINNER Victor]

Changes by STINNER Victor :


--
title: Deadlock in thread.join -> Deadlock in thread.join on Python 2.7/Mac OS 
X 10.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[STINNER Victor]

STINNER Victor added the comment:

"Each iteration takes around 60 seconds, and I typically get a deadlock within 
the first 5 iterations."

I ran a modified version of threading_hang.py (I added some debug print) a 
whole night: it's still running after 118 iterations.

Each iteration takes ~5 minutes, I'm using a debug build of Python 2.7 on Linux 
(Fedora 23).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[Mark Dickinson]

Mark Dickinson added the comment:

> I'm not sure that "Waiting for the GIL" line is reliable

Right; I was interpreting that line more as "I don't have the GIL at the 
moment", which I think applies to all threads, including the main thread. It's 
pretty clear that threads 2-5 really are waiting for the GIL, and (as you point 
out), for thread 1 to be where it is, it must have also released the GIL by 
that point.

Agreed about the gdb results for the -O3 build.

So it looks as though something's gone wrong with the "I've just released the 
GIL; now someone else can pick it up" signaling mechanism.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[STINNER Victor]

STINNER Victor added the comment:

>> lock=0x0 looks suspicious.

> Agreed. I think that *could* possibly be a consequence of the earlier 
> multiple GILs, though I haven't figured out exactly how yet.

Don't trust gdb output when Python is compiled in release mode (with -O3).

gdb output is only reliable when Python is compiled with -O0 (./configure 
--with-pydebug is enough for Python 2.7).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[STINNER Victor]

STINNER Victor added the comment:

> I've also not yet managed to reproduce on Python 3, but I haven't tried that 
> hard. I suspect it's a Python 2-only problem, though.

Python 3.2 got a new GIL and many bugs related to the GIL were fixed in Python 
3 too.

The "workaround" is to use Python 3, until this Python 2 issue is fixed (if we 
are able to fix it!).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[STINNER Victor]

STINNER Victor added the comment:

debug-build-stacktrace.txt:
* Thread 1 is waiting on threading_hang.py:21: thread.join()
* Thread 2-6 (5 threads) are waiting for the GIL

The thread 1 is not supposed to hold the GIL: thread.join() is implemented by 
lock_PyThread_acquire_lock() at Modules/threadmodule.c:52:

Py_BEGIN_ALLOW_THREADS
i = PyThread_acquire_lock(self->lock_lock, i);
Py_END_ALLOW_THREADS

Note: I backported enhancements of python-gdb.py from Python 3, but I'm not 
sure that "Waiting for the GIL" line is reliable :-/ I recall vaguely a 
technical issue specific to Python 2 to check if a thread is waiting on the GIL 
or not.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[Mark Dickinson]

Mark Dickinson added the comment:

Update: I *have* been able to reproduce in a debug build. Here's the stack 
trace.

--
Added file: http://bugs.python.org/file41924/debug-build-stacktrace.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[Mark Dickinson]

Mark Dickinson added the comment:

One thing that looks a bit odd to my eyes is the different values for `lock` in 
the PyThread_acquire_lock calls (see below). All of threads 2-5 should be 
waiting on the GIL, so I'd expect to see the same value for `lock` in all those 
calls. Instead, threads 2 and 5 seem to be waiting on the same lock, but 
threads 3 and 4 seem to have their own lock. But I don't know this code well 
enough to know whether it's supposed to be this way.

Thread 5 (Thread 0x1903 of process 14075):
#2  0x0001000e7e21 in PyThread_acquire_lock (lock=0xed3ed500, 
waitflag=) at Python/thread_pthread.h:429

Thread 4 (Thread 0x1803 of process 14075):
#2  0x0001000e7e21 in PyThread_acquire_lock (lock=0xed3ed300, 
waitflag=) at Python/thread_pthread.h:429

Thread 3 (Thread 0x1703 of process 14075):
#2  0x0001000e7e21 in PyThread_acquire_lock (lock=0xed3ed600, 
waitflag=) at Python/thread_pthread.h:429

Thread 2 (Thread 0x1603 of process 14075):
#2  0x0001000e7e21 in PyThread_acquire_lock (lock=0xed3ed300, 
waitflag=) at Python/thread_pthread.h:429

Thread 1 (Thread 0x1503 of process 14075):
#2  0x0001000e7e21 in PyThread_acquire_lock (lock=0x0, waitflag=) at Python/thread_pthread.h:429

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[Mark Dickinson]

Mark Dickinson added the comment:

> threads 2 and 5

Sorry, that should have said "threads 2 and 4".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[Mark Dickinson]

Mark Dickinson added the comment:

> lock=0x0 looks suspicious.

Agreed. I think that *could* possibly be a consequence of the earlier multiple 
GILs, though I haven't figured out exactly how yet.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

lock=0x0 looks suspicious.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[Mark Dickinson]

Mark Dickinson added the comment:

Here's the hanging code as an attached file.

--
Added file: http://bugs.python.org/file41921/threading_hang.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[Mark Dickinson]

New submission from Mark Dickinson:

On OS X (10.9.5), I'm getting an apparent deadlock in the following simple 
Python script:

#
import itertools
import threading

def is_prime(n):
return n >= 2 and all(n % d for d in xrange(2, n))

def count_primes_in_range(start, stop):
return sum(is_prime(n) for n in xrange(start, stop))

def main():
threads = [
threading.Thread(
target=count_primes_in_range,
args=(12500*i, 12500*(i+1))
)
for i in xrange(8)
]
for thread in threads:
thread.start()
for thread in threads:
thread.join()


if __name__ == '__main__':
for i in itertools.count():
print "Iteration: ", i
main()
#

Each iteration takes around 60 seconds, and I typically get a deadlock within 
the first 5 iterations. It looks as though the deadlock happens during the 
"thread.join", at a stage where some of the threads have already completed and 
been joined. The code hangs with no CPU activity, and a backtrace (attached) 
shows that all the background threads are waiting for the GIL, while the main 
thread is in a blocking `thread.join` call.

I've attached a gdb-generated stack trace.

I was unable to reproduce this with a debug build of Python. I *have* 
reproduced with a normal build of Python, and on various Python 2.7 executables 
from 3rd party sources (Apple, Macports, Enthought Canopy).

I've also not yet managed to reproduce on Python 3, but I haven't tried that 
hard. I suspect it's a Python 2-only problem, though.

(And yes, this is horrible code that doesn't make much sense. It's actually a 
cut-down example from a "how not to do it" part of a course on concurrency. 
Nevertheless, it shouldn't be deadlocking.)

--
components: Interpreter Core
files: hang_backtrace.txt
messages: 260273
nosy: mark.dickinson
priority: normal
severity: normal
status: open
title: Deadlock in thread.join
type: behavior
versions: Python 2.7
Added file: http://bugs.python.org/file41920/hang_backtrace.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +pitrou, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26360] Deadlock in thread.join

[Mark Dickinson]

Mark Dickinson added the comment:

Build details: this was a version of Python built with a simple `./configure && 
make` from commit 32893d8a52a9.

taniyama:cpython mdickinson$ ./python.exe
Python 2.7.10+ (2.7:32893d8a52a9, Feb 14 2016, 11:08:14) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com