[issue45274] Race condition in Thread._wait_for_tstate_lock()

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[Géry]

Change by Géry :


--
nosy: +maggyero

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[Ben]

Change by Ben :


--
nosy: +bjs
nosy_count: 6.0 -> 7.0
pull_requests: +29450
pull_request: https://github.com/python/cpython/pull/31290

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[STINNER Victor]

STINNER Victor  added the comment:

> This is the same as bpo-21822, so I suppose it should be closed as well with 
> a reference to this issue.

You're right. I marked bpo-21822 as a duplicate of this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[Eryk Sun]

Eryk Sun  added the comment:

This is the same as bpo-21822, so I suppose it should be closed as well with a 
reference to this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[STINNER Victor]

Change by STINNER Victor :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.10, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[STINNER Victor]

STINNER Victor  added the comment:


New changeset fae2694bea5e9e5a114af8cb40b60e7131a6340c by Miss Islington (bot) 
in branch '3.10':
bpo-45274: Fix Thread._wait_for_tstate_lock() race condition (GH-28532) 
(GH-28580)
https://github.com/python/cpython/commit/fae2694bea5e9e5a114af8cb40b60e7131a6340c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[miss-islington]

miss-islington  added the comment:


New changeset 1ecb641b887af2feb026a266e2fceedee0815ca8 by Miss Islington (bot) 
in branch '3.9':
bpo-45274: Fix Thread._wait_for_tstate_lock() race condition (GH-28532)
https://github.com/python/cpython/commit/1ecb641b887af2feb026a266e2fceedee0815ca8


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[miss-islington]

Change by miss-islington :


--
pull_requests: +26963
pull_request: https://github.com/python/cpython/pull/28580

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 5.0 -> 6.0
pull_requests: +26962
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/28579

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[STINNER Victor]

STINNER Victor  added the comment:


New changeset a22be4943c119fecf5433d999227ff78fc2e5741 by Victor Stinner in 
branch 'main':
bpo-45274: Fix Thread._wait_for_tstate_lock() race condition (GH-28532)
https://github.com/python/cpython/commit/a22be4943c119fecf5433d999227ff78fc2e5741


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[STINNER Victor]

STINNER Victor  added the comment:

> I am not sure that it can be solved at Python level.

Right. In pure Python, we cannot write code which works in all cases. My PR 
28532 fix the most common case: application interrupted by a single CTRL+C.

> with suppress_interrupt(): (...)

It's important to be able to interrupt acquire() which can be called in 
blocking mode with no timeout: it's exactly what tox does, and users expect to 
be able to interrupt tox in this case.

> * Add a Lock method (or just a builtin function) which acquires and 
> immediately releases the lock, without possibility to interrupt.

The acquire()+release() sequence can be made atomic in C, but it doesn't solve 
the problem of _stop() which can be interrupted by a second exception.

This bug is likely as old as Python. I don't think that we should attempt to 
design a perfect solution. I only propose to make the race condition (way) less 
likely.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[Eryk Sun]

Change by Eryk Sun :


--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[gaborjbernat]

gaborjbernat  added the comment:

I just want to note that my maximal reproducer moved to 
https://github.com/tox-dev/tox/commit/e5d1a439be0790c8104d4caf943b3d82f23a0039 
(that has been merged on the branch, so should be now stable).

--
nosy:  -eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[Eryk Sun]

Eryk Sun  added the comment:

See bpo-21822 from 2014, which I think was the first report of this issue.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[gaborjbernat]

Change by gaborjbernat :


--
nosy: +gaborjbernat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> Add a context manager which suppresses keyboard interruption.

That's not enough technically. This can be any signal handler that raises an 
exception, no?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I am not sure that it can be solved at Python level.

Possible solutions:

* Add a Lock method (or just a builtin function) which acquires and immediately 
releases the lock, without possibility to interrupt.

if lock._blink(block, timeout):
self._stop()

* Add a context manager which suppresses keyboard interruption.

with suppress_interrupt():
if not lock._blink(block, timeout):
return
self._stop()

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[STINNER Victor]

Change by STINNER Victor :


--
nosy: +pablogsal
stage: patch review -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +26920
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/28532

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[STINNER Victor]

STINNER Victor  added the comment:

For curious people, see also bpo-44422 "Fix threading.enumerate() reentrant 
call": another example of race condition recently fixed in the threading 
module. But it's unrelated to this bug ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[STINNER Victor]

Change by STINNER Victor :


Added file: https://bugs.python.org/file50300/threading_bug.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45274] Race condition in Thread._wait_for_tstate_lock()

[STINNER Victor]

New submission from STINNER Victor :

Bernát Gábor found an interesting bug on Windows. Sometimes, when a process is 
interrupted on Windows with CTRL+C, its parent process hangs in thread.join():
https://twitter.com/gjbernat/status/1440949682759426050

Reproducer:

* Install https://github.com/gaborbernat/tox/tree/2159
* Make an empty folder and put the above tox.ini in it
* Invoke python -m tox and once that command is sleeping press CTRL+C (the app 
should lock there indefinitely).

tox.ini:
---

[testenv]
skip_install = true
commands = python -c 'import os; print(f"start {os.getpid()}"); import time; 
time.sleep(100); print("end");'
---

Source: https://gist.github.com/gaborbernat/f1e1aff0f2ee514b50a92a4d019d4d13

I tried to attach the Python process in Python: there is a single thread, the 
main thread which is blocked in thread.join(). You can also see it in the 
faulthandler traceback.

I did a long analysis of the _tstate_lock and I checked that thread really 
completed. Raw debug traces:

== thread 6200 exit ==
 
thread_run[pid=3984, thread_id=6200]: clear
PyThreadState_Clear[pid=3984, thread_id=6200]: on_delete()
release_sentinel[pid=3984, thread_id=6200]: enter
release_sentinel[pid=3984, thread_id=6200]: release(obj=01C1122669C0, 
lock=01C110BBDA00)
release_sentinel[pid=3984, thread_id=6200]: exit
PyThreadState_Clear[pid=3984, thread_id=6200]: on_delete()--
 
== main thread is calling join() but gets a KeyboardInterrupt ==
 
[pid=3984, thread_id=8000] Lock.acquire() -> ACQUIRED
Current thread 0x1f40 (most recent call first):
  File "C:\vstinner\python\3.10\lib\threading.py", line 1118 in 
_wait_for_tstate_lock
  File "C:\vstinner\python\3.10\lib\threading.py", line 1092 in join
  File "C:\vstinner\env\lib\site-packages\tox\session\cmthread_run[pid=3984, 
thread_id=6200]: exit
d\run\common.py", line 203 in execute
  File "C:\vstinner\env\lib\site-packages\tox\session\cmd\run\sequential.py", 
line 20 in run_sequential
  File "C:\vstinner\env\lib\site-packages\tox\session\cmd\legacy.py", line 104 
in legacy
  File "C:\vstinner\env\lib\site-packages\tox\run.py", line 49 in main
  File "C:\vstinner\env\lib\site-packages\tox\run.py", line 23 in run
  File "C:\vstinner\env\lib\site-packages\tox\__main__.py", line 4 in 
  File "C:\vstinner\python\3.10\lib\runpy.py", line 86 in _run_code
  File "C:\vstinner\python\3.10\lib\runpy.py", line 196 in _run_module_as_main
_wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: 
EXC: KeyboardInterrupt(); acquired? None
 
== main thread calls repr(thread) ==
 
ROOT: [3984] KeyboardInterrupt - teardown started
_wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: 
acquire(block=False, timeout=-1): lock obj= 0x1c1122669c0
  File "C:\vstinner\python\3.10\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
  File "C:\vstinner\python\3.10\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
  File "C:\vstinner\env\lib\site-packages\tox\__main__.py", line 4, in 
run()
  File "C:\vstinner\env\lib\site-packages\tox\run.py", line 23, in run
result = main(sys.argv[1:] if args is None else args)
  File "C:\vstinner\env\lib\site-packages\tox\run.py", line 49, in main
result = handler(state)
  File "C:\vstinner\env\lib\site-packages\tox\session\cmd\legacy.py", line 104, 
in legacy
return run_sequential(state)
  File "C:\vstinner\env\lib\site-packages\tox\session\cmd\run\sequential.py", 
line 20, in run_sequential
return execute(state, max_workers=1, has_spinner=False, live=True)
  File "C:\vstinner\env\lib\site-packages\tox\session\cmd\run\common.py", line 
217, in execute
print(f'join {thread}')
  File "C:\vstinner\python\3.10\lib\threading.py", line 901, in __repr__
self.is_alive() # easy way to get ._is_stopped set when appropriate
  File "C:\vstinner\python\3.10\lib\threading.py", line 1181, in is_alive
self._wait_for_tstate_lock(False)
  File "C:\vstinner\python\3.10\lib\threading.py", line 1113, in 
_wait_for_tstate_lock
traceback.print_stack()
_wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: 
failed to acquire 0x1c1122669c0
_wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: 
exit 0x1c1122669c0
join 
 
== main thread calls thread.join()... which hangs ==
 
_wait_for_tstate_lock[pid=3984, current thread_id=8000, self thread_id=6200]: 
acquire(block=True, timeout=-1): lock obj= 0x1c1122669c0
  File "C:\vstinner\python\3.10\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
  File "C:\vstinner\python\3.10\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
  File "C:\vstinner\env\lib\site-packages\tox\__main__.py", line 4, in 
run()
  File "C:\vstinner\env\lib\site-packages\tox\run.py", line 23, in run
result = main(sys.argv[1:] if args is None else args)
  File