[issue44422] threading.enumerate(): reentrant call during a GC collection hangs

[STINNER Victor]

Change by STINNER Victor :


--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44422] threading.enumerate(): reentrant call during a GC collection hangs

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 0729694246174a5c2f0ae197f2e0dbea61b90c9f by Victor Stinner in 
branch 'main':
bpo-44422: threading.Thread reuses the _delete() method (GH-26741)
https://github.com/python/cpython/commit/0729694246174a5c2f0ae197f2e0dbea61b90c9f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44422] threading.enumerate(): reentrant call during a GC collection hangs

[STINNER Victor]

STINNER Victor  added the comment:

Oh, I reopen the issue for PR 26741.

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44422] threading.enumerate(): reentrant call during a GC collection hangs

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +25327
pull_request: https://github.com/python/cpython/pull/26741

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44422] threading.enumerate(): reentrant call during a GC collection hangs

[STINNER Victor]

STINNER Victor  added the comment:

Ok, the deadlock on reentrant call to threading.enumerate() is now fixed in 
3.9, 3.10 and main (future 3.11) branches. I close the issue. Thanks for the 
reviews, as usual :-)

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44422] threading.enumerate(): reentrant call during a GC collection hangs

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 8fe57aacc7bf9d9af84803b69dbb1d66597934c6 by Miss Islington (bot) 
in branch '3.9':
bpo-44422: Fix threading.enumerate() reentrant call (GH-26727) (GH-26738)
https://github.com/python/cpython/commit/8fe57aacc7bf9d9af84803b69dbb1d66597934c6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44422] threading.enumerate(): reentrant call during a GC collection hangs

[miss-islington]

miss-islington  added the comment:


New changeset c3b776f83b4c765da6d7b8854e844e07bd33c375 by Miss Islington (bot) 
in branch '3.10':
bpo-44422: Fix threading.enumerate() reentrant call (GH-26727)
https://github.com/python/cpython/commit/c3b776f83b4c765da6d7b8854e844e07bd33c375


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44422] threading.enumerate(): reentrant call during a GC collection hangs

[miss-islington]

Change by miss-islington :


--
pull_requests: +25324
pull_request: https://github.com/python/cpython/pull/26738

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44422] threading.enumerate(): reentrant call during a GC collection hangs

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 1.0 -> 2.0
pull_requests: +25323
pull_request: https://github.com/python/cpython/pull/26737

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44422] threading.enumerate(): reentrant call during a GC collection hangs

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 243fd01047ddce1a7eb0f99a49732d123e942c63 by Victor Stinner in 
branch 'main':
bpo-44422: Fix threading.enumerate() reentrant call (GH-26727)
https://github.com/python/cpython/commit/243fd01047ddce1a7eb0f99a49732d123e942c63


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44422] threading.enumerate(): reentrant call during a GC collection hangs

[STINNER Victor]

STINNER Victor  added the comment:

There are different ways to fix this issue:

* (A) Rewrite threading.enumerate() in C with code which cannot trigger a GC 
collection
* (B) Disable temporarily the GC
* (C) Use a reentrant lock (PR 26727)

(A) The problem is that functions other than threading.enumerate() 
also rely on this lock, like threading.active_count(). I would prefer to not 
have to rewrite "half" of threading.py in C. Using a RLock is less intrusive.

(B) This is a simple and reliable option. But gc.disable() is process-wide: it 
affects all Python threads, and so it might have surprising side effects. Some 
code might rely on the current exact GC behavior. I would prefer to not disable 
the GC temporarily.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44422] threading.enumerate(): reentrant call during a GC collection hangs

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +25317
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/26727

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44422] threading.enumerate(): reentrant call during a GC collection hangs

[STINNER Victor]

New submission from STINNER Victor :

The threading.enumerate() code is simple:
---
# Active thread administration
_active_limbo_lock = _allocate_lock()
_active = {}# maps thread id to Thread object
_limbo = {}

def enumerate():
with _active_limbo_lock:
return list(_active.values()) + list(_limbo.values())
---

values(), list() and list+list operations can call trigger a GC collection 
depending on the GC thresholds, created Python objects, etc.

During a GC collection, a Python object destructor (__del__) can again call 
threading.enumerate().

Problem: _active_limbo_lock is not re-entrant lock and so the second call hangs.

In practice, this issue was seen in OpenStack which uses eventlet, when a 
destructor uses the logging module. The logging module uses 
threading.current_thread(), but this function is monkey-patched by the eventlet 
module.

Extract of the eventlet function:
---
def current_thread():
...
found = [th for th in __patched_enumerate() if th.ident == g_id]
...
---
https://github.com/eventlet/eventlet/blob/master/eventlet/green/threading.py

Attached PR makes _active_limbo_lock re-entrant to avoid this issue. I'm not 
sure if it's ok or not, I would appreciate to get a review and your feedback ;-)

Attached file triggers the threading.enumerate() hang on re-entrant call.

--
components: Library (Lib)
files: rec_threading.py
messages: 395851
nosy: vstinner
priority: normal
severity: normal
status: open
title: threading.enumerate(): reentrant call during a GC collection hangs
versions: Python 3.10, Python 3.11, Python 3.9
Added file: https://bugs.python.org/file50110/rec_threading.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com