[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Zachary Ware added the comment: As 2.7 is now EOL, I'm closing the issue. -- nosy: +zach.ware resolution: -> out of date stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

STINNER Victor added the comment: > But! I suppose we could fix the bug only for platforms with > PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT. Yes, this is my proposal. >> I propose to cast pthread_key_create() result to int, but only define >> PyThread_create_key() in

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Petr Viktorin added the comment: > PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT is defined on most (pthread) platforms, > no? > I propose to cast pthread_key_create() result to int, but only define > PyThread_create_key() in Python/thread_pthread.h if >

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

STINNER Victor added the comment: > the old TLS API is only available if PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT. PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT is defined on most (pthread) platforms, no? I understood that the PEP 539 is mostly designed for Cygwin, a platform which

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Petr Viktorin added the comment: pthread is not generally compatible with int, so it can't be used with Python 2's API. This was changed in py3 with PEP 539, which however makes it so that the old TLS API is only available if PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT.

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

STINNER Victor added the comment: Oh, it seems like I was wrong in my previous comment. Python 2.7 code base is already designed to support native TLS. It's just that we only implement native TLS on Windows. So yeah, it seems doable to implement native TLS for pthread.

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Petr Viktorin added the comment: Not immediately, but it is on my TODO list. If anyone wants to tackle it in the mean time, I'd be happy to answer any questions -- ___ Python tracker

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Stéphane Wirtel added the comment: Hi Petr, Do you continue this patch/issue? Thank you -- nosy: +matrixise ___ Python tracker ___

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Petr Viktorin added the comment: WIP pull request: https://github.com/python/cpython/pull/5141 (I'll not get back to this for a few days, sadly.) -- ___ Python tracker

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Change by Petr Viktorin : -- pull_requests: +5000 stage: -> patch review ___ Python tracker ___

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Petr Viktorin added the comment: Gah! The more I look into locks & forks ... the more I learn, to put it mildly. Instead of piling on workarounds, I'll try my hand at using native thread-local storage for pthread, and avoid the locking altogether. Hopefully that can make

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

STINNER Victor added the comment: Python 3 is not affected by this issue because it uses native thread locale storage (TLS): * pthread: pthread_getspecific() / pthread_setspecific() * Windows: TlsGetValue() / TlsSetValue() I'm not sure that it's doable to backport

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

STINNER Victor added the comment: There is a more general issue for any lock and fork(): bpo-6721, "Locks in the standard library should be sanitized on fork". -- ___ Python tracker

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Petr Viktorin added the comment: I'm a bit out of my depth here. Victor, could you chime in? The problem with Harris' patch is that, once fork() is protected by the thread lock, acquiring that lock (by e.g. calling `PyGILState_GetThisThreadState`) from an `atfork` handler

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Thomas Mortensson added the comment: Hey, any status update on this bug? Suffered a similar issue on a Centos 6.5 kernel when spawning multiple processes in a Twisted environment. Is this PR targeted for inclusion into the source tree? Thanks, Tom -- nosy: +Thomas Mortensson

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Changes by Antoine Pitrou : -- nosy: +neologix ___ Python tracker ___ ___ Python-bugs-list

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Changes by Raymond Hettinger : -- nosy: +davin ___ Python tracker ___ ___

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Changes by Charalampos Stratakis : -- pull_requests: +687 ___ Python tracker ___ ___

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Charalampos Stratakis added the comment: Patch for protecting the key list while forking. -- Added file: http://bugs.python.org/file46753/0001-Protect-key-list-during-fork.patch ___ Python tracker

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Charalampos Stratakis added the comment: In order to reproduce: Apply the python.patch from bz1268226_reproducer2.tar.gz Compile python Run the reproduce4.py from bz1268226_reproducer2.tar.gz As indicated by the reproducer, the status returned by os.wait() for the child is 139. I will

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Petr Viktorin added the comment: Here is a proof of concept patch from Jaroslav Škarvada. It fixes the problem by holding the mutex used for PyThread_create_key while forking. To make it more than PoC it needs adding _PyThread_AcquireKeyLock and _ReleaseKeyLock (similar to

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

Changes by Charalampos Stratakis : -- nosy: +cstratak ___ Python tracker ___ ___

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

New submission from Ján Stanček: Following crash is sporadically observed in RHEL7 anaconda: (gdb) f 0 #0 PyThread_ReInitTLS () at /usr/src/debug/Python-2.7.5/Python/thread.c:411 411 if (p->id != id) { (gdb) list 406 keymutex = PyThread_allocate_lock(); 407 408 /*