[issue42097] Python 3.7.9 logging/threading/fork hang

Alexey Izbyshev added the comment: By the way, I don't see a direct relation between `test.py` (which doesn't use `subprocess` directly) and your comment describing `subprocess` usage with threads. So if you think that the bug in `test.py` is unrelated to the problem you face, feel free to

[issue42097] Python 3.7.9 logging/threading/fork hang

Alexey Izbyshev added the comment: (Restored test.py attachment) The issue happens due to an incorrect usage of `multiprocessing.Pool`. ``` # Set up multiprocessing pool, initialising logging in each subprocess with multiprocessing.Pool(initializer=process_setup,

[issue42097] Python 3.7.9 logging/threading/fork hang

Change by Alexey Izbyshev : Added file: https://bugs.python.org/file49531/test.py ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue42097] Python 3.7.9 logging/threading/fork hang

Andrew de Quincey added the comment: Hi, sorry, I was ill on Friday, apologies for the delayed reply. We're not using os.fork() directly, and I'm afraid the multiprocessing thing I mentioned was a bit of a red herring. We're using a capped pool of threads to spawn subprocesses (using

[issue42097] Python 3.7.9 logging/threading/fork hang

Gregory P. Smith added the comment: If you use os.fork() or any of the multiprocessing start methods that call os.fork() with a process involving threads, this is expected behavior. os.fork() cannot be used in processes that have threads without potential for deadlock. Specifically, make

[issue42097] Python 3.7.9 logging/threading/fork hang

Change by Antoine Pitrou : -- nosy: +gregory.p.smith ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue42097] Python 3.7.9 logging/threading/fork hang

Terry J. Reedy added the comment: Have you found this in 3.8+? Since last June, 3.7 only gets security fixes, and this might have been fixed in later versions. -- nosy: +terry.reedy ___ Python tracker

[issue42097] Python 3.7.9 logging/threading/fork hang

Change by Karthikeyan Singaravelan : -- nosy: +vinay.sajip ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue42097] Python 3.7.9 logging/threading/fork hang

Change by Andrew de Quincey : Removed file: https://bugs.python.org/file49531/test.py ___ Python tracker ___ ___ Python-bugs-list mailing

[issue42097] Python 3.7.9 logging/threading/fork hang

Andrew de Quincey added the comment: Before anyone suggests it might be the logstash code we're calling in the comment, it ain't. The bug is easily replicated using entirely python built in code. See test.py attached. If you're having trouble triggering it (seems to vary per machine),

[issue42097] Python 3.7.9 logging/threading/fork hang

Andrew de Quincey added the comment: My colleague reports it does happen on their Mac as well, but that it took about 50 tries for it to trigger. -- ___ Python tracker ___

[issue42097] Python 3.7.9 logging/threading/fork hang

Andrew de Quincey added the comment: Debugging a bit more shows our code is deadlocked at: File "/usr/lib/python3.7/logging/__init__.py", line 219, in _acquireLock _lock.acquire() File "/usr/lib/python3.7/logging/config.py", line 501, in configure logging._acquireLock() File

[issue42097] Python 3.7.9 logging/threading/fork hang

New submission from Andrew de Quincey : We've been having some mysterious hangs in our threaded code. Debugging with GDB shows it ends up deadlocking during logging inside logging._acquireLock() I googled and found someone else with the same issue: