[issue45021] Race condition in thread.py

[Antoine Pitrou]

Antoine Pitrou  added the comment:

As the multiprocessing doc says 
(https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods):
"""Note that safely forking a multithreaded process is problematic."""

The reproducer is trivially fixed by adding a call to 
`multiprocessing.set_start_method("forkserver")` (you can also replace 
"forkserver" with "spawn")

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45021] Race condition in thread.py

[Antoine Pitrou]

Change by Antoine Pitrou :


--
versions: +Python 3.10, Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45021] Race condition in thread.py

[0x0L]

0x0L  added the comment:

In the last reproducing example, one can drop the .result() and just submit the 
task: we're not waiting on any result explicitly. It's the shutdown of the 
ProcessPoolExecutor that waits forever for the children to exit.

Generally speaking it's probably a bad idea to mix multi-thread and forks. See 
https://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45021] Race condition in thread.py

[hai shi]

hai shi  added the comment:

Is it a defined behavior?
I got this sentence from pep-3148: Deadlock can occur when the callable 
associated with a Future waits on the results of another Future.

--
nosy: +shihai1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45021] Race condition in thread.py

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45021] Race condition in thread.py

[STINNER Victor]

STINNER Victor  added the comment:

I don't understand anything about multiprocessing, nor its API, but some people 
seem to find it useful :-D I remove myself from the nosy list ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45021] Race condition in thread.py

[Raymond Hettinger]

Change by Raymond Hettinger :


--
nosy: +davin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45021] Race condition in thread.py

[nullptr]

nullptr  added the comment:

Simplifying the reproducing example a bit more:


from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from time import sleep

def submit(pool):
pool.submit(submit, pool)


if __name__ == '__main__':
pool = ThreadPoolExecutor(1)
pool.submit(submit, pool)

while True:
with ProcessPoolExecutor() as workers:
print('WORK')
workers.submit(sleep, 0.01).result()
print('DONE')
print('OK')

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45021] Race condition in thread.py

[nullptr]

nullptr  added the comment:

I don't think so: this issue would only arise when using fork on linux and not 
spawn (win/osx) like it seems your system is doing

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45021] Race condition in thread.py

[Irit Katriel]

Irit Katriel  added the comment:

I'm seeing this (intermittently) on main branch, could be related?

iritkatriel@Irits-MBP cpython-1 % ./python.exe 
Lib/test/test_importlib/partial/pool_in_threads.py

Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File 
"/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py",
 line 9, in t
with multiprocessing.Pool(1):
 ^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 
119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
   
  File 
"/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py",
 line 9, in t
with multiprocessing.Pool(1):
 ^^^
  File 
"/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py",
 line 9, in t
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 
212, in __init__
self._repopulate_pool()
^^^
  File 
"/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py",
 line 9, in t
with multiprocessing.Pool(1):
 ^^^
  File 
"/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py",
 line 9, in t
with multiprocessing.Pool(1):
 ^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 
119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
   
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 
119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
   
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 
119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
   
  File 
"/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py",
 line 9, in t
with multiprocessing.Pool(1):
 ^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 
119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
   
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 
212, in __init__
self._repopulate_pool()
^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 
212, in __init__
self._repopulate_pool()
^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 
212, in __init__
self._repopulate_pool()
^^^
Traceback (most recent call last):
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 
303, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
   ^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 
303, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
   ^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 
303, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
   ^
  File 
"/Users/iritkatriel/src/cpython-1/Lib/test/test_importlib/partial/pool_in_threads.py",
 line 9, in t
with multiprocessing.Pool(1):
 ^^^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 
326, in _repopulate_pool_static
w.start()
^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 
326, in _repopulate_pool_static
w.start()
^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/pool.py", line 
326, in _repopulate_pool_static
w.start()
^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 
119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
   
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/process.py", line 
121, in start
self._popen = self._Popen(self)
  ^
  File "/Users/iritkatriel/src/cpython-1/Lib/multiprocessing/context.py", line 
284, in _Popen
return Popen(process_obj)
   

[issue45021] Race condition in thread.py

[Irit Katriel]

Change by Irit Katriel :


--
nosy: +pitrou, vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45021] Race condition in thread.py

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch
nosy: +python-dev
nosy_count: 1.0 -> 2.0
pull_requests: +26449
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/28007

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45021] Race condition in thread.py

[nullptr]

nullptr  added the comment:

A more direct way to reproduce


from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from time import sleep

def worker():
with ProcessPoolExecutor() as pool:
r = list(pool.map(sleep, [0.01] * 8))


def submit(pool):
pool.submit(submit, pool)


if __name__ == '__main__':
pool = ThreadPoolExecutor(2)
pool.submit(submit, pool)

i = 0
while True:
r = pool.submit(worker)
r = r.result()
print(i)
i += 1

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45021] Race condition in thread.py

[nullptr]

Change by nullptr :


--
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45021] Race condition in thread.py

[nullptr]

New submission from nullptr :

The following code can sometimes hang up


import random 
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
from time import sleep

def worker():   
with ProcessPoolExecutor() as pool:  
r = list(pool.map(sleep, [0.01] * 8))   

   
if __name__ == '__main__': 
pool = ThreadPoolExecutor() 
   
i = 0  
while True:
if random.random() < 0.9:
pool.submit(sleep, 0.001)  
else: 
r = pool.submit(worker) 
r = r.result()
i += 1   
print('alive', i)


It's a bit hard to trigger that way but with some luck and many restarts it'll 
eventually freeze as r.result() never returns.

The backtrace from a child process shows that the child is stuck in 
Lib/concurrent/futures/thread.py:_python_exit waiting for _global_shutdown_lock.

The fork happened while the lock was already grabbed i.e. while executing 
ThreadPoolExecutor.submit

--
components: Library (Lib)
messages: 400378
nosy: xavier.lacroze
priority: normal
severity: normal
status: open
title: Race condition in thread.py
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com