[issue45077] multiprocessing.Pool(64) crashes on Windows

[Dong-hee Na]

Change by Dong-hee Na :


--
nosy: +corona10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45077] multiprocessing.Pool(64) crashes on Windows

[Eryk Sun]

Eryk Sun  added the comment:

See bpo-26903 for a similar problem in concurrent.futures.ProcessPoolExecutor. 
It was resolved by adding a limit constant, _MAX_WINDOWS_WORKERS == 61. 

WaitForMultipleObjects() can wait on up to 64 object handles, but in this case 
3 slots are already taken. The pool wait includes two events for its output and 
change-notifier queues (named pipes), plus the _winapi module always reserves a 
slot for the SIGINT event, even though this event is only used by waits on the 
main thread.

To avoid the need to limit the pool size, connection._exhaustive_wait() could 
be modified to combine simultaneous waits on up to 63 threads, for which each 
thread exhaustively populates a list of up to 64 signaled objects. I wouldn't 
want to modify _winapi.WaitForMultipleObjects, but the exhaustive wait should 
still be implemented in C, probably in the _multiprocessing extension module. A 
benefit of implementing _exhaustive_wait() in C is lightweight thread creation, 
directly with CreateThread() and a relatively small stack commit size.

--
components: +Library (Lib)
nosy: +eryksun
versions: +Python 3.10, Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45077] multiprocessing.Pool(64) crashes on Windows

[Kagami Sascha Rosylight]

Kagami Sascha Rosylight  added the comment:

The argument-less instantiation also fails, which is worse.

```
>>> multiprocessing.Pool()

>>> Exception in thread Thread-1:
Traceback (most recent call last):
  File 
"C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 
973, in _bootstrap_inner
self.run()
  File 
"C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 
910, in run
self._target(*self._args, **self._kwargs)
  File 
"C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py",
 line 519, in _handle_workers
cls._wait_for_updates(current_sentinels, change_notifier)
  File 
"C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py",
 line 499, in _wait_for_updates
wait(sentinels, timeout=timeout)
  File 
"C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py",
 line 884, in wait
ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File 
"C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py",
 line 816, in _exhaustive_wait
res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 66
```

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45077] multiprocessing.Pool(64) crashes on Windows

[Kagami Sascha Rosylight]

New submission from Kagami Sascha Rosylight :

Similar issue as the previous issue 26903.

```
Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit 
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> multiprocessing.cpu_count()
64
>>> multiprocessing.Pool(multiprocessing.cpu_count())
Exception in thread Thread-1:
Traceback (most recent call last):

  File 
"C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 
973, in _bootstrap_inner
>>> self.run()
  File 
"C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 
910, in run
self._target(*self._args, **self._kwargs)
  File 
"C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py",
 line 519, in _handle_workers
cls._wait_for_updates(current_sentinels, change_notifier)
  File 
"C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py",
 line 499, in _wait_for_updates
wait(sentinels, timeout=timeout)
  File 
"C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py",
 line 884, in wait
ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File 
"C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py",
 line 816, in _exhaustive_wait
res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 66
```

--
components: Windows
messages: 400832
nosy: paul.moore, saschanaz, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: multiprocessing.Pool(64) crashes on Windows
type: behavior
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com