[issue44615] Multiprocessing Pool example and comments incomplete/wrong?

[Florian Streibelt]

New submission from Florian Streibelt :

The docs at the url

https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool

state in a red warning box:

"Warning

multiprocessing.pool objects have internal resources that need to be properly 
managed (like any other resource) by using the pool as a context manager or by 
calling close() and terminate() manually. Failure to do this can lead to the 
process hanging on finalization."


however, when using the context manager, as it is also shown in the examples, 
the context manager will call terminate() instead of close() and join() on the 
pool, possible leading to deadlocks if I understand the documentation correct.

It seems the only safe way of using a Pool as Context Manager is to manually 
call pool.close() and pool.join() prior to leaving it, which, to be honest, 
does not make sense to me.


see pool.py:

def __exit__(self, exc_type, exc_val, exc_tb):
self.terminate()

--
assignee: docs@python
components: Documentation
messages: 397360
nosy: docs@python, mutax
priority: normal
severity: normal
status: open
title: Multiprocessing Pool example and comments incomplete/wrong?
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue44615>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44614] Broken Pipe in Server of Manager in multiprocessing when finalizing, sometimes

[Florian Streibelt]

New submission from Florian Streibelt :

It seems that the shutdown() method of the Server in in managers.py is already 
sending the result of its execution directly into the connection, and then the 
handle_request() method is trying to send ('#RETURN', None) yet again after 
returning from the call to shutdown() which sometimes yields a Broken Pipe 
Exception.

with the following line in handle_request():
result = func(c, *args, **kwds)
a call to shutown() is made, and in shutdown() I see:
c.send(('#RETURN', None))

and back in handle_request():
msg = ('#RETURN', result)
…
c.send(msg)

attached you find a minimal example that shows this behaviour sometimes, if 
logging is set to not more than DEBUG. I am guessing that the Broken Pipe 
depends on a race with the other side being closed.

Although the race condition does not look like having an impact, it is highly 
confusing when debugging. 


log:
  INFO   MainProcess util.py:54  sending shutdown message 
to manager
 DEBUGBaseManager-12 util.py:50  manager received shutdown 
message
  INFOBaseManager-12 util.py:54  Failure to send message: 
('#RETURN', None)
  INFOBaseManager-12 util.py:54  process shutting down
  INFOBaseManager-12 util.py:54   ... request was (None, 
'shutdown', (), {})
 DEBUGBaseManager-12 util.py:50  running all "atexit" 
finalizers with priority >= 0
  INFOBaseManager-12 util.py:54   ... exception was 
BrokenPipeError(32, 'Broken pipe')
 DEBUGBaseManager-12 util.py:50  running the remaining 
"atexit" finalizers
  INFOBaseManager-12 util.py:54  process exiting with 
exitcode 0

--
components: Library (Lib)
files: race2.py
messages: 397358
nosy: mutax
priority: normal
severity: normal
status: open
title: Broken Pipe in Server of Manager in multiprocessing when finalizing, 
sometimes
type: behavior
versions: Python 3.9
Added file: https://bugs.python.org/file50145/race2.py

___
Python tracker 
<https://bugs.python.org/issue44614>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com