[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[STINNER Victor]

STINNER Victor added the comment:

We started to get *random* failures of test_queue_feeder_donot_stop_onexc() of 
test_multiprocessing_spawn since a few days. I may be related to this change. 
Can you please take a look at bpo-30595?

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Thomas Moreau]

Thomas Moreau added the comment:

I think this is a good solution as it let the user define easily the behavior 
it needs in other situation too. I would recommend adding the object 
responsible for the failure to the _on_queue_thread_error callback. This would 
simplify the error handling.


@@ -260,8 +260,16 @@ class Queue(object):
 info('error in queue thread: %s', e)
 return
 else:
-import traceback
-traceback.print_exc()
+self._on_queue_thread_error(e, obj)
+
+def _on_queue_thread_error(self, e, obj):
+"""
+Private API called when feeding data in the background thread
+raises an exception.  For overriding by concurrent.futures.
+"""
+import traceback
+traceback.print_exc()
+

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Antoine Pitrou]

Antoine Pitrou added the comment:

Thomas, thanks for the heads up.  I would suggest something like the following 
patch to multiprocessing.Pool:

$ git diff
diff --git a/Lib/multiprocessing/queues.py b/Lib/multiprocessing/queues.py
index 7f77837..ebbb360 100644
--- a/Lib/multiprocessing/queues.py
+++ b/Lib/multiprocessing/queues.py
@@ -260,8 +260,16 @@ class Queue(object):
 info('error in queue thread: %s', e)
 return
 else:
-import traceback
-traceback.print_exc()
+self._on_queue_thread_error(e)
+
+def _on_queue_thread_error(self, e):
+"""
+Private API called when feeding data in the background thread
+raises an exception.  For overriding by concurrent.futures.
+"""
+import traceback
+traceback.print_exc()
+
 
 _sentinel = object()
 


Then you can write your own Queue subclass in concurrent.futures to handle that 
error and clean up/restart whatever needs to be cleaned up or restarted.  What 
do you think?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Thomas Moreau]

Changes by Thomas Moreau :


--
pull_requests: +2001

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Thomas Moreau]

Thomas Moreau added the comment:

This fix, while preventing the Queue to crash, does not give any way to  
programatically detect that the message was dropped. This is a problem as we 
can no longer assume that the Queue will not drop messages. For instance, we 
can no longer detect deadlocks in concurrent.futures.ProcessPoolExecutor as 
done in https://github.com/python/cpython/pull/1013 where the crashed 
QueueFeederThread was used to monitor the working state of the executor.

We could either:
- Put a flag highlighting the fact that some messages where dropped.
- Add an argument to the Queue to close on pickling errors.

I'd be happy to work on a PR to implement any solution that you think is 
reasonable.

--
nosy: +tomMoral

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Antoine Pitrou]

Antoine Pitrou added the comment:

This is committed and pushed, thank you!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Antoine Pitrou]

Antoine Pitrou added the comment:


New changeset bdd964710deffe8593063dcb63157e5b55a82c61 by Antoine Pitrou in 
branch '2.7':
[2.7] bpo-30414: multiprocessing.Queue._feed do not break from main loop on exc 
(GH-1683) (#1817)
https://github.com/python/cpython/commit/bdd964710deffe8593063dcb63157e5b55a82c61


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Antoine Pitrou]

Antoine Pitrou added the comment:


New changeset 89004d761361332314beb08b443bff5b092ec36e by Antoine Pitrou in 
branch '3.5':
[3.5] bpo-30414: multiprocessing.Queue._feed do not break from main loop on exc 
(GH-1683) (#1816)
https://github.com/python/cpython/commit/89004d761361332314beb08b443bff5b092ec36e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Antoine Pitrou]

Antoine Pitrou added the comment:


New changeset 2783cc42629b9445ea848ce36bbf213ef7789271 by Antoine Pitrou in 
branch '3.6':
[3.6] bpo-30414: multiprocessing.Queue._feed do not break from main loop on exc 
(GH-1683) (#1815)
https://github.com/python/cpython/commit/2783cc42629b9445ea848ce36bbf213ef7789271


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
pull_requests: +1902

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
pull_requests: +1901

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
pull_requests: +1900

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Antoine Pitrou]

Antoine Pitrou added the comment:


New changeset bc50f03db4f58c869b78e98468e374d7e61f1227 by Antoine Pitrou 
(grzgrzgrz3) in branch 'master':
bpo-30414: multiprocessing.Queue._feed do not break from main loop on exc 
(#1683)
https://github.com/python/cpython/commit/bc50f03db4f58c869b78e98468e374d7e61f1227


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Antoine Pitrou]

Antoine Pitrou added the comment:

Nevermind, I saw the PR and the test case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
stage:  -> patch review
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Antoine Pitrou]

Antoine Pitrou added the comment:

Can you expand on which exceptions you are getting in the feeder thread?

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Xiang Zhang]

Changes by Xiang Zhang :


--
nosy: +davin, xiang.zhang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Grzegorz Grzywacz]

Changes by Grzegorz Grzywacz :


--
pull_requests: +1778

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

[Grzegorz Grzywacz]

New submission from Grzegorz Grzywacz:

multiprocessing.Queue is running background thread feeder. Feeder serialize and 
sends buffered data to pipe. 

The issue is with exception handling, feeder is catching all exceptions but out 
of main loop, so after exception is handled feeder is not going back to loop - 
thread finish. If feeder thread is not running any Queue.put will execute 
without exceptions but message not gonna be delivered.

Solution is to move exception handling inside main loop. I will provide PR.

I have run performance tests (found: #17025) and submitted patch do not affect 
performance.

--
components: Library (Lib)
messages: 294044
nosy: grzgrzgrz3
priority: normal
severity: normal
status: open
title: multiprocesing.Queue silently ignore messages after exc in _feeder
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com