[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Marc Schlaich]

Marc Schlaich  added the comment:

Davin, why isn't just one of the simple one-line patches applied. This would be 
much more reasonable instead of closing this as won't fix. 

This is a really ugly bug and it took me hours to find the cause...

--
nosy: +schlamar

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Davin Potts]

Davin Potts added the comment:

Richard already backported the fix for objects which can not be pickled but the 
second fix (applied to 3.3 in #1692335) would be problematic to similarly 
backport to 2.7.  Because there are reasonable workarounds for this in 2.7, 
going ahead with marking this closed.

--
resolution:  -> wont fix
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Davin Potts]

Davin Potts added the comment:

Reading through issue1692335 provides a sense of the concerns surrounding the 
patches they applied to the 3.x branches.  Attempting to backport those patches 
to the 2.7 branch involves non-trivial risk, magnified by the now numerous 
differences between those branches.

If we had a reasonable workaround to use in 2.7, that would simplify things.

Example workaround that produces behavior in 2.7.12 very like what's observed 
when using 3.5 or 3.6:
try:
subprocess.check_call(['dcon'], stdout=fh, stderr=fh)
except subprocess.CalledProcessError as cpe:
raise Exception(str(cpe))

--
assignee: sbt -> davin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Davin Potts]

Davin Potts added the comment:

For clarity:
* Confirmed, can repro on 2.7.12 on OSX.
* Also confirmed that this can not be reproduced on the 3.5 or 3.6 branches.

--
versions:  -Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Stanislaw Pitucha]

Stanislaw Pitucha added the comment:

Confirming this on python 2.7.12 on DragonflyBSD

--
nosy: +viraptor

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Doug Coleman]

Doug Coleman added the comment:

Six years later and I'm still running into this exact bug with 
``subprocess.CalledProcessError`` on python 2.7.12 when doing a 
``multiprocessing.Pool.map`` and trying to catch errors from 
``subprocess.check_output``. 

What's the reason it was never fixed and backported?

--
nosy: +Doug Coleman

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Ned Deily]

Changes by Ned Deily :


--
nosy: +davin
versions: +Python 3.5, Python 3.6 -Python 3.2, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Martin Dengler]

Changes by Martin Dengler :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Martin Dengler]

Changes by Martin Dengler mar...@martindengler.com:


--
nosy: +mdengler

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Torsten Bronger]

Changes by Torsten Bronger bron...@physik.rwth-aachen.de:


--
nosy: +bronger

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Richard Oudkerk]

Richard Oudkerk added the comment:

I am reopening this issue because 26bbff4562a7 only dealt with objects which 
cannot be pickled.  But CalledProcessError instances *can* be pickled: the 
problem is that the resulting data cannot be unpickled.

Note that in Python 3.3 CalledProcessError can be pickled then unpickled 
because of the fix for #1692335.  But there are other types which will still 
produce corrupt pickles.

--
assignee:  - sbt
resolution: fixed - 
stage: committed/rejected - needs patch
status: closed - open
versions: +Python 3.2, Python 3.3, Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Richard Oudkerk]

Changes by Richard Oudkerk shibt...@gmail.com:


--
resolution:  - fixed
stage: patch review - committed/rejected
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Roundup Robot]

Roundup Robot devn...@psf.upfronthosting.co.za added the comment:

New changeset 26bbff4562a7 by Richard Oudkerk in branch '2.7':
Issue #9400: Partial backport of fix for #9244
http://hg.python.org/cpython/rev/26bbff4562a7

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Richard Oudkerk]

Richard Oudkerk shibt...@gmail.com added the comment:

I have backported the fix for issue #9244 to 2.7.  This should fix the hang and 
produce a traceback containing a representation of the original error.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Richard Oudkerk]

Richard Oudkerk shibt...@gmail.com added the comment:

There are plenty of other bad exception classes apart from 
CalledProcessError, including TimeoutExpired in the same file.  In fact I 
suspect this is true of the majority of the exception classes in the stdlib 
which override __init__.  So I am not sure how much good it would do to fix 
just one example.

Python 3.x's Pool wraps bad exception instances in a MaybeEncodingError class 
which at least lets you see a stringification of the original exception.  I am 
not sure whether you would want to see a backport of this.  Even though 2.7 is 
in bug fix mode, I think a backport would still be appropriate since it stops a 
pickling error from killing a worker process, causing a hang.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Daniel Blanchard]

Daniel Blanchard dan.blanch...@gmail.com added the comment:

The patch appears to fix the issue, so is there any chance of this actually 
getting accepted this time? It seems bizarre that such a simple bug that has 
been on the books for almost two years now can't get a patch accepted.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +sbt

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Daniel Blanchard]

Daniel Blanchard dan.blanch...@gmail.com added the comment:

I believe I'm still encountering this issue in 2.7:


Exception in thread Thread-3:
Traceback (most recent call last):
  File /opt/python/2.7/lib/python2.7/threading.py, line 552, in 
__bootstrap_inner
self.run()
  File /opt/python/2.7/lib/python2.7/threading.py, line 505, in run
self.__target(*self.__args, **self.__kwargs)
  File /opt/python/2.7/lib/python2.7/multiprocessing/pool.py, line 347, in 
_handle_results
task = get()
TypeError: ('__init__() takes at least 3 arguments (1 given)', class 
'subprocess.CalledProcessError', ())

--
nosy: +Daniel.Blanchard
versions: +Python 2.7 -Python 2.6

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc amaur...@gmail.com added the comment:

Here is another patch, with test.

--
nosy: +amaury.forgeotdarc
stage: test needed - patch review
Added file: http://bugs.python.org/file25111/picklable_process_exception.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Ray.Allen]

Ray.Allen ysj@gmail.com added the comment:

This is because when an subprocess.CalledProcessError is raised, the 
CalledProcessError instance is picked through a socket and then read by 
parent process, but in fact CalledProcessError instances cannot be picked 
correctly. Because CalledProcessError extends PyExc_BaseException, which 
defines a __reduce__ method, that special method cause the pickle load to call 
the exception type's __init__ method with packed self.args as arguments. So if 
a subclass of Exception needs to behave correctly in pickling, it should make 
self.args meats its __init__ method's function signature. That is, ensure the 
calling to:
  self.__init__(*self.args)
has no problem. 

But CalledProcessError doesn't meat this requirement. Here is a patch fixing 
the CalledProcessError.__init__ to call its base class's __init__ to store its 
arguments in self.args, which can fix this bug.

--
keywords: +patch
Added file: http://bugs.python.org/file18309/issue9400.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Ray.Allen]

Ray.Allen ysj@gmail.com added the comment:

I got OSError: [Errno 2] No such file or directory instead of TypeError when 
running the bug.py, did I miss something?

--
nosy: +ysj.ray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Jesse Noller]

Jesse Noller jnol...@gmail.com added the comment:

@ray - you probably don't have the dcon binary on your path. bug.py calls a 
subprocess call.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Nikolaus Rath]

Nikolaus Rath nikol...@rath.org added the comment:

@ray: Try it with the following dummy dcon program:

$ cat dcon 
#!/bin/sh

exit 127

(and change the path to dcon in bug.py accordingly).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[Nikolaus Rath]

New submission from Nikolaus Rath nikol...@rath.org:

The attached test program calls apply_async with a function that will raise 
CalledProcessError. However, when result.get() is called, it raises a TypeError 
and the program hangs:

$ ./bug.py
ERROR:root:ops
Traceback (most recent call last):
  File ./bug.py, line 19, in run_with
dW1 = run_dcon()
  File ./bug.py, line 26, in run_dcon
subprocess.check_call(['dcon'], stdout=fh, stderr=fh)
  File /usr/lib/python2.6/subprocess.py, line 498, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['dcon']' returned non-zero exit status 127
Exception in thread Thread-2:
Traceback (most recent call last):
  File /usr/lib/python2.6/threading.py, line 532, in __bootstrap_inner
self.run()
  File /usr/lib/python2.6/threading.py, line 484, in run
self.__target(*self.__args, **self.__kwargs)
  File /usr/lib/python2.6/multiprocessing/pool.py, line 259, in 
_handle_results
task = get()
TypeError: ('__init__() takes exactly 3 arguments (1 given)', class 
'subprocess.CalledProcessError', ())

--
components: Library (Lib)
files: bug.py
messages: 111827
nosy: Nikratio
priority: normal
severity: normal
status: open
title: multiprocessing.pool.AsyncResult.get() messes up exceptions
type: behavior
versions: Python 2.6
Added file: http://bugs.python.org/file18233/bug.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9400] multiprocessing.pool.AsyncResult.get() messes up exceptions

[R. David Murray]

Changes by R. David Murray rdmur...@bitdance.com:


--
nosy: +jnoller
stage:  - unit test needed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9400
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com