[issue13751] multiprocessing.pool hangs if any worker raises an Exception whose constructor requires a parameter
Richard Oudkerk shibt...@gmail.com added the comment: This is a duplicate of #9244 and #9400 which have been fixed by wrapping unpicklable exceptions in picklable exceptions. The larger issue of many exception classes being unpicklable, is dealt with in #1692335. -- resolution: - duplicate stage: - committed/rejected ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13751 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13751] multiprocessing.pool hangs if any worker raises an Exception whose constructor requires a parameter
Changes by Richard Oudkerk shibt...@gmail.com: -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13751 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13751] multiprocessing.pool hangs if any worker raises an Exception whose constructor requires a parameter
Terry J. Reedy tjre...@udel.edu added the comment: 2.6 and 3.1 only get security fixes. I am only guessing that this is still an issue for 3.2. A test with 3.2.2 would be good. If this is really about pickling behavior w/r/t exceptions that cannot be changed then this should be closed. -- nosy: +alexandre.vassalotti, jnoller, pitrou, terry.reedy versions: +Python 2.7, Python 3.2, Python 3.3 -Python 2.6 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13751 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13751] multiprocessing.pool hangs if any worker raises an Exception whose constructor requires a parameter
Faheem Mitha fah...@faheem.info added the comment: This is an issue with SQLAlchemy exceptions, and has been worked around by Mike Bayer in http://www.sqlalchemy.org/trac/ticket/2371 For the record, I think the real problem is that Python exception pickling is broken, see http://bugs.python.org/issue1692335 It would be nice to see this fixed, otherwise this issue will continue to plague the Python standard libraries as well as other libraries. Library writers don't seem to be aware of the issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13751 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13751] multiprocessing.pool hangs if any worker raises an Exception whose constructor requires a parameter
New submission from Faheem Mitha fah...@faheem.info:
See my question at
http://stackoverflow.com/questions/8785899/hang-in-python-script-using-sqlalchemy-and-multiprocessing
I can't improve on the analysis by Lorenzo Bolla,
so I reproduce his example below. This example hangs if
BadExc is thrown, but not if GoodExc is thrown.
The only difference between these is that the GoodExc does
not require an argument be passed to its constructor, while
BadExc does.
This looks like a bug to me, though I suppose it might
be some pickling limitation.
I have confirmed this behavior is present in 2.6.6, 2.7.2,
and 3.1.3, all tested on Debian squeeze.
Regards, Faheem
#
import multiprocessing
class BadExc(Exception):
def __init__(self, a):
'''Non-optional param in the constructor.'''
self.a = a
class GoodExc(Exception):
def __init__(self, a=None):
'''Optional param in the constructor.'''
self.a = a
def do(kwargs):
i = kwargs['i']
print i
raise BadExc('a')
# raise GoodExc('a')
return i
pool = multiprocessing.Pool(processes=5)
results = []
arglist = []
for i in range(10):
arglist.append({'i':i})
r = pool.map_async(do, arglist, callback=results.append)
try:
# set a timeout in order to be able to catch C-c
r.get(1e100)
except KeyboardInterrupt:
pass
print results
--
components: Library (Lib)
messages: 150975
nosy: fmitha
priority: normal
severity: normal
status: open
title: multiprocessing.pool hangs if any worker raises an Exception whose
constructor requires a parameter
type: behavior
versions: Python 2.6
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13751
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13751] multiprocessing.pool hangs if any worker raises an Exception whose constructor requires a parameter
sbt shibt...@gmail.com added the comment:
This is not specific to multiprocessing. It is really an issue with the
pickling of exceptions:
import cPickle
class BadExc(Exception):
... def __init__(self, a):
... '''Non-optional param in the constructor.'''
... self.a = a
...
a = cPickle.dumps(BadExc(1))
cPickle.loads(a)
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: ('__init__() takes exactly 2 arguments (1 given)', class
'__main__.BadExc', ())
I think that when you create a new exception class with an __init__() method,
you need to make sure that self.args is set properly by calling the __init__()
method of the parent class using the same arguments. So you can instead do
class BadExc(Exception):
def __init__(self, a):
'''Non-optional param in the constructor.'''
Exception.__init__(self, a)
self.a = a
--
nosy: +sbt
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13751
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13751] multiprocessing.pool hangs if any worker raises an Exception whose constructor requires a parameter
Faheem Mitha fah...@faheem.info added the comment: Thanks to sbt for his helpful and clear explanation. The following bug report appears relevant, http://bugs.python.org/issue1692335. It seems the proposed fix was never applied because it caused breakage to existing gcode. It is not clear to me whether this behavior is considered a bug or a feature. :-) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13751 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
