On 4/4/11 7:05 PM, Robert Kern wrote:
On 4/4/11 3:20 PM, John Ladasky wrote:
However, at least in Python 2.7, multiprocessing seems to have a C extension
module defining the Connection objects. Unfortunately, it looks like this C
extension just imports the regular pickler that is not aware of these custom
types. That's why you get this error. I believe this is a bug in Python.
So what did you try, and what output did you get? What version of Python are you
using?
I think that my understanding of multiprocessing
needs to evolve beyond the use of Pool, but I'm not sure yet. This
post suggests as much.
http://mail.scipy.org/pipermail/scipy-user/2009-February/019696.html
Maybe. If the __reduce_ex__() method is implemented properly (and
multiprocessing bugs aren't getting in the way), you ought to be able to pass
them to a Pool just fine. You just need to make sure that the shared arrays are
allocated before the Pool is started. And this only works on UNIX machines. The
shared memory objects that shmarray uses can only be inherited. I believe that's
what Sturla was getting at.
I'll take that back a little bit. Since the underlying shared memory types can
only be shared by inheritance, the ForkingPickler is only used for the arguments
passed to Process(), and not used for things sent through Queues (which Pool
uses). Since Pool cannot guarantee that the data exists before the Pool starts
its subprocesses, it must use the general mechanism.
So in short, if you pass the shmarrays as arguments to the target function in
Process(), it should work fine.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
--
http://mail.python.org/mailman/listinfo/python-list
