Yohei Uemura wrote: > Hi all, > > I'm Yohei. > I have a problem with making a script which use both multiprocessing and > C++ library. > My C++ library is made by using boost python. > The library worked well if it was imported normally. > That is, > "import simple as F > A = range(0,100) > for x in A: > print F.addition(x) > " > However, it couldn't work if it was called from multiprocessing. > That is, > "import simple as F > import multiprocessing as mp > pool = mp.Pool() > A = range(0,100) > results = pool.map(F.simple,A) > " > The error messages are shown in the attached file. > It seems a 'pickle' problem...
Attachments may be stripped off, so it's better to include the traceback into the message body. > Traceback (most recent call last): > File "testCXX.py", line 15, in <module> > results = pool.map(F.addition,A) > File "/usr/lib64/python2.7/multiprocessing/pool.py", line 251, in map > return self.map_async(func, iterable, chunksize).get() > File "/usr/lib64/python2.7/multiprocessing/pool.py", line 558, in get > raise self._value > cPickle.PicklingError: Can't pickle <type 'builtin_function_or_method'>: > attribute lookup __builtin__.builtin_function_or_method failed The processes communicate by passing pickled objects to each other. For some reason F.simple() doesn't have a __module__ attribute that correctly tells Python its origin, and thus pickling fails. You can use https://docs.python.org/dev/library/copyreg.html to work around that limitation or -- even easier -- wrap it into a Python function that can be pickled: ... def mysimple(arg): return F.simple(arg) ... results = pool.map(mysimple, A) _______________________________________________ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor