New submission from Dimitrios Pritsos dprit...@extremepro.gr:
My name is Dimitrios and I am newbie in python. I am working on a Project (part
of my PhD) that is called Synergeticprocessing module. Initially is imitating
the multiprocessing built in module but the processes are distributed on a LAN
and not Locally. The main issue I have is with Pickle module. And I think I
found some kind of BUG in the built in multiprocessing module.
(Synergeticprocessing module is located at GitHub:
https://github.com/dpritsos/synergeticprocessing)
Starting with the BUG. In case someone uses the multiprocessing.Pool of
processes he/she has to face the problem of types.MehtodType Impossible
pickling. That is you cannot dispatch an class instance method to the to the
Process Pool. However, digging in to the Source Code of the module there are
few lines that resolve this issue however are not activated or they are
faultily activated so they do not work. This is the 'BUG'
@ ../multiprocessing/forking.py
.
.
.
#
# Try making some callable types picklable
#
from pickle import Pickler
class ForkingPickler(Pickler):
dispatch = Pickler.dispatch.copy()
@classmethod
def register(cls, type, reduce):
def dispatcher(self, obj):
rv = reduce(obj)
self.save_reduce(obj=obj, *rv)
cls.dispatch[type] = dispatcher
def _reduce_method(m):
if m.im_self is None:
return getattr, (m.im_class, m.im_func.func_name)
else:
return getattr, (m.im_self, m.im_func.func_name)
ForkingPickler.register(type(ForkingPickler.save), _reduce_method)
def _reduce_method_descriptor(m):
return getattr, (m.__objclass__, m.__name__)
ForkingPickler.register(type(list.append), _reduce_method_descriptor)
ForkingPickler.register(type(int.__add__), _reduce_method_descriptor)
#def _reduce_builtin_function_or_method(m):
#return getattr, (m.__self__, m.__name__)
#ForkingPickler.register(type(list().append),
_reduce_builtin_function_or_method)
#ForkingPickler.register(type(int().__add__),
_reduce_builtin_function_or_method)
.
.
.
The RED lines are not doing the job, for some reason they are not managing to
register the GREEN function as a global reduce/pickling function even if you
call the registration function into you __main__ script.
The solution I found is just to do this
import copy_reg
import types
def _reduce_method(m):
if m.im_self is None:
return getattr, (m.im_class, m.im_func.func_name)
else:
return getattr, (m.im_self, m.im_func.func_name)
copy_reg.pickle(types.MethodType, _reduce_method)
.
.
.
Doing that everything works FINE. But ONLY for local methods i.e. the ones that
their class is defined on the __main__ script or other import-ed.
In case you want to send something remotely (in an other machine) or to an
other __main__ script running separately then you get a message like this:
'module' object has no attribute 'my_class'
The only way to resolve this is firstly to import a script that has my_class
defined there and everything works fine.
SO the problems it seems to be that the m.im_class (look code above) has some
attribute __module__ defined as __module__ = '__main__' or something like that.
And this is the reason why remote script cannot execute the function. I mean
that the _reduce_method() above DOES is pickling the whole CLASS object so
there is no reason not to be executed at the remote script. Besides it does as
mentioned above in you just import this the user defined class form an other
script.
I have already spent about 12 weeks working on building my synergeticPool and
resolve the issue of Pickling and only 2 days needed for the code of the Pool
and the rest of the time was spent for the Pickling issues, and study all the
Class related mechanics of python. That was the reason I ve started digging the
multipocessessing module and found this say 'BUG', and finally sent this email.
Best Regards,
Dimitrios
--
components: None
messages: 124133
nosy: dimitrios
priority: normal
severity: normal
status: open
title: Multiprocessing module Pickling unPickling issues
type: behavior
versions: Python 2.6
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10717
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
