[issue43093] Make modules picklable

[Myst]

Change by Myst :


--
nosy: +LeMyst

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43093] Make modules picklable

[Cornelius Krupp]

Cornelius Krupp  added the comment:

I would also like to see this added, and I can also provided a usecase where I 
naturally came across this.

I am helping on a parsing library which uses regular expressions internally. We 
allow the user to select the stdlib re module or the third party regex module 
and store the selected module as an attribute in multiple different objects, 
since those modules (by design) have pretty much compatible interfaces. This 
however makes it unpicklable, unless one adds a custom __reduce__ function for 
each of those classes.

Most notably, this issue also extends to deepcopy and the multiprocessing 
module.

--
nosy: +MegaIng

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43093] Make modules picklable

[Arusekk]

Arusekk  added the comment:

Sorry, I forgot to state what is my actual goal: for the module objects to be 
pickleable in native CPython (possibly from C layer) without the need to add 
this to every code that wants to pickle module objects.

The point is that they can already be unpickled from the representation 
generated by the code, so it should only be necessary to implement __reduce__ 
in moduleobject.c, and probably to change the object's qualname / module 
attributes. Or to introduce a helper function, like PyPy did, for the fake 
module case; I just found supporting existing unpickling paths more elegant 
(and working across all Python versions since 2.0.0), as much as unelegant is 
the monkey-patching done in my example. It is possible that (for the actual 
module case) _compat_pickle.REVERSE_IMPORT_MAPPING should be considered as well 
for the old protocols (which would probably imply using __reduce_ex__ instead), 
but I did not explore that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43093] Make modules picklable

[Arusekk]

New submission from Arusekk :

Currently pickling functions and types stores modules by their name.

So I believe it is possible to support pickling module objects with the 
following code (based on the logic in PyPy, which supports pickling modules):


import copyreg
import types
import pickle
import sys

def picklemod(mod):
if mod.__name__ in sys.modules:  # real modules
return (__import__, (mod.__name__, None, None, ('',)))

# module objects created manually:
return (types.ModuleType, (mod.__name__,), mod.__dict__)
copyreg.pickle(types.ModuleType, picklemod)

pickle.loads(pickle.dumps(sys))  # works
import http.server
pickle.loads(pickle.dumps(http.server))  # works for nested modules

fakemod = types.ModuleType('fakemod')
fakemod.field1 = 'whatever'

# This fake type is used instead of types.ModuleType in order to re-confuse 
pickle back on track.
# Should not have been necessary in the first place,
# but types.ModuleType has misconfigured fields according to pickle
# (and they are read-only).
class _types_ModuleType(types.ModuleType):
__module__ = 'types'
__name__ = __qualname__ = 'ModuleType'

_orig_types_ModuleType = types.ModuleType
# bad monkey-patching, but needed for the confusion to work
types.ModuleType = _types_ModuleType
dump = pickle.dumps(fakemod)
# not necessary, but to show unpickling is regardless correct
types.ModuleType = _orig_types_ModuleType

pickle.loads(dump).field1  # works


Disclaimer: I do not see any specific use for this, I was just surprised while 
trying to port snakeoil library to PyPy, which (snakeoil) uses sys module as an 
example of an unpicklable object (they should switch to a traceback for 
instance, but that is not the scope of this issue).

--
components: Library (Lib)
messages: 386090
nosy: Arusekk
priority: normal
severity: normal
status: open
title: Make modules picklable
type: enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com