I hadn't seen the original message (perhaps it fell through the GMane migration). I think this would indeed be a worthwhile addition to pickle.
Is it possible to continue this discussion on python-dev? I don't think this is a PEP-level topic. Regards Antoine. On Fri, 7 Feb 2020 12:48:48 -0500 Daniel Spitz <spitz.da...@gmail.com> wrote: > I just want to say, this would be extremely powerful. > > Awhile ago I implemented a demo that wraps the pickle extension machinery > to enable "object graph transformers": > https://gist.github.com/spitz-dan-l/3375a61fac6fe150574fac791567af4f (demo > usage starts at line 307). Effectively it lets you match-and-transform > arbitrary python object graphs that can include cycles and deep nesting, > without writing any traversal logic yourself. In the demo I used it to > implement a simple arithmetic expression parser/evaluator, that lets you go > from strings like "3 + (4 + (5) + 7)" to the int 19. > > But, it uses pickle under the hood, because pickle is what knows how to > traverse object graphs with cycles, and pickle is what knows how to use the > extensible __reduce__ machinery. And this makes the entire demo silly and > really slow. (And, that was kind of the point, to do something interesting > using esoteric, inappropriate corners of python.) > > If Andrew's proposal happened, pickle would be uncoupled from that core > machinery, and my idea, as well as other interesting ideas, could be > implemented in a non-silly way. > > Daniel Spitz > > On Thu, Jan 23, 2020 at 7:11 PM Andrew Barnert via Python-ideas < > python-ideas@python.org> wrote: > > > Pickling uses an extensible protocol that lets any class determine how its > > instances can be deconstructed and reconstructed. Both `pickle` and `copy` > > use this protocol, but it could be useful more generally. Unfortunately, to > > use it more generally requires relying on undocumented details. I think we > > should expose a couple of helpers to fix that: > > > > # Return the same (shallow) reduction tuple that pickle.py, copy.py, > > and _pickle.c would use > > pickle.reduce(obj) -> (callable, args[, state[, litems[, ditem[, > > statefunc]]]]) > > > > # Return a callable and arguments to construct a (shallow) equivalent > > object > > # Raise a TypeError when that isn't possible > > pickle.deconstruct(obj) -> callable, args, kw > > > > So, why do you want these? > > > > There are many cases where you want to "deconstruct" an object if > > possible. Pattern matching depends on being able to deconstruct objects > > like this. Auto-generating a `__repr__` as suggested in Chris's thread. > > Quick&dirty REPL stuff, and deeper reflection stuff using > > `inspect.Signature` and friends. > > > > Of course not every type tells `pickle` what to do in an appropriate way > > that we can use, but a pretty broad range of types do, including (I think; > > I haven't double-checked all of them) `@dataclass`, `namedtuple`, > > `@attr.s`, many builtin and extension types, almost all reasonable types > > that use `copyreg`, and any class that pickles via the simplest > > customization hook `__getnewargs[_ex]__`. That's more than enough to be > > useful. And, just as important, it won't (except in intentionally > > pathological cases) give us a false positive, where a type is correctly > > pickleable and we think we can deconstruct it but the deconstruction is > > wrong. (For some uses, you are going to want to fall back to heuristics > > that are often right but sometimes misleadingly wrong, but I don't think > > the `pickle` module should offer anything like that. Maybe `inspect` > > should.) > > > > The way to get the necessary information isn't fully documented, and > > neither is the way to interpret it. And I don't think it _should_ be > > documented, because it changes every so often, and for good reasons; we > > don't want anyone writing third-party code that relies on those details. > > Plus, a different Python implementation might conceivably do it > > differently. Public helpers exposed from `pickle` itself won't have those > > problems. > > > > Here's a first take at the code. > > > > def reduce(obj, proto=pickle.DEFAULT_PROTOCOL): > > """reduce(obj) -> (callable, args[, state[, litems[, ditem[, > > statefunc]]]]) > > Return the same reduction tuple that the pickle and copy modules > > use > > """ > > cls = type(obj) > > if reductor := copyreg.dispatch_table.get(cls): > > return reductor(obj) > > # Note that this is not a special method call (not looked up on the type) > > if reductor := getattr(obj, "__reduce_ex__"): > > return reductor(proto) > > if reductor := getattr(obj, "__reduce__"): > > return reductor() > > raise TypeError(f"{cls.__name__} objects are not reducible") > > > > def deconstruct(obj): > > """deconstruct(obj) -> callable, args, kw > > callable(*args, **kw) will construct an equivalent object > > """ > > reduction = reduce(obj) > > # If any of the optional members are included, pickle/copy has to > > # modify the object after construction, so there is no useful single > > # call we can deconstruct to. > > if any(reduction[2:]): > > raise TypeError(f"{type(obj).__name__} objects are not > > deconstrutable") > > func, args, *_ = reduction > > # Most types (including @dataclass, namedtuple, and many builtins) > > # use copyreg.__newobj__ as the constructor func. The args tuple is > > # the type (or, when appropriate, some other registered > > # constructor) followed by the actual args. However, any function > > # with the same name will be treated the same way (because under the > > # covers, this is optimized to a special opcode). > > if func.__name__ == "__newobj__": > > return args[0], args[1:], {} > > # Mainly only used by types that implement __getnewargs_ex__ use > > # copyreg.__newobj_ex__ as the constructor func. The args tuple > > # holds the type, *args tuple, and **kwargs dict. Again, this is > > # special-cased by name. > > if func.__name__ == "__newobj_ex__": > > return args > > # If any other special copyreg functions are added in the future, > > # this code won't know how to handle them, so bail. > > if func.__module__ == 'copyreg': > > raise TypeError(f"{type(obj).__name__} objects are not > > deconstrutable") > > # Otherwise, the type implements a custom __reduce__ or __reduce_ex__, > > # and whatever it specifies as the constructor is the real constructor. > > return func, args, {} > > > > Actually looking at that code, I think it makes a better argument for why > > we don't want to make all the internal details public. :) > > > > Here are some quick (completely untested) examples of other things we > > could build on it. > > > > # in inspect > > def deconstruct(obj): > > """deconstruct(obj) -> callable, bound_args > > Calling the callable on the bound_args would construct an equivalent object > > """ > > func, args, kw = pickle.deconstruct(obj) > > sig = inspect.signature(func) > > return func, sig.bind(*args, **kw) > > > > # in reprlib, for your __repr__ to delegate to > > def auto_repr(obj): > > func, bound_args = inspect.deconstruct(obj) > > args = itertools.chain( > > map(repr, bound_args.args), > > (f"{key!r}={value!r}" for key, value in bound_args.kwargs.items())) > > return f"{func.__name__}({', '.join(args)})" > > > > # or maybe as a class decorator > > def auto_repr(cls): > > def __repr__(self): > > func, bound_args = inspect.deconstruct(self) > > args = itertools.chain( > > map(repr, bound_args.args), > > (f"{key!r}={value!r}" for key, value in bound_args.kwargs.items())) > > return f"{func.__name__}({', '.join(args)})" > > cls.__repr__ = __repr__ > > return cls > > > > _______________________________________________ > > Python-ideas mailing list -- python-ideas@python.org > > To unsubscribe send an email to python-ideas-le...@python.org > > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > > Message archived at > > https://mail.python.org/archives/list/python-ideas@python.org/message/RTZGM7L7JOTKQQICN6XDSLOAMU4A62CA/ > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/6I6LDMNELAK7HZVEOLOLXUUCURH4YLN3/ Code of Conduct: http://python.org/psf/codeofconduct/
