Heh, interestingly, if I add the following to the local dir and files when trying to unpickle under cpython, it works (note that cpython to pypy actually works out of the box, which I hadn't realized):
$ cat _numpypy/__init__.py from numpy.core import * $ cat _numpypy/multiarray.py from numpy.core.multiarray import * import numpy.core.multiarray as _ncm _reconstruct = _ncm._reconstruct This is obviously a total hack, and not one I'm comfortable with (since I need to use this codebase from both cpython and pypy), but it demonstrates that it's just bookkeeping that needs to change to get things to work. My first approach would be to add a wrapper around save_global here https://bitbucket.org/pypy/pypy/src/a0105e0d00dbd0f73d06fc704db704868a6c6ed2/lib-python/2.7/pickle.py?at=default&fileviewer=file-view-default#pickle.py-814 that special-cases the global '_numpypy.multiarray' to instead be 'numpy.core.multiarray'. That seem like a reasonable thing to do? Cheers, Eli On Fri, Jun 24, 2016 at 5:46 PM, Eli Stevens (Gmail) <wickedg...@gmail.com> wrote: > Okay, if I pass the pickles through pickletools.optimize, they look > identical except for the very first line (and a resulting systematic > shift in offset): > >>>> pt.dis(pt.optimize(open('cp123.pkl').read())) > 0: c GLOBAL 'numpy.core.multiarray _reconstruct' > >>>> pt.dis(pt.optimize(open('pp123.pkl').read())) > 0: c GLOBAL '_numpypy.multiarray _reconstruct' > > So I suspect that simply lying about what class we just pickled would > do the trick. > > I have no idea how acceptable that would be as a general solution, > though. Thoughts? > > Eli > > On Fri, Jun 24, 2016 at 2:29 PM, Eli Stevens (Gmail) > <wickedg...@gmail.com> wrote: >> Doesn't look like they are exactly the same: >> >> https://gist.github.com/elistevens/03e22f4684fb77d3edfe13ffcd406ef4 >> >> Certainly some similarities, though. >> >> I'm not familiar with the pickle format, and I haven't yet had time to >> dig in beyond this, though. Hoping I can tonight. >> >> Cheers, >> Eli >> >> >> On Fri, Jun 24, 2016 at 1:21 PM, matti picus <matti.pi...@gmail.com> wrote: >>> The first step would be to pickle the same dtype/shape/data ndarray once >>> from numpy and again from _numpypy, and to compare the binary result. The >>> only difference should be the class name, if the difference goes deeper >>> that difference must be fixed. Then it it just a matter of patching >>> pickle.py to use the desired class instead of the class name encoded into >>> the pickled binary result. >>> Matti >>> >>>> On 24 Jun 2016, at 10:43 PM, Eli Stevens (Gmail) <wickedg...@gmail.com> >>>> wrote: >>>> >>>> Yeah, looks like that's still the case: >>>> >>>>>>>> z = np.zeros((2,3), dtype=np.float32) >>>>>>>> z.tofile >>>> Traceback (most recent call last): >>>> File "<stdin>", line 1, in <module> >>>> AttributeError: 'numpy.ndarray' object has no attribute 'tofile' >>>> >>>> What would it take to get cross-interpreter numpy array pickles working? >>>> >>>> Thanks, >>>> Eli >>>> >>>> _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
