On Mon, Oct 9, 2023 at 2:44 PM Oscar Benjamin <oscar.j.benja...@gmail.com> wrote:
> On Mon, 9 Oct 2023 at 17:03, Nathan <nathan.goldb...@gmail.com> wrote: > > > > On Mon, Oct 9, 2023 at 12:57 AM Aaron Meurer <asmeu...@gmail.com> wrote: > >> > >> Is it possible to convert a NumPy 1 pickle file into a generic pickle > >> file that works in both NumPy 1 and 2? As far as I understand, pickle > >> is Turing complete, so I imagine it should be theoretically possible, > >> but I don't know how easy it would be to actually do this or how it > >> would affect the pickle file size. > > There are many ways that this could be made to work with the various > options like reduce() etc. > > > The issue is that the pickle protocol needs a reference to a > reconstructor to recreate numpy types. For ndarray, that function is > currently `numpy.core.multiarray._reconstruct` and in numpy 2 becomes > numpy._core.multiarray.reconstruct. For a pickle file containing only an > ndarray, this is the first thing in the pickle file and the import happens > inside of the pickle implementation. I am not aware of a hook that Python > gives us to intercept that path before Python imports it. > > > > So, even if there is a way to correct subsequent paths in the pickle > file, we won't be able to fix the most problematic path that will occur in > any pickle that includes a numpy array. That means some user-visible pain > no matter what. If we can't avoid that, I'd prefer to offer a solution that > will allow people to continue loading old pickle files indefinitely (albeit > with a minor code change). This also gives us a place to put compatibility > fixes for future changes that impact old pickle files. > > Hi Oscar, > Suppose that there is NumPy v1 and that in future there will be NumPy > v2. Also suppose that there will be two NumPy pickle formats fmtA and > a future fmtB. One possibility is that NumPy v1 only reads and writes > fmtA and then NumPy v2 only reads and writes fmtB. One problem with > this is that when NumPy v2 comes out there is no easy way to convert > pickles from fmtA to fmtB for compatibility with NumPy v2. Another > problem with this is that it does not make a nice transition during > any period of time when both NumPy v1 and v2 might be used in > different parts of a software stack. > Doesn't NumpyUnpickler solve this? It will be present in both v1 and v2 and will allow loading files either np.core or np._core in either version. > An alternative is to introduce fmtB as part of the NumPy v1 series. > NumPy could be changed now so that it can read both fmtA and fmtB but > by default it would write fmtB which would be designed ahead of time > so that in future NumPy v2 would be able to read fmtB as well. It > would also be possible to design it so that fmtB would be readable by > older versions of NumPy that were released before fmtB was designed. > > Then there is a version of NumPy (v1) which can read fmtA and write to > fmtB. This version of NumPy can be used to convert pickles from fmtA > to fmtB. Then when NumPy v2 is released it can already read any > pickles that were generated by the most recent releases of NumPy v1.x. > Anyone who still has older pickles in fmtA could use NumPy v1 to do > dumps(loads(f)) which would convert from fmtA to fmtB. > > In this scenario the only part that does not work is reading fmtA in > NumPy v2 which is unavoidable if numpy.core is removed or renamed in > v2. > I agree it would have been better to anticipate this and move the _reconstruct function to np._core many releases ago. Sadly this was not done and the next release is Numpy 2.0. I also want to emphasize that using pickle like this - to share data between different python installations - is inherently insecure and should never be done outside of an organization that fully controls all of the python installations. In that case, the organization can use NumpyUnpickler. In any other case, I think it's good to perhaps nudge people away from doing things like this. > > -- > Oscar > _______________________________________________ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: nathan12...@gmail.com >
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
