Is there a way to make pickle not depend on the specific submodule that a class is defined in? Wouldn't this happen again if you ever decided to rename _core.
The underscores in numpy._core._reconstruct don't actually do anything here in terms of making the interface not public, and if anything, are really misleading. I'm also curious about this more generally. We tend to think of the fully qualified name of a class as being an implementation detail for many libraries and only the top-level lib.Name should be used, but many things in the language (including this) break this. Aaron Meurer On Wed, Oct 11, 2023 at 4:49 PM Nathan <nathan.goldb...@gmail.com> wrote: > > > > On Wed, Oct 11, 2023 at 4:24 PM Mateusz Sokol <mso...@quansight.com> wrote: >> >> Hi! Thank you for all your feedback this week! >> >> We have made a decision to take a less disruptive option that we considered >> and that came up in this discussion. >> >> We back out of the `NumpyUnpickler` class solution for reading pickles >> across major NumPy versions. >> >> Instead, we will retain `numpy.core` stubs in NumPy 2.0 to allow loading >> NumPy 1.x pickles. >> Additionally, `numpy._core` stubs will be backported to 1.26 to ensure >> compatibility the other way around - loading NumPy 2.0 pickles with NumPy >> 1.26 installed. >> >> Both major versions will continue to create pickles with their own contents >> (NumPy 1.26 with `numpy.core` paths and NumPy 2.0 with `numpy._core` paths). >> >> This way any pickle will be loadable by both major versions. > > > Thanks for the summary Mateusz! > > I want to add that there will still be module-level `__getattr__` > implementations that will raise deprecation warnings on any attribute access > in `np.core`, `numpy.core.multiarray` or `numpy.core._multiarray_umath`, but > direct imports will not generate any warnings. Since pickles directly import > types that appear in pickle files, loading a pickle that refers to types or > functions in these modules won’t generate any warnings. > > Searching on github indicates that direct imports like this are relatively > rare in user code, which tend to either just import the top-level numpy > module and use attribute access or use `from` imports, which both invoke the > module-level `__getattr__`. Hopefully we’ll get most of the benefit of > alerting users that they are using private internals without needing to break > old pickles. > >> >> >> >> On Tue, Oct 10, 2023 at 3:33 PM Nathan <nathan.goldb...@gmail.com> wrote: >>> >>> >>> >>> On Tue, Oct 10, 2023 at 7:03 AM Ronald van Elburg >>> <r.a.j.van.elb...@hetnet.nl> wrote: >>>> >>>> I have one more useCase to consider from our ecosystem. >>>> >>>> We dump numpy arrays into a MongoDB using GridFS for subsequent >>>> visualization, some snippets: >>>> >>>> '''Python >>>> with BytesIO() as BIO: >>>> np.save(BIO, numpy_array) >>>> serialized_A = BIO.getvalue() >>>> filehandle_id = self.representations_files.put(serialized_A) >>>> ''' >>>> >>>> and then restore them in the other application: >>>> >>>> '''Python >>>> numpy_array = np.load(BytesIO(serializedA)) >>>> ''' >>>> For us this is for development work only and I am less concerned about >>>> having mixed versions in my database, but in principle that is a scenario. >>>> But it seems to me that for this to work the reading application needs to >>>> be migrated to version 2 and temporarily extended with the NumpyUnpickler >>>> before the writing application is migrated. Or they need to be migrated at >>>> the same time. Is that correct? >>> >>> >>> np.save and np.load will use NumpyUnpickler under the hood so you won’t >>> have any issues, you would only have issues if you saved or loaded pickles >>> using the pickle module directly. >>> >>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list -- numpy-discussion@python.org >>>> To unsubscribe send an email to numpy-discussion-le...@python.org >>>> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ >>>> Member address: nathan12...@gmail.com >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list -- numpy-discussion@python.org >>> To unsubscribe send an email to numpy-discussion-le...@python.org >>> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ >>> >>> Member address: mso...@quansight.com >> >> _______________________________________________ >> NumPy-Discussion mailing list -- numpy-discussion@python.org >> To unsubscribe send an email to numpy-discussion-le...@python.org >> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ >> Member address: nathan12...@gmail.com > > _______________________________________________ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: asmeu...@gmail.com _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
