By the way, I think I am referring to integer arrays. (Or integer part of floats.)
I don’t think what I am saying sensibly applies to floats as they are. Although, new float type could base its integer part on such concept. — Where I am coming from is that I started to hit maximum bounds on integer arrays, where most of values are very small and some become very large. And I am hitting memory limits. And I don’t have many zeros, so sparse arrays aren’t an option. Approximately: 90% of my arrays could fit into `np.uint8` 1% requires `np.uint64` the rest 9% are in between. And there is no predictable order where is what, so splitting is not an option either. > On 13 Mar 2024, at 17:53, Nathan <nathan.goldb...@gmail.com> wrote: > > Yes, an array of references still has a fixed size width in the array buffer. > You can think of each entry in the array as a pointer to some other memory on > the heap, which can be a dynamic memory allocation. > > There's no way in NumPy to support variable-sized array elements in the array > buffer, since that assumption is key to how numpy implements strided ufuncs > and broadcasting., > > On Wed, Mar 13, 2024 at 9:34 AM Dom Grigonis <dom.grigo...@gmail.com > <mailto:dom.grigo...@gmail.com>> wrote: > Thank you for this. > > I am just starting to think about these things, so I appreciate your patience. > > But isn’t it still true that all elements of an array are still of the same > size in memory? > > I am thinking along the lines of per-element dynamic memory management. Such > that if I had array [0, 1e10000], the first element would default to > reasonably small size in memory. > >> On 13 Mar 2024, at 16:29, Nathan <nathan.goldb...@gmail.com >> <mailto:nathan.goldb...@gmail.com>> wrote: >> >> It is possible to do this using the new DType system. >> >> Sebastian wrote a sketch for a DType backed by the GNU multiprecision float >> library: >> https://github.com/numpy/numpy-user-dtypes/tree/main/mpfdtype >> <https://github.com/numpy/numpy-user-dtypes/tree/main/mpfdtype> >> >> It adds a significant amount of complexity to store data outside the array >> buffer and introduces the possibility of use-after-free and dangling >> reference errors that are impossible if the array does not use embedded >> references, so that’s the main reason it hasn’t been done much. >> >> On Wed, Mar 13, 2024 at 8:17 AM Dom Grigonis <dom.grigo...@gmail.com >> <mailto:dom.grigo...@gmail.com>> wrote: >> Hi all, >> >> Say python’s builtin `int` type. It can be as large as memory allows. >> >> np.ndarray on the other hand is optimized for vectorization via strides, >> memory structure and many things that I probably don’t know. Well the point >> is that it is convenient and efficient to use for many things in comparison >> to python’s built-in list of integers. >> >> So, I am thinking whether something in between exists? (And obviously >> something more clever than np.array(dtype=object)) >> >> Probably something similar to `StringDType`, but for integers and floats. >> (It’s just my guess. I don’t know anything about `StringDType`, but just >> guessing it must be better than np.array(dtype=object) in combination with >> np.vectorize) >> >> Regards, >> dgpb >> >> _______________________________________________ >> NumPy-Discussion mailing list -- numpy-discussion@python.org >> <mailto:numpy-discussion@python.org> >> To unsubscribe send an email to numpy-discussion-le...@python.org >> <mailto:numpy-discussion-le...@python.org> >> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ >> <https://mail.python.org/mailman3/lists/numpy-discussion.python.org/> >> Member address: nathan12...@gmail.com <mailto:nathan12...@gmail.com> >> _______________________________________________ >> NumPy-Discussion mailing list -- numpy-discussion@python.org >> <mailto:numpy-discussion@python.org> >> To unsubscribe send an email to numpy-discussion-le...@python.org >> <mailto:numpy-discussion-le...@python.org> >> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ >> <https://mail.python.org/mailman3/lists/numpy-discussion.python.org/> >> Member address: dom.grigo...@gmail.com <mailto:dom.grigo...@gmail.com> > > _______________________________________________ > NumPy-Discussion mailing list -- numpy-discussion@python.org > <mailto:numpy-discussion@python.org> > To unsubscribe send an email to numpy-discussion-le...@python.org > <mailto:numpy-discussion-le...@python.org> > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > <https://mail.python.org/mailman3/lists/numpy-discussion.python.org/> > Member address: nathan12...@gmail.com <mailto:nathan12...@gmail.com> > _______________________________________________ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: dom.grigo...@gmail.com
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
