My array is growing in a manner of: array[slice] += values so for now will just clip values: res = np.add(array[slice], values, dtype=np.int64) array[slice] = res mask = res > MAX_UINT16 array[slice][mask] = MAX_UINT16
For this case, these large values do not have that much impact. And extra operation overhead is acceptable. --- And adding more involved project to my TODOs for the future. After all, it would be good to have an array, which (at preferably as minimal cost as possible) could handle anything you throw at it with near-optimal memory consumption and sensible precision handling, while keeping all the benefits of numpy. Time will tell if that is achievable. If anyone had any good ideas regarding this I am all ears. Much thanks to you all for information and ideas. dgpb > On 13 Mar 2024, at 21:00, Homeier, Derek <dhom...@gwdg.de> wrote: > > On 13 Mar 2024, at 6:01 PM, Dom Grigonis <dom.grigo...@gmail.com> wrote: >> >> So my array sizes in this case are 3e8. Thus, 32bit ints would be needed. So >> it is not a solution for this case. >> >> Nevertheless, such concept would still be worthwhile for cases where >> integers are say max 256bits (or unlimited), then even if memory addresses >> or offsets are 64bit. This would both: >> a) save memory if many of values in array are much smaller than 256bits >> b) provide a standard for dynamically unlimited size values > > In principle one could encode individual offsets in a smarter way, using just > the minimal number of bits required, > but again that would make random access impossible or very expensive – > probably more or less amounting to > what smart compression algorithms are already doing. > Another approach might be to to use the mask approach after all (or just flag > all you uint8 data valued 2**8 as > overflows) and store the correct (uint64 or whatever) values and their > indices in a second array. > May still not vectorise very efficiently with just numpy if your typical > operations are non-local. > > Derek > > _______________________________________________ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: dom.grigo...@gmail.com
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
