[Numpy-discussion] Re: Proposing a flattening functionality for deeply nested lists in NumPy

Hi Mark, I think this has already been implemented. For Iterables (list included), there is `more_itertools.collapse`, which you can use to achieve examples in your code as `list(more_itertools.collapse(list))`. See: https://more-itertools.readthedocs.io/en/stable/api.html#more_itertools.colla

[Numpy-discussion] Re: Applying a function on local environments

Hello Thomas, It seems that what you are after is application of custom kernels. Similar existing things to look at: * For linear kernels, have a look at `np.correlate` function. * For custom kernels, see `numba` stencils. See: https://numba.pydata.org/numba-doc/latest/user/stencil.html One rea

[Numpy-discussion] Re: A better syntax for using ufunc.at?

It would be nicer, yes. However, at least from my experience, it does not seem to be used often enough to go extra mile for the sake of convenience alone. Regards, dg > On 24 Jul 2024, at 13:49, Oras P. wrote: > > I am aware that to do unbuffered addition operation, I can use `np.add.at` > l

[Numpy-discussion] Re: feature request: N-D Gaussian function (not random distribution)

For statistics functions there is `scipy` package. If you are referring to pdf of n-dimensional gaussian distribution, `scipy.stats.multivariate_normal.pdf` should do the trick. If you are referring to something else, then a bit of clarification would be helpful. Regards, dg > On 20 Jul 2024,

[Numpy-discussion] Re: JSON format for multi-dimensional data

Looks good, I am currently using very low complexity data structures. Could you post some examples how would the data look like for simple stuff like 2d array with 2 coordinate arrays without any extra attributes? Regards, dgpb > On 15 Apr 2024, at 15:38, phili...@loco-labs.io wrote: > > Hell

[Numpy-discussion] Re: NumPy-Financial: RFC: Dropping decimal.Decimal support

Hi Kai, I am not using numpy financial, but from what I know: Given numpy 2.0 has a support for MPFD types I think it is very reasonable to drop Decimal support and remove unnecessary complexity. Regards, dgpb > On 30 Mar 2024, at 02:20, kaistri...@gmail.com wrote: > > Hi all, > > I'm curren

[Numpy-discussion] Re: Polyfit error in displacement

Hello Luca, I am a bit confused by the output of VB script. Equation is: y = f(x), where x is in the order of 0-2K y is in the order of 5-10K The output of fitted polynomial is in y-space, thus I would expect fitted values to be similar to those of Y. Now, sc values are very small and alternat

[Numpy-discussion] Re: Arrays of variable itemsize

> current version of the Python interface to Blosc, so I'd expect it to cover > the same use-cases.) > > -- Jim > > > > > > On Wed, Mar 13, 2024 at 4:45 PM Dom Grigonis <mailto:dom.grigo...@gmail.com>> wrote: > My array is growing in a manner of: &

[Numpy-discussion] Re: Arrays of variable itemsize

ion handling, while keeping all the benefits of numpy. Time will tell if that is achievable. If anyone had any good ideas regarding this I am all ears. Much thanks to you all for information and ideas. dgpb > On 13 Mar 2024, at 21:00, Homeier, Derek wrote: > > On 13 Mar 2024, at 6:01 PM,

[Numpy-discussion] Re: Arrays of variable itemsize

:matti.pi...@gmail.com>> wrote: > I am not sure what kind of a scheme would support various-sized native > ints. Any scheme that puts pointers in the array is going to be worse: > the pointers will be 64-bit. You could store offsets to data, but then > you would need to store b

[Numpy-discussion] Re: Arrays of variable itemsize

r storage. > What shape are your arrays, that would be the minimum size of the offsets? > > Matti > > > On 13/3/24 18:15, Dom Grigonis wrote: >> By the way, I think I am referring to integer arrays. (Or integer part of >> floats.) >> >> I don’t t

[Numpy-discussion] Re: Arrays of variable itemsize

> the heap, which can be a dynamic memory allocation. > > There's no way in NumPy to support variable-sized array elements in the array > buffer, since that assumption is key to how numpy implements strided ufuncs > and broadcasting., > > On Wed, Mar 13, 2024 at 9

[Numpy-discussion] Re: Arrays of variable itemsize

> reference errors that are impossible if the array does not use embedded > references, so that’s the main reason it hasn’t been done much. > > On Wed, Mar 13, 2024 at 8:17 AM Dom Grigonis <mailto:dom.grigo...@gmail.com>> wrote: > Hi all, > > Say python’s builtin `i

[Numpy-discussion] Arrays of variable itemsize

Hi all, Say python’s builtin `int` type. It can be as large as memory allows. np.ndarray on the other hand is optimized for vectorization via strides, memory structure and many things that I probably don’t know. Well the point is that it is convenient and efficient to use for many things in com

[Numpy-discussion] Re: Automatic Clipping of array to upper / lower bounds of dtype

to robustly achieve this it would be good to know. Regards, dg > On 10 Mar 2024, at 18:43, Ralf Gommers wrote: > > > > On Sun, Mar 10, 2024 at 9:14 AM Dom Grigonis <mailto:dom.grigo...@gmail.com>> wrote: > Much thanks! > > Another related question while I a

[Numpy-discussion] Re: Automatic Clipping of array to upper / lower bounds of dtype

10 Mar 2024, at 09:59, Ralf Gommers wrote: > > > > On Sat, Mar 9, 2024 at 11:23 PM Dom Grigonis <mailto:dom.grigo...@gmail.com>> wrote: > Hello, > > Can't find answer to this anywhere. > > What I would like is to automatically clip the values if t

[Numpy-discussion] Automatic Clipping of array to upper / lower bounds of dtype

Hello, Can't find answer to this anywhere. What I would like is to automatically clip the values if they breach the bounds. I have done a simple clipping, and overwritten __iadd__, __isub__, __setitem__, … But I am wandering if there is a specified way to do this. Or maybe at least a centrali

[Numpy-discussion] ENH: tobytes / frombuffer extension

I suggest adding flag - `full: bool` by default it is False. And if it is True, then it runs: https://github.com/numpy/numpy/blob/main/doc/neps/nep-0001-npy-format.rst The whole point is that it should be implemented in C. Currently: ``` def save(a): m = io.BytesIO() np.save(m, a) #

[Numpy-discussion] Re: JSON format for multi-dimensional data

Could be of interest to scipp. They already put work to link to https://www.nexusformat.org. Maybe they would be interested in JData as well. — Also, what could be interesting is "format independent standard”. So that there is a general standard of data structures, which is adapted by differen

[Numpy-discussion] Re: JSON format for multi-dimensional data

I sure like the idea of this. I agree that this should be external to numpy. At least until it becomes a standard in a sense that json itself is. And that n-dimensional array should ideally be extended to indexed structures with named dimensions and attributes. To accommodate a superset of: xar

[Numpy-discussion] Re: Converting array to string

od to know that if it becomes a bottleneck I can substitute array value with its bytes representation with little to none extra work. DG > On 25 Feb 2024, at 21:02, Robert Kern wrote: > > On Sun, Feb 25, 2024 at 1:52 PM Dom Grigonis <mailto:dom.grigo...@gmail.com>> wrot

[Numpy-discussion] Re: Converting array to string

(after a fair bit of research and attempts) that it can not be done so I can eliminate this possibility as feasible and concentrate on other options. Regards, DG > On 25 Feb 2024, at 20:30, Robert Kern wrote: > > On Sat, Feb 24, 2024 at 7:17 PM Dom Grigonis <mailto:dom.grigo.

[Numpy-discussion] Converting array to string

Hello, I am seeking a bit of help. I need a fast way to transfer numpy arrays in json format. Thus, converting array to list is not an option and I need something similar to: a = np.ones(1000) %timeit a.tobytes() 17.6 ms This is quick, fast and portable. In other words I am very happy with t

[Numpy-discussion] Re: ENH: Introducing a pipe Method for Numpy arrays

Good to know it is not only on my PC. I have done a fair bit of work trying to find more efficient sum. The only faster option that I have found was PyTorch. (although thinking about it now maybe it’s because it was using MKL, don’t remember) MKL is faster, but I use OpenBLAS. Scipp library is

[Numpy-discussion] Re: ENH: Introducing a pipe Method for Numpy arrays

Thanks for this, every little helps. One more thing to mention on this topic. From a certain size dot product becomes faster than sum (due to parallelisation I guess?). E.g. def dotsum(arr): a = arr.reshape(1000, 100) return a.dot(np.ones(100)).sum() a = np.ones(10) In [45]: %time

[Numpy-discussion] Re: ENH: Introducing a pipe Method for Numpy arrays

Just to clarify, I am not the one who suggested pipes. :) Read the issue. My 2 cents: From my experience, calling methods is generally faster than functions. I figure it is due to having less overhead figuring out the input. Maybe it is not significant for large data, but it does make a differe

[Numpy-discussion] Re: ENH: Introducing a pipe Method for Numpy arrays

What were your conclusions after experimenting with chained ufuncs? If the speed is comparable to numexpr, wouldn’t it be `nicer` to have non-string input format? It would feel a bit less like a black-box. Regards, DG > On 15 Feb 2024, at 22:52, Marten van Kerkwijk wrote: > > Hi Oyibo, > >>

[Numpy-discussion] Re: API: make numpy.lib._arraysetops.intersect1d work on multiple arrays #25688

> arrays, using `return_indices=True`. > > All the Best > Stephan > > Am 02.02.24 um 17:36 schrieb Dom Grigonis: >> Also, I don’t know if this could be of value, but my use case for this is to >> find overlaps, then split arrays into overlapping and non-overlapping

[Numpy-discussion] Re: API: make numpy.lib._arraysetops.intersect1d work on multiple arrays #25688

> Seems reasonable. I don't know if it is faster, but NumPy is all about > vectorization. Surely speed needs to fit into equation somehow? Another point to consider. Is function `in1d` able to achieve everything that `intersect1d` does? For 2 arrays of length 1600. def intersect1d_via_in1d(x,

[Numpy-discussion] Re: API: make numpy.lib._arraysetops.intersect1d work on multiple arrays #25688

Also, I don’t know if this could be of value, but my use case for this is to find overlaps, then split arrays into overlapping and non-overlapping segments. Thus, it might be useful for `return_indices=True` to return indices of all instances, not only the first. Also, in my case I need both ov

[Numpy-discussion] Re: API: make numpy.lib._arraysetops.intersect1d work on multiple arrays #25688

Just curious, how much faster is it compared to currently recommended `reduce` approach? DG > On 2 Feb 2024, at 17:31, Marten van Kerkwijk wrote: > >> For my own work, I required the intersect1d function to work on multiple >> arrays while returning the indices (using `return_indizes=True`).

[Numpy-discussion] Re: Change definition of complex sign (and use it in copysign)

I think this suggestion regarding sign is solid. From both theoretical and practical points of view. And agree with all of Aaron’s points as well. Regards, DG > On 4 Jan 2024, at 22:58, Robert Kern wrote: > > On Wed, Jan 3, 2024 at 4:09 PM Aaron Meurer > wrote: > s

[Numpy-discussion] Re: Change definition of complex sign (and use it in copysign)

To me this sounds like a reasonable change. It does seem like there is a return value which is more sensible than alternatives. And the fact that sympy is already doing that indicates that same conclusion was reached more than once. I am not dealing much with complex numbers at the moment, but

[Numpy-discussion] Re: Seeking feedback: design doc for `namedarray`, a lightweight array data structure with named dimensions

b.com/scikit-learn/enhancement_proposals/pull/25> > > Might be useful to y'all. > > On Fri, Oct 20, 2023 at 8:49 AM Dom Grigonis <mailto:dom.grigo...@gmail.com>> wrote: > I would make use of it if it was also supporting pure-numpy indices too. > Pure-numpy n-dim array with indices

[Numpy-discussion] Re: How to sample unique vectors

Is numba solution an option for you? > On 17 Nov 2023, at 20:49, Stefan van der Walt via NumPy-Discussion > wrote: > > Hi all, > > I am trying to sample k N-dimensional vectors from a uniform distribution > without replacement. > It seems like this should be straightforward, but I can't seem

[Numpy-discussion] Re: Function that searches arrays for the first element that satisfies a condition

lambda: npi.first_above(arr2, 5), # 0.00021 ]).print(5, idx=1, t=True) # {'reps': 5, 'n': 10} > On 1 Nov 2023, at 16:19, Dom Grigonis wrote: > > > Your comparisons do not paint correct picture. > a) Most of time

[Numpy-discussion] Re: Function that searches arrays for the first element that satisfies a condition

In [30]: arr2 = np.empty_like(arr, dtype=bool) > > In [32]: %timeit np.greater(arr, 5, out=arr2) > 13.9 ms ± 69.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) > > But it's certainly much better than pure Python! And it's not a huge cost for >

[Numpy-discussion] Re: Function that searches arrays for the first element that satisfies a condition

yle 'black'. It should be easy >> to add functions like 'first_below' if necessary. >> >> A more detailed description of these functions can be found here >> <https://betterprogramming.pub/the-numpy-illustrated-library-7531a7c43ffb?sk=8dd60bfafd6d492

[Numpy-discussion] Re: Function that searches arrays for the first element that satisfies a condition

ar superior, not even evaluating the array of exponentials > (which my example clearly still does--and in the use cases I've had for such > a function, I can't predict the array elements like this--they come from > loaded data, the output of a simulation, etc., and are all alr

[Numpy-discussion] Re: Function that searches arrays for the first element that satisfies a condition

nstead more of a convenience. More general discussion I opened > is in https://github.com/data-apis/array-api/issues/675 > <https://github.com/data-apis/array-api/issues/675> > > > > > > On Thu, Oct 26, 2023 at 2:52 PM Dom Grigonis <mailto:dom.grigo...@gmail.com

[Numpy-discussion] Re: Function that searches arrays for the first element that satisfies a condition

Could you please give a concise example? I know you have provided one, but it is engrained deep in verbose text and has some typos in it, which makes hard to understand exactly what inputs should result in what output. Regards, DG > On 25 Oct 2023, at 22:59, rosko37 wrote: > > I know this que

[Numpy-discussion] Re: Seeking feedback: design doc for `namedarray`, a lightweight array data structure with named dimensions

I would make use of it if it was also supporting pure-numpy indices too. Pure-numpy n-dim array with indices is what I was after for a while now. The reason is exactly that - to shed heavy dependencies as pandas and have performance of pure numpy. Regards, DG > On 20 Oct 2023, at 00:51, Anders

[Numpy-discussion] Preserving subclass on concatenation

Hi everyone, Quick question. Is there a way to concatenate arrays while preserving subclasses or is it left for custom implementation due to ambiguities of this operation? Regards, DG ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To

[Numpy-discussion] Re: Proposal to accept NEP 52 (Python API cleanup for 2.0)

Hello, I have a couple of questions: 1. What is equivalent of np.byte_bounds? I have recently started using this. 2. Why are you removing business day functionality? Are there faster methods in python space for it? As far as I remember, for performance critical applications I have always resorte

[Numpy-discussion] Re: Find location of slice in it's base

Thank you, sounds just what I need. Will work on this in due time. > On 1 Sep 2023, at 00:38, Robert Kern wrote: > > On Thu, Aug 31, 2023 at 3:25 PM Dom Grigonis <mailto:dom.grigo...@gmail.com>> wrote: > Hi everyone, > > I am working with shared arrays and their sli

[Numpy-discussion] Re: Find location of slice in it's base

;s what you'd > want. > > Aaron Meurer > > On Thu, Aug 31, 2023 at 1:25 PM Dom Grigonis wrote: >> >> Hi everyone, >> >> I am working with shared arrays and their slices. And trying to subclass >> ndarray in such way so that they remap to memory on un

[Numpy-discussion] Find location of slice in it's base

Hi everyone, I am working with shared arrays and their slices. And trying to subclass ndarray in such way so that they remap to memory on unpickling. I am using package SharedArray, which doesn’t micro-manage memory locations, but rather stores the whole map via shm using unique name. One issue

[Numpy-discussion] Re: Add to NumPy a function to compute cumulative sums from 0.

22 Aug 2023, at 17:36, john.daw...@camlingroup.com wrote: > > Dom Grigonis wrote: >> 1. Dimension length stays constant, while cumusm0 extends length to n+1, >> then np.diff, truncates it back. This adds extra complexity, while things >> are very convenient to work with

[Numpy-discussion] Re: Add to NumPy a function to compute cumulative sums from 0.

Unfortunately, I don’t have a good answer. For now, I can only tell you what I think might benefit from improvement. 1. Verbosity. I appreciate that bracket syntax such as one in julia or matlab `[A B C ...]` is not possible, so functional is the only option. E.g. julia has functions named ‘cat

[Numpy-discussion] Re: Add to NumPy a function to compute cumulative sums from 0.

With this I agree, this sounds like a more radical (in a good way) solution. > So I think this is a feature request of "prepend", "append" in a convenient > fashion not to ufuncs but to ndarray. Because concatenation is just pain in > NumPy and ubiquitous operation all around. Hence probably we

[Numpy-discussion] Re: Add to NumPy a function to compute cumulative sums from 0.

> On 14 Aug 2023, at 15:22, john.daw...@camlingroup.com wrote: > >> From my point of view, such function is a bit of a corner-case to be added >> to numpy. And it doesn’t justify it’s naming anymore. It is not one >> operation anymore. It is a cumsum and prepending 0. And it is very difficult

[Numpy-discussion] Re: Add to NumPy a function to compute cumulative sums from 0.

From my point of view, such function is a bit of a corner-case to be added to numpy. And it doesn’t justify it’s naming anymore. It is not one operation anymore. It is a cumsum and prepending 0. And it is very difficult to argue why prepending 0 to cumsum is a part of cumsum. What I would rath