[Numpy-discussion] Re: reminder: put an upper bound on setuptools if you use numpy.distutils!

[Kevin Sheppard]

On Sun, Aug 21, 2022 at 6:36 PM Ralf Gommers  wrote:

> Hi all,
>
> After setuptools 65.0 was released a few days ago, all users of
> numpy.distutils had their builds broken. This is already fixed in
> setuptools 65.0.2 because the breakage was particularly bad. However, the
> next breakage may not be fixed anymore (and more breakages *are* expected).
> So this is a good time to remind you all that you should put an upper bound
> on the setuptools version you allow in the releases of your package - to
> the last version that is known to work with your package.
>
> Our official stance here is that setuptools versions >=60 are not
> supported - see the "deprecating numpy.distutils" thread:
> https://mail.python.org/archives/list/numpy-discussion@python.org/message/PMU4P4YRP2FZA2Z6Z6Z74ZFYD6PCRXQ5/.
> Newer versions may work for you, depending on what features you use. They
> don't for NumPy and for SciPy; both projects pin to 59.X to avoid problems.
>
> For the recent issue with setuptools 65.0.0, see
> https://github.com/numpy/numpy/issues/22135. We have also made the
> warnings about this topic in our docs more explicit, see
> https://github.com/numpy/numpy/pull/22154.
>
> Cheers,
> Ralf
>
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: kevin.k.shepp...@gmail.com


Is there a good way to pin <2 this when using oldest-supported-numpy?

-Kevin
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: reminder: put an upper bound on setuptools if you use numpy.distutils!

[Ralf Gommers]

On Thu, Aug 31, 2023 at 6:39 PM Kevin Sheppard 
wrote:

>
>
> On Sun, Aug 21, 2022 at 6:36 PM Ralf Gommers 
> wrote:
>
>> Hi all,
>>
>> After setuptools 65.0 was released a few days ago, all users of
>> numpy.distutils had their builds broken. This is already fixed in
>> setuptools 65.0.2 because the breakage was particularly bad. However, the
>> next breakage may not be fixed anymore (and more breakages *are* expected).
>> So this is a good time to remind you all that you should put an upper bound
>> on the setuptools version you allow in the releases of your package - to
>> the last version that is known to work with your package.
>>
>> Our official stance here is that setuptools versions >=60 are not
>> supported - see the "deprecating numpy.distutils" thread:
>> https://mail.python.org/archives/list/numpy-discussion@python.org/message/PMU4P4YRP2FZA2Z6Z6Z74ZFYD6PCRXQ5/.
>> Newer versions may work for you, depending on what features you use. They
>> don't for NumPy and for SciPy; both projects pin to 59.X to avoid problems.
>>
>> For the recent issue with setuptools 65.0.0, see
>> https://github.com/numpy/numpy/issues/22135. We have also made the
>> warnings about this topic in our docs more explicit, see
>> https://github.com/numpy/numpy/pull/22154.
>>
>> Cheers,
>> Ralf
>>
>>
>> ___
>> NumPy-Discussion mailing list -- numpy-discussion@python.org
>> To unsubscribe send an email to numpy-discussion-le...@python.org
>> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
>> Member address: kevin.k.shepp...@gmail.com
>
>
> Is there a good way to pin <2 this when using oldest-supported-numpy?
>

oldest-supported-numpy gives you build-time `==` constraints. So you
already have fixed (<2.0) versions there. What you are asking about is
related to runtime upper bounds I think, not build-time? The way to do that
is to use

  dependencies = ["numpy<2.0"]  # or: "numpy>=1.22.4,<2.0"

in your pyproject.toml.

This question is unrelated to an upper bound on setuptools, right?

Cheers,
Ralf
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: NEP 55 - Add a UTF-8 Variable-Width String DType to NumPy

[Stephan Hoyer]

On Wed, Aug 30, 2023 at 4:25 AM Ralf Gommers  wrote:

>
>
> On Tue, Aug 29, 2023 at 4:08 PM Nathan  wrote:
>
>> The NEP was merged in draft form, see below.
>>
>> https://numpy.org/neps/nep-0055-string_dtype.html
>>
>
> This is a really nice NEP, thanks Nathan! I see that questions and
> constructive feedback is still coming in on GitHub, but for now it seems
> like everyone is pretty happy with moving forward with implementing this
> new dtype in NumPy.
>
> Cheers,
> Rafl
>

To echo Ralf comments, thank you for this very well-written proposal! I
particularly appreciate the detailed consideration of how to handle
different models of missing values.

Overall, I am very excited about this work. A UTF8 dtype in NumPy is long
overdue, and will bring significant benefits to the entire scientific
Python ecosystem.
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Find location of slice in it's base

[Dom Grigonis]

Hi everyone,

I am working with shared arrays and their slices. And trying to subclass 
ndarray in such way so that they remap to memory on unpickling. I am using 
package SharedArray, which doesn’t micro-manage memory locations, but rather 
stores the whole map via shm using unique name. One issue I ran into is that 
when I pickle & unpickle slice of memory mapped array, I need to store the spec 
of the subset so that I can retrieve the same portion. 

The issue I am working on has been briefly discussed on stack: 
https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy-array
 
.
 Although I am not sure if it is the exact same one.

So, in short, is there a way to determine location of a slice in original 
array. Ideally, it would be indexing which works for any non-copy slice/subset 
possible.

Kind regards,
Dg

—Nothing ever dies, just enters the state of deferred evaluation—

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Find location of slice in it's base

[Aaron Meurer]

In principle this can be reconstructed from the strides, shape, and
base memory address (a.ctypes.data) of the view and base arrays.
However, not all views can be reconstructed using slices alone, for
example, views from reshape operations or stride tricks. I don't know
if it's possible to just construct a view directly given a base,
offset, shape, and strides, but I think ideally that's what you'd
want.

Aaron Meurer

On Thu, Aug 31, 2023 at 1:25 PM Dom Grigonis  wrote:
>
> Hi everyone,
>
> I am working with shared arrays and their slices. And trying to subclass 
> ndarray in such way so that they remap to memory on unpickling. I am using 
> package SharedArray, which doesn’t micro-manage memory locations, but rather 
> stores the whole map via shm using unique name. One issue I ran into is that 
> when I pickle & unpickle slice of memory mapped array, I need to store the 
> spec of the subset so that I can retrieve the same portion.
>
> The issue I am working on has been briefly discussed on stack: 
> https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy-array.
>  Although I am not sure if it is the exact same one.
>
> So, in short, is there a way to determine location of a slice in original 
> array. Ideally, it would be indexing which works for any non-copy 
> slice/subset possible.
>
> Kind regards,
> Dg
>
> —Nothing ever dies, just enters the state of deferred evaluation—
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: asmeu...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Find location of slice in it's base

[Dom Grigonis]

I am looking for something that would work for stride tricks too. Essentially, 
any possible view.

> On 31 Aug 2023, at 23:03, Aaron Meurer  wrote:
> 
> In principle this can be reconstructed from the strides, shape, and
> base memory address (a.ctypes.data) of the view and base arrays.
> However, not all views can be reconstructed using slices alone, for
> example, views from reshape operations or stride tricks. I don't know
> if it's possible to just construct a view directly given a base,
> offset, shape, and strides, but I think ideally that's what you'd
> want.
> 
> Aaron Meurer
> 
> On Thu, Aug 31, 2023 at 1:25 PM Dom Grigonis  wrote:
>> 
>> Hi everyone,
>> 
>> I am working with shared arrays and their slices. And trying to subclass 
>> ndarray in such way so that they remap to memory on unpickling. I am using 
>> package SharedArray, which doesn’t micro-manage memory locations, but rather 
>> stores the whole map via shm using unique name. One issue I ran into is that 
>> when I pickle & unpickle slice of memory mapped array, I need to store the 
>> spec of the subset so that I can retrieve the same portion.
>> 
>> The issue I am working on has been briefly discussed on stack: 
>> https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy-array.
>>  Although I am not sure if it is the exact same one.
>> 
>> So, in short, is there a way to determine location of a slice in original 
>> array. Ideally, it would be indexing which works for any non-copy 
>> slice/subset possible.
>> 
>> Kind regards,
>> Dg
>> 
>> —Nothing ever dies, just enters the state of deferred evaluation—
>> 
>> ___
>> NumPy-Discussion mailing list -- numpy-discussion@python.org
>> To unsubscribe send an email to numpy-discussion-le...@python.org
>> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
>> Member address: asmeu...@gmail.com
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: dom.grigo...@gmail.com

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Find location of slice in it's base

[Robert Kern]

On Thu, Aug 31, 2023 at 3:25 PM Dom Grigonis  wrote:

> Hi everyone,
>
> I am working with shared arrays and their slices. And trying to subclass
> ndarray in such way so that they remap to memory on unpickling. I am using
> package SharedArray, which doesn’t micro-manage memory locations, but
> rather stores the whole map via shm using unique name. One issue I ran into
> is that when I pickle & unpickle slice of memory mapped array, I need to
> store the spec of the subset so that I can retrieve the same portion.
>
> The issue I am working on has been briefly discussed on stack:
> https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy-array.
> Although I am not sure if it is the exact same one.
>
> So, in short, is there a way to determine location of a slice in original
> array. Ideally, it would be indexing which works for any non-copy
> slice/subset possible.
>

If you chase the `.base` chain all the way down, you get to a
`SharedArray.map_owner` object with its `.addr` attribute giving the memory
address that's assigned to it in the current process. This will be the same
address that's in the `'data'` key of that first `ndarray` returned from
`SharedArray.create()` that you are making your views from. In your view
`ndarray` (which may be a view of views, with slices, transposes, reshapes,
and arbitrary `stride_tricks` manipulations in between). If you store the
difference between the view `ndarray`'s `'data'` address from the
`map_owner.addr` address, I think that's all you need to recreate the array
in a different process which gets assigned a different memory address for
the same `shm` name. Just add that offset to the `map_owner.addr` and
restore the rest of the information in `__array_interface__`, and I think
you should be good to go. I don't think you'll need to infer or represent
the precise path of Python-level operations (slices, transposes, reshapes,
etc.) to which it got to that point.

-- 
Robert Kern
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Find location of slice in it's base

[Dom Grigonis]

Thank you, sounds just what I need. Will work on this in due time.

> On 1 Sep 2023, at 00:38, Robert Kern  wrote:
> 
> On Thu, Aug 31, 2023 at 3:25 PM Dom Grigonis  > wrote:
> Hi everyone,
> 
> I am working with shared arrays and their slices. And trying to subclass 
> ndarray in such way so that they remap to memory on unpickling. I am using 
> package SharedArray, which doesn’t micro-manage memory locations, but rather 
> stores the whole map via shm using unique name. One issue I ran into is that 
> when I pickle & unpickle slice of memory mapped array, I need to store the 
> spec of the subset so that I can retrieve the same portion. 
> 
> The issue I am working on has been briefly discussed on stack: 
> https://stackoverflow.com/questions/12421770/find-location-of-slice-in-numpy-array
>  
> .
>  Although I am not sure if it is the exact same one.
> 
> So, in short, is there a way to determine location of a slice in original 
> array. Ideally, it would be indexing which works for any non-copy 
> slice/subset possible.
> 
> If you chase the `.base` chain all the way down, you get to a 
> `SharedArray.map_owner` object with its `.addr` attribute giving the memory 
> address that's assigned to it in the current process. This will be the same 
> address that's in the `'data'` key of that first `ndarray` returned from 
> `SharedArray.create()` that you are making your views from. In your view 
> `ndarray` (which may be a view of views, with slices, transposes, reshapes, 
> and arbitrary `stride_tricks` manipulations in between). If you store the 
> difference between the view `ndarray`'s `'data'` address from the 
> `map_owner.addr` address, I think that's all you need to recreate the array 
> in a different process which gets assigned a different memory address for the 
> same `shm` name. Just add that offset to the `map_owner.addr` and restore the 
> rest of the information in `__array_interface__`, and I think you should be 
> good to go. I don't think you'll need to infer or represent the precise path 
> of Python-level operations (slices, transposes, reshapes, etc.) to which it 
> got to that point.
> 
> -- 
> Robert Kern
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org 
> 
> To unsubscribe send an email to numpy-discussion-le...@python.org 
> 
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ 
> 
> Member address: dom.grigo...@gmail.com 
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com