[Numpy-discussion] Rules for argument parsing/forwarding in __array_function__ and __array_ufunc__

2020-12-02 Thread Sebastian Berg 
Hi all,

I am curious about the correct argument for "normalizing" and
forwarding arguments in `__array_function__` and `__array_ufunc__`. I
have listed the three rules as I understand how the "right" way is
below.

Mainly this is just to write down the rules that I think we should aim
for in case it comes up.

I admit, I do have "hidden agendas" where it may come up:

* pytorch breaks rule 2 in their copy of `__array_function__`, because
  it allows to easily optimize away some overheads.
* `__array_ufunc__` breaks rule 1 (it allows too much) and I think we
  should be OK to change that [1].
* A few `__array_function__`s break rule 3. That is completely
  harmless, but it might be nice to just be clear whether we consider
  it technically wrong. [2]


Rules
-

1. If an argument is invalid in NumPy it is considered and error.
   For example:

   np.log(arr, my_weird_argument=True)

   is always an error even if the `__array_function__` implementation
   of `arr` would support it.
   NEP 18 explicitly says that allowing forwarding could be done, but
   will not be done at this time.

2. Arguments must only be forwarded if they are passed in:

   np.mean(cupy_array)

   ends up as `cupy.mean(cupy_array)` and not:

   cupy.mean(cupy_array, axis=None, dtype=None, out=None,
 keepdims=False, where=True)

   meaning that CuPy does not need to implement all of those kwargs and
   NumPy can add new ones without breaking anyones code.

3. NumPy should not check the *validity* of the arguments. For example:
   `np.add.reduce(xarray, axis="long")` should probably work in xarray.
   (`xarray.DataArray` does not actually implement the above.)
   But a string cannot be used as an axis in NumPy.


Cheers,

Sebastian



[1] I think `dask` breaks this rule by using an `output_dtypes`
keyword. I would just consider this a valid exception and keep allowing
it.  In fact, `output_dtypes` may very well be a useful argument for
NumPy itself. `dtype` looks like it serves that purpose, but it does
not have quite the same meaning.
This has been discussed also here: 
https://github.com/numpy/numpy/issues/8892

[2] This is done for performance reasons, although it is entirely
avoidable. However, avoiding it might just add a bunch of annoying code
unless part of a larger maintenance effort.


signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] problem with numpy 1.19.4 install via pip on Win 10

2020-12-02 Thread Alan G. Isaac 

numpy 1.19.3 installs fine.
numpy 1.19.4 appears to install but does not work.
(Details below. The supplied tinyurl appears relevant.)
Alan Isaac

PS test> python38 -m pip install -U numpy
Collecting numpy
  Using cached numpy-1.19.4-cp38-cp38-win_amd64.whl (13.0 MB)
Installing collected packages: numpy
Successfully installed numpy-1.19.4
PS test> python38
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit 
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
 ** On entry to DGEBAL parameter number  3 had an illegal value
 ** On entry to DGEHRD  parameter number  2 had an illegal value
 ** On entry to DORGHR DORGQR parameter number  2 had an illegal value
 ** On entry to DHSEQR parameter number  4 had an illegal value
Traceback (most recent call last):
  File "", line 1, in 
  File "C:\Program Files\Python38\lib\site-packages\numpy\__init__.py", line 305, in 

_win_os_check()
  File "C:\Program Files\Python38\lib\site-packages\numpy\__init__.py", line 
302, in _win_os_check
raise RuntimeError(msg.format(__file__)) from None
RuntimeError: The current Numpy installation ('C:\\Program Files\\Python38\\lib\\site-packages\\numpy\\__init__.py') fails to pass a sanity check due to a bug 
in the windows runtime. See this issue for more information: https://tinyurl.com/y3dm3h86

>>>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] problem with numpy 1.19.4 install via pip on Win 10

2020-12-02 Thread Ilhan Polat 
Yes this is known and we are waiting MS to roll out a solution for this.
Here are more details
https://developercommunity2.visualstudio.com/t/fmod-after-an-update-to-windows-2004-is-causing-a/1207405

On Thu, Dec 3, 2020 at 12:57 AM Alan G. Isaac  wrote:

> numpy 1.19.3 installs fine.
> numpy 1.19.4 appears to install but does not work.
> (Details below. The supplied tinyurl appears relevant.)
> Alan Isaac
>
> PS test> python38 -m pip install -U numpy
> Collecting numpy
>Using cached numpy-1.19.4-cp38-cp38-win_amd64.whl (13.0 MB)
> Installing collected packages: numpy
> Successfully installed numpy-1.19.4
> PS test> python38
> Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64
> bit (AMD64)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import numpy
>   ** On entry to DGEBAL parameter number  3 had an illegal value
>   ** On entry to DGEHRD  parameter number  2 had an illegal value
>   ** On entry to DORGHR DORGQR parameter number  2 had an illegal value
>   ** On entry to DHSEQR parameter number  4 had an illegal value
> Traceback (most recent call last):
>File "", line 1, in 
>File "C:\Program Files\Python38\lib\site-packages\numpy\__init__.py",
> line 305, in 
>  _win_os_check()
>File "C:\Program Files\Python38\lib\site-packages\numpy\__init__.py",
> line 302, in _win_os_check
>  raise RuntimeError(msg.format(__file__)) from None
> RuntimeError: The current Numpy installation ('C:\\Program
> Files\\Python38\\lib\\site-packages\\numpy\\__init__.py') fails to pass a
> sanity check due to a bug
> in the windows runtime. See this issue for more information:
> https://tinyurl.com/y3dm3h86
>  >>>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] problem with numpy 1.19.4 install via pip on Win 10

2020-12-02 Thread Sebastian Berg 
On Thu, 2020-12-03 at 01:13 +0100, Ilhan Polat wrote:
> Yes this is known and we are waiting MS to roll out a solution for
> this.
> Here are more details
> https://developercommunity2.visualstudio.com/t/fmod-after-an-update-to-windows-2004-is-causing-a/1207405


I think one workaround was `pip install numpy==1.19.3`, which uses a
different OpenBLAS that avoids the worst of the windows bug.

- Sebastian


> 
> On Thu, Dec 3, 2020 at 12:57 AM Alan G. Isaac 
> wrote:
> 
> > numpy 1.19.3 installs fine.
> > numpy 1.19.4 appears to install but does not work.
> > (Details below. The supplied tinyurl appears relevant.)
> > Alan Isaac
> > 
> > PS test> python38 -m pip install -U numpy
> > Collecting numpy
> >Using cached numpy-1.19.4-cp38-cp38-win_amd64.whl (13.0 MB)
> > Installing collected packages: numpy
> > Successfully installed numpy-1.19.4
> > PS test> python38
> > Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC
> > v.1916 64
> > bit (AMD64)] on win32
> > Type "help", "copyright", "credits" or "license" for more
> > information.
> >  >>> import numpy
> >   ** On entry to DGEBAL parameter number  3 had an illegal value
> >   ** On entry to DGEHRD  parameter number  2 had an illegal value
> >   ** On entry to DORGHR DORGQR parameter number  2 had an illegal
> > value
> >   ** On entry to DHSEQR parameter number  4 had an illegal value
> > Traceback (most recent call last):
> >File "", line 1, in 
> >File "C:\Program Files\Python38\lib\site-
> > packages\numpy\__init__.py",
> > line 305, in 
> >  _win_os_check()
> >File "C:\Program Files\Python38\lib\site-
> > packages\numpy\__init__.py",
> > line 302, in _win_os_check
> >  raise RuntimeError(msg.format(__file__)) from None
> > RuntimeError: The current Numpy installation ('C:\\Program
> > Files\\Python38\\lib\\site-packages\\numpy\\__init__.py') fails to
> > pass a
> > sanity check due to a bug
> > in the windows runtime. See this issue for more information:
> > https://tinyurl.com/y3dm3h86
> >  >>>
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> > 
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion



signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Precompute shape of ufunc output.

2020-12-02 Thread drtodd13 
If I want to provide the "out" kwarg to, for example, a reduce ufunc then I
need to know the shape of the output given the other set of inputs.  Is
there a utility function to take these arguments and just compute what the
shape of the output is going to be without actually computing the result? 
Would be nice if there were a standard kwarg for many functions that just
tells you the shape of the output without actually computing it but I don't
think that exists.  Any ideas appreciated.  Thanks.



--
Sent from: http://numpy-discussion.10968.n7.nabble.com/
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Rules for argument parsing/forwarding in __array_function__ and __array_ufunc__

2020-12-02 Thread Stefan van der Walt 
Hi Sebastian,

Looking at these three rules, they all seem to stem from one simple question: 
do we desire for a single code snippet to be runnable on multiple array 
implementations?

On Wed, Dec 2, 2020, at 15:34, Sebastian Berg wrote:
> 1. If an argument is invalid in NumPy it is considered and error.
>For example:
> 
>np.log(arr, my_weird_argument=True)
> 
>is always an error even if the `__array_function__` implementation
>of `arr` would support it.
>NEP 18 explicitly says that allowing forwarding could be done, but
>will not be done at this time.

Relaxing this rule will mean that code working for one array implementation 
(which has this keyword) may not work for another.

> 2. Arguments must only be forwarded if they are passed in:
> 
>np.mean(cupy_array)
> 
>ends up as `cupy.mean(cupy_array)` and not:
> 
>cupy.mean(cupy_array, axis=None, dtype=None, out=None,
>  keepdims=False, where=True)
> 
>meaning that CuPy does not need to implement all of those kwargs and
>NumPy can add new ones without breaking anyones code.

This may ultimately make it harder for array implementors (they will only see 
errors once someone tries to pass in an argument that they forgot to 
implement).  Perhaps better to pass all so they know what they're dealing with?

> 3. NumPy should not check the *validity* of the arguments. For example:
>`np.add.reduce(xarray, axis="long")` should probably work in xarray.
>(`xarray.DataArray` does not actually implement the above.)
>But a string cannot be used as an axis in NumPy.

Getting back to the original question: if this code is to be run on multiple 
implementations, we should ensure that no strange values pass through.

Personally, I like the idea of a single API that works on multiple backends.  
As such, I would 1) not pass through unknown arguments, 2) always pass through 
all arguments, and 3) validate inputs to each call.

Best regards,
Stéfan
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion