[Numpy-discussion] NumPy Development Meeting Wednesday - Triage Focus

[Sebastian Berg]

Hi all,

Our bi-weekly triage-focused NumPy development meeting is Wednesday,
May 4th at 16:00 UTC (9:00am Pacific Time).
Everyone is invited to join in and edit the work-in-progress meeting
topics and notes:
https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg

I encourage everyone to notify us of issues or PRs that you feel should
be prioritized, discussed, or reviewed.

A calendar to can be found at: https://scientific-python.org/calendars/
please make sure to not import the static `ics` but the url.

Best regards

Sebastian

___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: NumPy Development Meeting Wednesday - Triage Focus

[Renato Fabbri]

On Tue, May 3, 2022 at 1:48 PM Sebastian Berg 
wrote:

> Hi all,
>
> Our bi-weekly triage-focused NumPy development meeting is Wednesday,
> May 4th at 16:00 UTC (9:00am Pacific Time).
> Everyone is invited to join in and edit the work-in-progress meeting
> topics and notes:
> https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg


--> 502 Bad Gateway


>
>
> I encourage everyone to notify us of issues or PRs that you feel should
> be prioritized, discussed, or reviewed.
>
> A calendar to can be found at: https://scientific-python.org/calendars/
> please make sure to not import the static `ics` but the url.
>
> Best regards
>
> Sebastian
>
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: renato.fab...@gmail.com
>


-- 
Renato Fabbri
--
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Numpy-related jobs, grants, etc

[Renato Fabbri]

This is not only for me, but a question for the community in general.
Do we have an open knowledge base of budget (interested companies or
individuals, public funding)
for developing numpy-related packages?

Explaining a bit with a specific case...
In the last few years, I have many times thought about the possibility of
working with Numpy:
finding payment for developing libraries using Numpy or even Numpy
internals.
Especially, I have written a music python library (called 'music' on PyPI).
It holds many useful things, equal-loudness contours, musical notes,
localization, HRTF, LUT, singing, glissandi, tremolos & vibratos with
transitions, etc etc.
I believe it is something well done, which involved scientific research and
over a decade of experience with the algorithms, but it still lacks a
better organization of the functionalities, better documentation, and
better maintenance.
To reach that point, one would probably have to be paid to take the
necessary time.

Beyond that, I know it is useful for making art/music, teaching, learning
music, learning music-related signal processing and programming, etc.

Maybe the most important point: this is probably the case with other
libraries/projects.

(((
On the other side, we as a company (here at Modena/Italy) may look into
(public, private, own) funding
to develop Numpy (internals, libraries).
We would also enjoy developing Numpy-related libraries (internals?) if
someone needs or wants them.
)))

Should this be posted in another list/forum?
Any thoughts?
R.

-- 
Renato Fabbri
--
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Types in pure Python sources

[Ralf Gommers]

On Sun, May 1, 2022 at 9:08 PM Sergei Lebedev 
wrote:

> > I think that's on purpose, because the type annotations are quite
> complex.
> > For reasons of correctness/completeleness, they use protocols, mixins,
> and
> > overloads. Inside pure Python code, that would be harder to read and
> > maintain.
>
> I agree that NumPy type annotations are quite complex. However, having
> them alongside the implementation might actually make maintenance easier,
> because there will be no need to update type stubs separately from the
> implementation.
>

I think I disagree, but let's see what others think - in particular Bas as
the lead maintainer for all things static typing related. I will note that
Pandas is moving in the opposite direction, moving type annotations out of
the code base completely.


> > IDEs should be able to deal with stub files anyway for compiled code
> (i.e.,
> > the majority of NumPy), so that shouldn't be a fundamental issue right?
>
> You are right, IDEs have to support type stubs anyway, but unlike type
> checkers IDEs cannot always prefer type stubs to Python sources (if they
> are available), because some IDE features need metadata which is *not*
> available in type stubs (e.g. docstring and source location).
>
> To make things a bit more specific, consider the following snippet
>
> ```
> import numpy as np
> np.array()
> ```
>
> Here a user is in the middle of calling np.array and an IDE could render
>
> * the signature(s) of np.array (available in type stubs)
> * the docstring of np.array (available in Python sources)
> * (maybe) the exact module where np.array is defined (available in both)
>
> "Go to definition" on np.array should take the user to the implementation
> of np.array or to both the implementation and the type stub.
>

"Go to definition" clearly should go to the code in the .py source file.
Which is also where the signature is. For rendering something like a
tooltip you really do not want to see the complex type annotations that we
have. Something like this will make zero sense to the average user in an
IDE:

```
def average(
   a: _ArrayLikeFloat_co,
   axis: None = ...,
   weights: None | _ArrayLikeFloat_co= ...,
   returned: L[False] = ...,
) -> floating[Any]: ...
```
And that's even ignoring the mountain of overloads we have - showing those
would be much worse. Instead, you want the unannotated signature in the .py
file:

def average(a, axis=None, weights=None, returned=False):

Ralf


> Hovering over np.array would also need the signature and the docstring,
> but you could also imagine it rendering e.g. a snippet of the
> implementation etc.
>
> Sergei
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: ralf.gomm...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Types in pure Python sources

[bas van beek]

I share Ralf’s general sentiment here; the abundance of overloads makes it in 
my opinion undesirable to move to inline annotations due the sheer amount of 
extra clutter.

This is in part due to dtype-typing support, which is currently difficult to 
express without overloads, though I do expect this to change at some point in 
future (hopefully…).
However, there are also other common patterns that I don’t think we can 
reasonably express without overloads (be it now or in the future),
namely parameters such as `out`, `dtype`, `axis` and `keepdims`, all of which 
have a profound effect on the output type.

Regards, Bas

From: Ralf Gommers 
Sent: Tuesday, 3 May 2022 20:45
To: Discussion of Numerical Python 
Subject: [Numpy-discussion] Re: Types in pure Python sources



On Sun, May 1, 2022 at 9:08 PM Sergei Lebedev 
mailto:sergei.a.lebe...@gmail.com>> wrote:
> I think that's on purpose, because the type annotations are quite complex.
> For reasons of correctness/completeleness, they use protocols, mixins, and
> overloads. Inside pure Python code, that would be harder to read and
> maintain.

I agree that NumPy type annotations are quite complex. However, having them 
alongside the implementation might actually make maintenance easier, because 
there will be no need to update type stubs separately from the implementation.

I think I disagree, but let's see what others think - in particular Bas as the 
lead maintainer for all things static typing related. I will note that Pandas 
is moving in the opposite direction, moving type annotations out of the code 
base completely.


> IDEs should be able to deal with stub files anyway for compiled code (i.e.,
> the majority of NumPy), so that shouldn't be a fundamental issue right?

You are right, IDEs have to support type stubs anyway, but unlike type checkers 
IDEs cannot always prefer type stubs to Python sources (if they are available), 
because some IDE features need metadata which is *not* available in type stubs 
(e.g. docstring and source location).

To make things a bit more specific, consider the following snippet

```
import numpy as np
np.array()
```

Here a user is in the middle of calling np.array and an IDE could render

* the signature(s) of np.array (available in type stubs)
* the docstring of np.array (available in Python sources)
* (maybe) the exact module where np.array is defined (available in both)

"Go to definition" on np.array should take the user to the implementation of 
np.array or to both the implementation and the type stub.

"Go to definition" clearly should go to the code in the .py source file. Which 
is also where the signature is. For rendering something like a tooltip you 
really do not want to see the complex type annotations that we have. Something 
like this will make zero sense to the average user in an IDE:

```
def average(
   a: _ArrayLikeFloat_co,
   axis: None = ...,
   weights: None | _ArrayLikeFloat_co= ...,
   returned: L[False] = ...,
) -> floating[Any]: ...
```
And that's even ignoring the mountain of overloads we have - showing those 
would be much worse. Instead, you want the unannotated signature in the .py 
file:

def average(a, axis=None, weights=None, returned=False):

Ralf


Hovering over np.array would also need the signature and the docstring, but you 
could also imagine it rendering e.g. a snippet of the implementation etc.

Sergei
___
NumPy-Discussion mailing list -- 
numpy-discussion@python.org
To unsubscribe send an email to 
numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: ralf.gomm...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Discussion: History & Possible Deprecation/Removal of numpy.linalg.{eig(), eigvals()}?

[Leo Fang]

Hi, I am catching up with an assigned task that I slipped. I am wondering a few 
things about the numpy.linalg.eig() API (eigvals() is also included in this 
discussion, but for simplicity let's focus on eig()). AFAIU the purpose of 
eig() is for handling general eigen-systems which have left and right 
eigenvectors (that could differ beyond transpose/conjugate). For this, I have 
two questions:

1. What's the history of NumPy adding this eig() routine? My guess would be 
NumPy added it because Matlab had it, but I couldn't tell from a quick commit 
search.
2. Would it be possible to deprecate/remove this API? From past discussions 
with users and developers, I found:
- numpy.linalg.eig() only returns the right eigenvectors, so it's not very 
useful:
  * When the left eigenvectors are not needed --- which is the majority of the 
use cases --- eigh() is the right routine, but most users (especially those 
transitioning from Matlab) don't know it's there
  * When the left eigenvectors are needed, users have to use scipy.linalg.eig() 
anyway
- A general support for eig() could be challenging (numerical issues), and 
doing SVDs instead is usually the better way out (in terms of both numerical 
stability and mathematical sense).

Thanks,
Leo
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Discussion: History & Possible Deprecation/Removal of numpy.linalg.{eig(), eigvals()}?

[Charles R Harris]

On Tue, May 3, 2022 at 6:41 PM Leo Fang  wrote:

> Hi, I am catching up with an assigned task that I slipped. I am wondering
> a few things about the numpy.linalg.eig() API (eigvals() is also included
> in this discussion, but for simplicity let's focus on eig()). AFAIU the
> purpose of eig() is for handling general eigen-systems which have left and
> right eigenvectors (that could differ beyond transpose/conjugate). For
> this, I have two questions:
>
> 1. What's the history of NumPy adding this eig() routine? My guess would
> be NumPy added it because Matlab had it, but I couldn't tell from a quick
> commit search.
> 2. Would it be possible to deprecate/remove this API? From past
> discussions with users and developers, I found:
> - numpy.linalg.eig() only returns the right eigenvectors, so it's not very
> useful:
>   * When the left eigenvectors are not needed --- which is the majority of
> the use cases --- eigh() is the right routine, but most users (especially
> those transitioning from Matlab) don't know it's there
>

eigh() is for Hermitian matrices, the svd won't return eigenvectors in the
general case, especially when they are not orthogonal or form a complete
basis. The left eigenvectors can be obtained by passing the transpose to
eig().

In my experience, the main problem with eig() is accuracy, and sometimes
(IIRC) it doesn't find all the eigenvectors, so it should not be used when
the matrix is Hermitian.


>   * When the left eigenvectors are needed, users have to use
> scipy.linalg.eig() anyway
> - A general support for eig() could be challenging (numerical issues), and
> doing SVDs instead is usually the better way out (in terms of both
> numerical stability and mathematical sense).
>
>
Chuck
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Discussion: History & Possible Deprecation/Removal of numpy.linalg.{eig(), eigvals()}?

[Robert Kern]

On Tue, May 3, 2022 at 8:40 PM Leo Fang  wrote:

> Hi, I am catching up with an assigned task that I slipped. I am wondering
> a few things about the numpy.linalg.eig() API (eigvals() is also included
> in this discussion, but for simplicity let's focus on eig()). AFAIU the
> purpose of eig() is for handling general eigen-systems which have left and
> right eigenvectors (that could differ beyond transpose/conjugate).


For this, I have two questions:
>
> 1. What's the history of NumPy adding this eig() routine? My guess would
> be NumPy added it because Matlab had it, but I couldn't tell from a quick
> commit search.
>

It essentially dates back to Numeric, though the name was a bit different.
But the core feature of only computing right-eigenvectors was there.
scipy.linalg.eig() came along and expose both left- and right-eigenvalues
(with close to the present signature, I think). Then numpy took over from
Numeric and adopted the scipy.linalg names for the routines, but cut out a
number of the features that made some of the interfaces complicated. Since
Numeric only did right-eigenvalues (what most people want, even in the
general-matrix case), so did numpy.


> 2. Would it be possible to deprecate/remove this API? From past
> discussions with users and developers, I found:
> - numpy.linalg.eig() only returns the right eigenvectors, so it's not very
> useful:
>   * When the left eigenvectors are not needed --- which is the majority of
> the use cases --- eigh() is the right routine, but most users (especially
> those transitioning from Matlab) don't know it's there
>

eigh() is only for symmetric/Hermitian matrices.


>   * When the left eigenvectors are needed, users have to use
> scipy.linalg.eig() anyway
>

It seems to me that you are assuming that in the non-Hermitian case, the
right-eigenvectors are useless without the left-eigenvectors. It isn't
obvious to me why that would be the case. It is true that the left- and
right-eigenvectors differ from each other, but it's not clear to me that in
every application with a general matrix I need to compute both.

- A general support for eig() could be challenging (numerical issues), and
> doing SVDs instead is usually the better way out (in terms of both
> numerical stability and mathematical sense).
>

Computing the left-eigenvectors is no real problem. The underlying _GEEV
LAPACK routine can do it if asked. It's the same routine under
scipy.linalg.eig().

-- 
Robert Kern
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com