Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Wed, Sep 26, 2018 at 9:19 AM Joris Van den Bossche < jorisvandenboss...@gmail.com> wrote: > Hi all, > > I have small question for clarification regarding some reducing numpy > functions like np.sum, np.mean, etc, i.e. functions that now work on > other array-like objects by checking if they hav

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

Hi all, I have small question for clarification regarding some reducing numpy functions like np.sum, np.mean, etc, i.e. functions that now work on other array-like objects by checking if they have 'sum', 'mean', etc method and if so, "dispatch" to the method. Those functions are several times ment

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Thu, Sep 20, 2018 at 2:05 PM Stephan Hoyer wrote: > I've gone ahead and merged the PR to mark the NEP as accepted. Let's get > started on the fun part of implementation! > > Cheers, > Stephan > I have started implementing NEP-18 in a pull request: https://github.com/numpy/numpy/pull/12005 I

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Thu, Sep 20, 2018 at 2:33 AM Nathaniel Smith wrote: > On Thu, Sep 13, 2018 at 10:30 AM, Stephan Hoyer wrote: > > I have now drafted these revisions to the NEP to clarify its stance > around > > backwards compatibility, and the type of the "types" argument: > > https://github.com/numpy/numpy/p

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Thu, Sep 13, 2018 at 10:30 AM, Stephan Hoyer wrote: > I have now drafted these revisions to the NEP to clarify its stance around > backwards compatibility, and the type of the "types" argument: > https://github.com/numpy/numpy/pull/11943 Okay, so this is a pretty substantial change! Before, th

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Thu, Sep 13, 2018 at 10:30 AM Stephan Hoyer wrote: > On Sat, Sep 8, 2018 at 7:12 PM Charles R Harris > wrote: > >> On Wed, Aug 1, 2018 at 6:27 PM Stephan Hoyer wrote: >> >>> I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level >>> array functions": >>> http://www.numpy.or

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

> On Thursday, Sep 13, 2018 at 7:30 PM, Stephan Hoyer (mailto:sho...@gmail.com)> wrote: > > > On Sat, Sep 8, 2018 at 7:12 PM Charles R Harris (mailto:charlesr.har...@gmail.com)> wrote: > > > > > > On Wed, Aug 1, 2018 at 6:27 PM Stephan Hoyer > (mailto:sho...@gmail.com)> wrote: > > > > > > > > >

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

Thanks for adding the clarifications. They read well to me, and I think make clear to a project like astropy what to expect (and I expect we'll be use it!). -- Marten ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/m

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Sat, Sep 8, 2018 at 7:12 PM Charles R Harris wrote: > On Wed, Aug 1, 2018 at 6:27 PM Stephan Hoyer wrote: > >> I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level >> array functions": >> http://www.numpy.org/neps/nep-0018-array-function-protocol.html >> >> Since the last

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Wed, Aug 1, 2018 at 6:27 PM Stephan Hoyer wrote: > I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level > array functions": > http://www.numpy.org/neps/nep-0018-array-function-protocol.html > > Since the last round of discussion, we added a new section on "Callable > object

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Sun, Sep 2, 2018 at 12:58 PM Charles R Harris wrote: > > > On Wed, Aug 1, 2018 at 6:27 PM Stephan Hoyer wrote: > >> I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level >> array functions": >> http://www.numpy.org/neps/nep-0018-array-function-protocol.html >> >> Since the

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Wed, Aug 1, 2018 at 6:27 PM Stephan Hoyer wrote: > I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level > array functions": > http://www.numpy.org/neps/nep-0018-array-function-protocol.html > > Since the last round of discussion, we added a new section on "Callable > object

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Wed, Aug 29, 2018 at 1:38 AM Nathaniel Smith wrote: > The official Tensorflow wheels flat out lie about being manylinux > compatible, and the Tensorflow team has never talked to anyone about > how to fix this, they just upload them to PyPI and leave others get to > deal with the fallout [1]. T

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Wed, Aug 29, 2018 at 9:34 AM Marten van Kerkwijk < m.h.vankerkw...@gmail.com> wrote: > Absolutely fine to have to deal with future chances - my main point is > that by accepting the NEP, I think numpy is committing to provide some way > to override whatever functions __array_function__ is intro

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Wed, Aug 29, 2018, 02:44 Matti Picus wrote: > On 29/08/18 10:37, Nathaniel Smith wrote: > > it's easy to imagine scenarios where the > > people being broken aren't the ones who had a chance to read the docs > > – e.g. if a major package starts relying on __array_function__, then > > it's all*t

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

Absolutely fine to have to deal with future chances - my main point is that by accepting the NEP, I think numpy is committing to provide some way to override whatever functions __array_function__ is introduced for, i.e., we cannot reasonably go back to not providing any way to override such a funct

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

631) 344-3281 (no voicemail set up) From: NumPy-Discussion on behalf of Marten van Kerkwijk Sent: Wednesday, August 29, 2018 10:43:55 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol On Wed, Aug 29, 2018 at 9:5

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Wed, Aug 29, 2018 at 9:53 AM Matthew Rocklin wrote: > > On the backwards compatibility: from an astropy perspective, I would > expect that the introduction of `__array_function__` implies a guarantee > that the *functionality* it provides will remain, > > My guess is that you wouldn't have thi

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

> On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain, My guess is that you wouldn't have this expectation if Numpy released this feature with explicit "Experi

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

HI All, On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain, i.e., that it will continue to be possible to override, say, concatenate. It is not a big deal if

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

>> 1. if we do find ourselves in a situation where changing this would break lots of users, will we consider ourselves beholden to them? I think that it would be useful for Numpy's continued evolution to develop the ability to include code on a provisional basis. Other projects do this and they j

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

> On 29. Aug 2018, at 11:44, Matti Picus wrote: > > On 29/08/18 10:37, Nathaniel Smith wrote: >> it's easy to imagine scenarios where the >> people being broken aren't the ones who had a chance to read the docs >> – e.g. if a major package starts relying on __array_function__, then >> it's all*th

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On 29/08/18 10:37, Nathaniel Smith wrote: it's easy to imagine scenarios where the people being broken aren't the ones who had a chance to read the docs – e.g. if a major package starts relying on __array_function__, then it's all*their* users who we'd be breaking, even though they had nothing t

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Fri, Aug 24, 2018 at 4:00 PM, Stephan Hoyer wrote: > On Fri, Aug 24, 2018 at 3:14 PM Nathaniel Smith wrote: >> >> Yeah, the reason warnings are normally recommended is because >> normally, you want to make it easy to silence. But this is the rare >> case where I didn't want to make it easy to

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Fri, Aug 24, 2018 at 3:14 PM Nathaniel Smith wrote: > Yeah, the reason warnings are normally recommended is because > normally, you want to make it easy to silence. But this is the rare > case where I didn't want to make it easy to silence, so I didn't > suggest using a warning :-). > > Callin

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

> I’m On 25. Aug 2018, at 00:13, Nathaniel Smith wrote: > >> On Fri, Aug 24, 2018 at 1:46 PM, Stephan Hoyer wrote: >> On Fri, Aug 24, 2018 at 1:36 AM Hameer Abbasi >> wrote: >>> >>> On Fri, Aug 24, 2018 at 9:38 AM Nathaniel Smith wrote: > On Thu, Aug 23, 2018 at 9:02 AM, wro

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Fri, Aug 24, 2018 at 1:46 PM, Stephan Hoyer wrote: > On Fri, Aug 24, 2018 at 1:36 AM Hameer Abbasi > wrote: >> >> >> On Fri, Aug 24, 2018 at 9:38 AM Nathaniel Smith wrote: >>> >>> On Thu, Aug 23, 2018 at 9:02 AM, wrote: >>> > I might add that most duck array authors are highly unlikely to b

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Fri, Aug 24, 2018 at 1:36 AM Hameer Abbasi wrote: > > On Fri, Aug 24, 2018 at 9:38 AM Nathaniel Smith wrote: > >> On Thu, Aug 23, 2018 at 9:02 AM, wrote: >> > I might add that most duck array authors are highly unlikely to be >> newcomers >> > to the Python space. We should just put a big w

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Fri, Aug 24, 2018, 09:07 Hameer Abbasi wrote: > > > On Fri, Aug 24, 2018 at 5:55 PM Stephan Hoyer wrote: > >> On Thu, Aug 23, 2018 at 1:06 PM Hameer Abbasi >> wrote: >> >>> I might add that if it’s a mandatory part of the protocol, then not all >>> things will work. For example, if XArray an

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Fri, Aug 24, 2018 at 5:55 PM Stephan Hoyer wrote: > On Thu, Aug 23, 2018 at 1:06 PM Hameer Abbasi > wrote: > >> I might add that if it’s a mandatory part of the protocol, then not all >> things will work. For example, if XArray and Dask want to support sparse >> arrays, they’ll need to add an

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Thu, Aug 23, 2018 at 1:06 PM Hameer Abbasi wrote: > I might add that if it’s a mandatory part of the protocol, then not all > things will work. For example, if XArray and Dask want to support sparse > arrays, they’ll need to add an explicit dependency. > I don't follow -- can you please elabo

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

Hi everyone, On Fri, Aug 24, 2018 at 9:38 AM Nathaniel Smith wrote: > On Thu, Aug 23, 2018 at 9:02 AM, wrote: > > I might add that most duck array authors are highly unlikely to be > newcomers > > to the Python space. We should just put a big warning there while > enabling > > and that’ll be e

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Thu, Aug 23, 2018 at 9:02 AM, wrote: > I might add that most duck array authors are highly unlikely to be newcomers > to the Python space. We should just put a big warning there while enabling > and that’ll be enough to scare away most devs from doing it by default. That's a reasonable idea..

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

> On 23. Aug 2018, at 18:37, Stephan Hoyer wrote: > > RE: the types argument > > On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith > wrote: > This is much more of a detail as compared to the rest of the > discussion, so I don't want to quibble too much about it. (Especia

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

RE: the types argument On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith wrote: > This is much more of a detail as compared to the rest of the > discussion, so I don't want to quibble too much about it. (Especially > since if we keep things really-provisional, we can change our mind > about the a

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

Hi everyone, > On 23. Aug 2018, at 17:35, Stephan Hoyer wrote: > >> On Tue, Aug 21, 2018 at 6:57 PM Nathaniel Smith wrote: >> I mean, the idea of the envvar is to be a temporary measure enable >> devs to experiment with a provisional feature, while being awkward >> enough that people don't buil

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Tue, Aug 21, 2018 at 6:57 PM Nathaniel Smith wrote: > I mean, the idea of the envvar is to be a temporary measure enable > devs to experiment with a provisional feature, while being awkward > enough that people don't build lots of stuff assuming its there. It > doesn't have to 100% supported i

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Tue, Aug 21, 2018 at 6:47 PM Nathaniel Smith wrote: > On Tue, Aug 21, 2018 at 9:39 AM, Stephan Hoyer wrote: > > It is quite possible that NumPy functions could be (re)written in a way > that > > is incompatible with some unit implementations but is perfectly valid for > > "full" duck arrays.

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Wed, Aug 22, 2018 at 4:22 AM Hameer Abbasi wrote: > May I propose an alternative that was already discussed, and one that I > think everyone will be okay with: > That's a dangerous assumption on this list:) We put all overridable functions inside a new submodule, numpy.api, that > will initi

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

Hi Nathaniel and Stephan, Since this conversation is getting a bit lengthy and I see a lot of repeated stuff, I’ll summarise the arguments for everyone’s benefit and then present my own viewpoints: Nathaniel: Undue maintenance burden on NumPy, since semantics have to match exactly Implementatio

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Tue, Aug 21, 2018 at 6:12 PM, Stephan Hoyer wrote: > On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith wrote: >> >> >> My suggestion: at numpy import time, check for an envvar, like say >> >> NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1. If it's not set, then all the >> >> __array_function__ dispatches

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Tue, Aug 21, 2018 at 9:39 AM, Stephan Hoyer wrote: > On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith wrote: >> >> On Wed, Aug 15, 2018 at 9:45 AM, Stephan Hoyer wrote: >> > This avoids a classic subclassing problem that has plagued NumPy for >> > years, >> > where overriding the behavior of

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith wrote: > >> My suggestion: at numpy import time, check for an envvar, like say > >> NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1. If it's not set, then all the > >> __array_function__ dispatches turn into no-ops. This lets interested > >> downstream librari

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Tue, Aug 21, 2018 at 12:21 AM Nathaniel Smith wrote: > On Wed, Aug 15, 2018 at 9:45 AM, Stephan Hoyer wrote: > > This avoids a classic subclassing problem that has plagued NumPy for > years, > > where overriding the behavior of method A causes apparently unrelated > method > > B to break, bec

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

I’m +0 on removing it, so mostly neutral, but slightly in favour. While I see the argument for having it, I also see it as a violation of DRY... The information is already available in relevant arguments. I doubt any people implementing this protocol are going to be lazy enough not to implement

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

> >> I don't really understand the 'types' frozenset. The NEP says "it will > >> be used by most __array_function__ methods, which otherwise would need > >> to extract this information themselves"... but they still need to > >> extract the information themselves, because they still have to examine

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Wed, Aug 15, 2018 at 9:45 AM, Stephan Hoyer wrote: > RE: accidental differences in behavior: > > I actually think that the __array_function__ approach is *less* prone to > accidental differences in behavior, because we require implementing every > function directly (or it raises an error). > >

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Mon, Aug 20, 2018 at 8:27 AM Charles R Harris wrote: > Ping to finish up this discussion so we can come to a conclusion. I'm in > favor of the NEP, as I don't see it as orthogonal to Nathaniel's concerns. > However, we might want to be selective as to which functions we expose via > the `__arr

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

Ping to finish up this discussion so we can come to a conclusion. I'm in favor of the NEP, as I don't see it as orthogonal to Nathaniel's concerns. However, we might want to be selective as to which functions we expose via the `__array_function__` method. On Wed, Aug 15, 2018 at 10:45 AM, Stephan

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On 15/08/18 19:44, Matthew Brett wrote: My suspicion is, to the extent that Matti and Tyler can devote time and energy to shepherding the discussion, these will become quicker and more productive. Since my name was mentioned .. Even if we could implement pull requ

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

Nathaniel, Thanks for raising these thoughtful concerns. Your independent review of this proposal is greatly appreciated! See my responses inline below: On Mon, Aug 13, 2018 at 2:44 AM Nathaniel Smith wrote: > The other approach would be to incrementally add clean, well-defined > dunder method

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

Hi, On Wed, Aug 15, 2018 at 5:36 PM, Hameer Abbasi wrote: > On 15. Aug 2018, at 18:25, Matthew Brett wrote: > > Hi, > > Thanks Nathaniel for this thoughtful response. > > On Mon, Aug 13, 2018 at 10:44 AM, Nathaniel Smith wrote: > ... > > The other approach would be to incrementally add clean, w

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Wed, Aug 15, 2018 at 10:25 AM, Matthew Brett wrote: > Hi, > > Thanks Nathaniel for this thoughtful response. > > On Mon, Aug 13, 2018 at 10:44 AM, Nathaniel Smith wrote: > ... > > The other approach would be to incrementally add clean, well-defined > > dunder methods like __array_ufunc__, __a

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

> On 15. Aug 2018, at 18:25, Matthew Brett wrote: > > Hi, > > Thanks Nathaniel for this thoughtful response. > > On Mon, Aug 13, 2018 at 10:44 AM, Nathaniel Smith wrote: > ... >> The other approach would be to incrementally add clean, well-defined >> dunder methods like __array_ufunc__, __arra

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

Hi, Thanks Nathaniel for this thoughtful response. On Mon, Aug 13, 2018 at 10:44 AM, Nathaniel Smith wrote: ... > The other approach would be to incrementally add clean, well-defined > dunder methods like __array_ufunc__, __array_concatenate__, etc. This > way we end up putting some thought into

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

Hi Nathaniel, I appreciate the clarification. Thank you for that. For what it's worth, I think that you may overestimate my involvement in the writing of that NEP. I sat down with Stephan during a Numpy dev meeting and we hacked something together. Afterwards several other people poured their

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

On Mon, Aug 13, 2018 at 2:44 AM, Nathaniel Smith wrote: > So this is like... an extreme version of technical debt. You're making > a deal with the devil for wealth and fame, and then eventually the > bill becomes due. It's hard for me to say categorically that this is a > bad idea – empirically, i

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

Hi Nathaniel, Very well written summary, it provides a lot of perspective into the different ways that this could go wrong. Here is a little commentary. > On 13. Aug 2018, at 11:44, Nathaniel Smith wrote: > > Hey all, > > So I've finally read through NEP 18 (__array_function__). Sorry again >

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

Hey all, So I've finally read through NEP 18 (__array_function__). Sorry again for the delay! It's an impressive piece of work! Thanks to the many authors; there's clearly been a lot of thought put into this. # The trade-off between comprehensive APIs versus clean APIs At a high-level, what mak

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

OK, I am can give you a few more days. I'll be camping through the weekend, but hope to accept it when I get back on Monday! On Wed, Aug 8, 2018 at 1:07 AM Nathaniel Smith wrote: > I'm sorry, I've been trying to find the time to read what you ended up > with, and haven't managed yet – could I ge

Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

I'm sorry, I've been trying to find the time to read what you ended up with, and haven't managed yet – could I get a few days extension? :-) -n On Wed, Aug 1, 2018 at 5:27 PM, Stephan Hoyer wrote: > I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level > array functions": > ht

[Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol

I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level array functions": http://www.numpy.org/neps/nep-0018-array-function-protocol.html Since the last round of discussion, we added a new section on "Callable objects generated at runtime" clarifying that to handle such objects is