On Thu, Apr 9, 2020 at 12:02 AM Sebastian Berg <sebast...@sipsolutions.net> wrote:
> On Wed, 2020-04-08 at 17:04 -0400, Andreas Mueller wrote: > > Hey all. > > Is there any update on this? Is there any input we can provide as > > users? > > I'm not entirely sure where you are in the decision making process > > right > > now :) > > > > Hey, > > thanks for the ping. Things are a bit stuck right now. I think what we > need is some clarity on the implications and alternatives. > I was thinking about organizing a small conference call with the main > people interested in the next weeks. > > There are also still some alternatives to this NEP in the race, and we > may need to clarify which ones are actually still in the race... > > > Maybe to see some of the possible sticking points: > > 1. What do we do about SciPy, have it under this umbrella? And how > would we want to design that. > Current feeling: best to ignore it for now. It's quite a bit of work to fix API incompatibilities for linalg that no one currently seems interested in tackling. We can revisit once that's done. > 2. Context managers have some composition issues, maybe less so if they > are in the downstream package. Or should we have global defaults as > well? > +1 for adding this right next to get_array_module(). > 3. How do we ensure safe transitions for users as much as possible. > * If you use this, can functions suddenly return a different type > in the future? > * Should we force you to cast to NumPy arrays in a transition > period, or force you to somehow silence a transition warning? > > 4. Is there a serious push to have a "reduced" API or even a versioned > API? > There is, it'll take a few months. > > But I am probably forgetting some other things. > > > In my personal opinion, I think NEP 37 with minor modifications is > still the best duck in the race. I feel we should be able to find a > reasonable solution for SciPy. > Point 2. about Context managers may be true, but this is much smaller > in scope from the ones uarray proposed IIRC, and I could not figure out > major scoping issues with it yet (the sklearn draft). > > About the safe transition, that may be the stickiest point. But e.g. if > you enable `get_array_module` sklearn could limit a certain function to > error out if it finds something other than NumPy? > The main problem is how to do opt-in into future behaviour. A context > manager can do that, although the danger is that someone just uses that > everywhere... > > On the reduced/versioned API front, I would hope that we can defer that > as a semi-orthogonal issue, basically saying that for now you have to > provide a NumPy API that faithfully reproduces whatever NumPy version > is installed on the system. > I think it would be nice to have a separate NEP 37 implementation outside of NumPy to play with. Unlike __array_function__, I don't think it has to go into NumPy immediately. This avoids the whole "experimental API" issue, it would be quite useful to test this with, e.g., CuPy + scikit-learn without being stuck with any decisions in a released NumPy version. Also makes switching on/off very easy for users, just (don't) `pip install numpy-array-module`. Cheers, Ralf > Cheers, > > Sebastian > > > > Cheers, > > Andy > > > > On 3/3/20 6:34 PM, Sebastian Berg wrote: > > > On Fri, 2020-02-28 at 11:28 -0500, Allan Haldane wrote: > > > > On 2/23/20 6:59 PM, Ralf Gommers wrote: > > > > > One of the main rationales for the whole NEP, and the argument > > > > > in > > > > > multiple places > > > > > ( > > > > > > https://numpy.org/neps/nep-0037-array-module.html#opt-in-vs-opt-out-for-users > > > > > ) > > > > > is that it's now opt-in while __array_function__ was opt-out. > > > > > This > > > > > isn't > > > > > really true - the problem is simply *moved*, from the duck > > > > > array > > > > > libraries to the array-consuming libraries. The end user will > > > > > still > > > > > see > > > > > the backwards incompatible change, with no way to turn it off. > > > > > It > > > > > will > > > > > be easier with __array_module__ to warn users, but this should > > > > > be > > > > > expanded on in the NEP. > > > > Might it be possible to flip this NEP back to opt-out while > > > > keeping > > > > the > > > > nice simplifications and configurabile array-creation routines, > > > > relative > > > > to __array_function__? > > > > > > > > That is, what if we define two modules, "numpy" and > > > > "numpy_strict". > > > > "numpy_strict" would raise an exception on duck-arrays defining > > > > __array_module__ (as numpy currently does). "numpy" would be a > > > > wrapper > > > > around "numpy_strict" that decorates all numpy methods with a > > > > call to > > > > "get_array_module(inputs).func(inputs)". > > > This would be possible, but I think we strongly leaned against the > > > idea. Basically, if you have to opt-out, from a library perspective > > > there may be `np.asarray` calls, which for example later call into > > > C > > > and expect arrays. > > > So, I have large doubts that an opt-out solution works easily for > > > library authors. Array function is opt-out, but effectively most > > > clean > > > library code already opted out... > > > > > > We had previously discussed the opposite, of having a namespace of > > > implicit dispatching based on get_array_module, but if we keep > > > array > > > function around, I am not sure there is much reason for it. > > > > > > > Then end-user code that did "import numpy as np" would accept > > > > ducktypes > > > > by default, while library developers who want to signal they > > > > don't > > > > support ducktypes can opt-out by doing "import numpy_strict as > > > > np". > > > > Issues with `np.as_array` seem mitigated compared to > > > > __array_function__ > > > > since that method would now be ducktype-aware. > > > My tendency is that if we want to go there, we would need to push > > > ahead > > > with the `np.duckarray()` idea instead. > > > > > > To be clear: I currently very much prefer the get_array_module() > > > idea. > > > It just seems much cleaner for library authors, and they are the > > > primary issue at the moment in my opinion. > > > > > > - Sebastian > > > > > > > > > > -Allan > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion@python.org > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > > > > > > > > > _______________________________________________ > > > > NumPy-Discussion mailing list > > > > NumPy-Discussion@python.org > > > > https://mail.python.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion@python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion >
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion