On Sun, 2020-02-23 at 22:44 -0800, Stephan Hoyer wrote: > > <snip> > On Sun, Feb 23, 2020 at 3:59 PM Ralf Gommers <ralf.gomm...@gmail.com> > wrote: > > > Also, I'm still not sure I agree with the tone of the discussion on > > this topic. It's very heavily inspired by what the JAX devs are > > telling you (the NEP still says PyTorch and scipy.sparse as well, > > but that's not true in both cases). If you ask Dask and CuPy for > > example, they're quite happy with __array_function__ and there > > haven't been many complaints about backwards compat breakage. > > > > I'm linking to comments you wrote in reference to PyTorch and > scipy.sparse in the current draft of the NEP, so I certainly want to > make sure that you agree my characterization :). > > Would it be fair to say: > - JAX is reluctant to implement __array_function__ because of > concerns about breaking existing code. JAX developers think that when > users use NumPy functions on JAX arrays, they are explicitly choosing > to convert from JAX to NumPy. This model is fundamentally > incompatible __array_function__, which we chose to override the > existing numpy namespace. > - PyTorch and scipy.sparse are not yet in position to implement > __array_function__ (due to a lack of a direct implementation of > NumPy's API), but these projects take backwards compatibility > seriously. > > Does "take backwards compatibility seriously" sound about right to > you? I'm very open to specific suggestions here. (TensorFlow could > probably also be safely added to this second list.) >
Just to be clear, the way scikit-learn would probably be handling backward compatibility concerns is by adding it to their configuration context manager, see: https://github.com/scikit-learn/scikit-learn/pull/16574 So the backward compat is in a sense solved (but there are project specific context managers involved – which is not perfect maybe, but OK). I am willing to consider pushing this off into its own namespace (and package, preferably in the NumPy org though) if necessary, the idea being that we keep it super minimal, and expand it as we go to keep up with scikit-learn needs. Possibly even with a function registration approach, so that you could have import time checks on function availability and signature mismatch easier. I still do not like the idea of context managers much though, I think I prefer the returned (bound) namespace a lot. Also I think we should *not* do implicit dispatching. Consider this case: def numpy_only(x): x = np.asarray(x) return x + _helper(len(x)) def generic(x): module = np.get_array_module(x) x = module.asarray(x) return x + _helper(len(x)) def _helper(n, module=np): return module.random.unform(size=n) If you try to make the above work with context managers, you _still_ need to pass in the module to _helper [1], because otherwise you would have to change the `numpy_only` function to ensure an outside context does not change its behaviour. - Sebastian [1] If "module" had a `module.set_backend()` and was a global instead `_helper` using the global module would do the wrong thing for `numpy_only`. This is of course also a bit of an issue with the sklearn context manager as well, but it seems to me _much_ less so, and probably not if most libraries slowly switch over and currently use `np.asarray`. > Best, > Stephan > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion
signature.asc
Description: This is a digitally signed message part
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
