On 10.09.19 05:32, Stephan Hoyer wrote:
On Mon, Sep 9, 2019 at 6:27 PM Ralf Gommers <ralf.gomm...@gmail.com
<mailto:ralf.gomm...@gmail.com>> wrote:
I think we've chosen to try the former - dispatch on functions so
we can reuse the NumPy API. It could work out well, it could give
some long-term maintenance issues, time will tell. The question is
now if and how to plug the gap that __array_function__ left. It's
main limitation is "doesn't work for functions that don't have an
array-like input" - that left out ~10-20% of functions. So now we
have a proposal for a structural solution to that last 10-20%. It
seems logical to want that gap plugged, rather than go back and
say "we shouldn't have gone for the first 80%, so let's go no
further".
I'm excited about solving the remaining 10-20% of use cases for
flexible array dispatching, but the unumpy interface suggested here
(numpy.overridable) feels like a redundant redo of __array_function__
and __array_ufunc__.
I would much rather continue to develop specialized protocols for the
remaining usecases. Summarizing those I've seen in this thread, these
include:
1. Overrides for customizing array creation and coercion.
2. Overrides to implement operations for new dtypes.
3. Overriding implementations of NumPy functions, e.g., FFT and ufuncs
with MKL.
(1) could mostly be solved by adding np.duckarray() and another
function for duck array coercion. There is still the matter of
overriding np.zeros and the like, which perhaps justifies another new
protocol, but in my experience the use-cases for truly an array from
scratch are quite rare.
While they're rare for libraries like XArray; CuPy, Dask and
PyData/Sparse need these.
(2) should be tackled as part of overhauling NumPy's dtype system to
better support user defined dtypes. But it should definitely be in the
form of specialized protocols, e.g., which pass in preallocated arrays
to into ufuncs for a new dtype. By design, new dtypes should not be
able to customize the semantics of array *structure*.
We already have a split in the type system with e.g. Cython's buffers,
Numba's parallel type system. This is a different issue altogether, e.g.
allowing a unyt dtype to spawn a unyt array, rather than forcing a
re-write of unyt to cooperate with NumPy's new dtype system.
(3) could potentially motivate a new solution, but it should exist
*inside* of select existing NumPy implementations, after checking for
overrides with __array_function__. If the only option NumPy provides
for overriding np.fft is to implement np.overrideable.fft, I doubt
that would suffice to convince MKL developers from monkey patching it
-- they already decided that a separate namespace is not good enough
for them.
That has already been addressed by Ralf in another email. We're
proposing to merge that into NumPy proper.
Also, you're missing a few:
4. Having default implementations that allow overrides of a large part
of the API while defining only a small part. This holds for e.g.
transpose/concatenate.
5. Generation of Random numbers (overriding RandomState). CuPy has its
own implementation which would be nice to override.
I also share Nathaniel's concern that the overrides in unumpy are too
powerful, by allowing for control from arbitrary function arguments
and even *non-local* control (i.e., global variables) from context
managers. This level of flexibility can make code very hard to debug,
especially in larger codebases.
Backend switching needs global context, in any case. There isn't a good
way around that other than the class dundermethods outlined in another
thread, which would require rewrites of large amounts of code.
Best,
Stephan
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
