[Numpy-discussion] "Spring cleaning" sprint
Hi all, we are planning to remote spring to try to reduce the number of open PRs and issues, and do triage work. This is planned for next Tuesday, October 14th, between 9:00 and 15:00 Pacific time. And everyone is invited to join in for as long as you wish. I assume we will have a video chat up and running at: https://berkeley.zoom.us/j/762261535 However, we will send a reminder/logistics email when we start with the sprint. Cheers, Sebastian signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NEP 31 — Context-local and global overrides of the NumPy API
Hi all, and thank you for all your hard work with this.
I wanted to provide more of an "end user" perspective than I think has been
present in this discussion so far. Over the past month, I've quickly skimmed
some emails on this thread and skipped others altogether. I am far from a NumPy
novice, but essentially *all* of the discussion went over my head. For a while
my attitude was "Oh well, far smarter people than me are dealing with this,
I'll let them figure it out." Looking at the participants in the thread, I
worry that this is the attitude almost everyone has taken, and that the
solution proposed will not be easy enough to deal with for any meaningful
adoption. Certainly with `__array_function__` I only took interest when our
tests broke with 1.17rc1.
Today I was particularly interested because I'm working to improve scikit-image
support for pyopencl.Array inputs. I went back and read the original NEP and
the latest iteration. Thank you again for the discussion, because the latest is
indeed a vast improvement over the original.
I think the very motivation has the wrong focus. I would summarise it as "we've
been coming up with all kinds of ways to do multiple dispatch for array-likes,
and we've found that we need more ways, so let's come up with the One True
Way." I think the focus should be on the users and community. Something along
the lines of: "New implementations of array computing are cropping up left,
right, and centre in Python (not to speak of other languages!). There are good
reasons for this (GPUs, distributed computing, sparse data, etc), but it leaves
users and library authors in a pickle: how can they ensure that their
functions, written with NumPy array inputs and outputs in mind, work well in
this ecosystem?"
With this new motivation in mind, I think that the user story below is (a) the
best part of the NEP, but (b) underdeveloped. The NEP is all about "if I want
my array implementation to work with this fancy dispatch system, what do I need
to do?". But there should be more of "in user situations X, Y, and Z, what is
the desired behaviour?"
> The way we propose the overrides will be used by end users is::
>
> # On the library side
>
> import numpy.overridable as unp
>
> def library_function(array):
> array = unp.asarray(array)
> # Code using unumpy as usual
> return array
>
> # On the user side:
>
> import numpy.overridable as unp
> import uarray as ua
> import dask.array as da
>
> ua.register_backend(da)
>
> library_function(dask_array) # works and returns dask_array
>
> with unp.set_backend(da):
> library_function([1, 2, 3, 4]) # actually returns a Dask array.
>
> Here, ``backend`` can be any compatible object defined either by NumPy or an
> external library, such as Dask or CuPy. Ideally, it should be the module
> ``dask.array`` or ``cupy`` itself.
Some questions about the above:
- What happens if I call `library_function(dask_array)` without registering
`da` as a backend first? Will `unp.asarray` try to instantiate a potentially
100GB array? This seems bad.
- To get `library_function`, I presumably have to do `from fancy_array_library
import library_function`. Can the code in `fancy_array_library` itself register
backends, and if so, should/would fancy array libraries that want to maximise
compatibility pre-register a bunch of backends so that users don't have to?
Here are a couple of code snippets that I would *want* to "just work". Maybe
it's unreasonable, but imho the NEP should provide these as use cases
(specifically: how library_function should be written so that they work, and
what dask.array and pytorch would need to do so that they work, OR, why the NEP
doesn't solve them).
1.
from dask import array as da
from fancy_array_library import library_function # hopefully skimage one day ;)
data = da.from_zarr('myfile.zarr')
result = library_function(data) # result should still be dask, all things being
equal
result.to_zarr('output.zarr')
2.
from dask import array as da
from magic_library import pytorch_predict
data = da.from_zarr('myfile.zarr')
result = pytorch_predict(data) # normally here I would use e.g.
data.map_overlap, but could this be done magically?
result.to_zarr('output.zarr')
There's probably a whole bunch of other "user stories" one can concoct, and no
doubt many from the authors of the NEP themselves, but they don't come through
in the NEP text. My apologies that I haven't read *all* the references: I
understand that it is frustrating if the above are addressed there, but I think
it's important to have this kind of context in the NEP itself.
Thank you again, and I hope the above is helpful rather than feels like more
unnecessary churn.
Juan.___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] "Spring cleaning" sprint
Hi Sebastian, It is Tuesday October 15 or Monday October 14? Regards, ZJ On Wed, Oct 9, 2019 at 5:57 PM Sebastian Berg wrote: > Hi all, > > we are planning to remote spring to try to reduce the number of open > PRs and issues, and do triage work. > > This is planned for next Tuesday, October 14th, between 9:00 and 15:00 > Pacific time. And everyone is invited to join in for as long as you > wish. > > I assume we will have a video chat up and running at: > https://berkeley.zoom.us/j/762261535 > > However, we will send a reminder/logistics email when we start with the > sprint. > > Cheers, > > Sebastian > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] "Spring cleaning" sprint
On Wed, 2019-10-09 at 21:26 -0700, Zijie Poh wrote: > Hi Sebastian, > > It is Tuesday October 15 or Monday October 14? > Sorry, its Tuesday the 15th [0]. Monday is a holiday in California at least. Cheers, Sebastian [0] Probably happens to me because I am still used to weeks starting with Monday and not Sunday (as my calendar now shows). > Regards, > ZJ > > On Wed, Oct 9, 2019 at 5:57 PM Sebastian Berg < > sebast...@sipsolutions.net> wrote: > > Hi all, > > > > we are planning to remote spring to try to reduce the number of > > open > > PRs and issues, and do triage work. > > > > This is planned for next Tuesday, October 14th, between 9:00 and > > 15:00 > > Pacific time. And everyone is invited to join in for as long as you > > wish. > > > > I assume we will have a video chat up and running at: > > https://berkeley.zoom.us/j/762261535 > > > > However, we will send a reminder/logistics email when we start with > > the > > sprint. > > > > Cheers, > > > > Sebastian > > ___ > > NumPy-Discussion mailing list > > NumPy-Discussion@python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NEP 31 — Context-local and global overrides of the NumPy API
On Wed, Oct 9, 2019 at 6:00 PM Juan Nunez-Iglesias wrote:
> Hi all, and thank you for all your hard work with this.
>
> I wanted to provide more of an "end user" perspective than I think has
> been present in this discussion so far. Over the past month, I've quickly
> skimmed some emails on this thread and skipped others altogether. I am far
> from a NumPy novice, but essentially *all* of the discussion went over my
> head. For a while my attitude was "Oh well, far smarter people than me are
> dealing with this, I'll let them figure it out." Looking at the
> participants in the thread, I worry that this is the attitude almost
> everyone has taken, and that the solution proposed will not be easy enough
> to deal with for any meaningful adoption. Certainly with
> `__array_function__` I only took interest when our tests broke with 1.17rc1.
>
> Today I was particularly interested because I'm working to improve
> scikit-image support for pyopencl.Array inputs. I went back and read the
> original NEP and the latest iteration. Thank you again for the discussion,
> because the latest is indeed a vast improvement over the original.
>
> I think the very motivation has the wrong focus. I would summarise it as
> "we've been coming up with all kinds of ways to do multiple dispatch for
> array-likes, and we've found that we need more ways, so let's come up with
> the One True Way." I think the focus should be on the users and community.
> Something along the lines of: "New implementations of array computing are
> cropping up left, right, and centre in Python (not to speak of other
> languages!). There are good reasons for this (GPUs, distributed computing,
> sparse data, etc), but it leaves users and library authors in a pickle: how
> can they ensure that their functions, written with NumPy array inputs and
> outputs in mind, work well in this ecosystem?"
>
> With this new motivation in mind, I think that the user story below is (a)
> the best part of the NEP, but (b) underdeveloped. The NEP is all about "if
> I want my array implementation to work with this fancy dispatch system,
> what do I need to do?". But there should be more of "in user situations X,
> Y, and Z, what is the desired behaviour?"
>
> The way we propose the overrides will be used by end users is::
>
> # On the library side
>
> import numpy.overridable as unp
>
> def library_function(array):
> array = unp.asarray(array)
> # Code using unumpy as usual
> return array
>
> # On the user side:
>
> import numpy.overridable as unp
> import uarray as ua
> import dask.array as da
>
> ua.register_backend(da)
>
> library_function(dask_array) # works and returns dask_array
>
> with unp.set_backend(da):
> library_function([1, 2, 3, 4]) # actually returns a Dask array.
>
> Here, ``backend`` can be any compatible object defined either by NumPy or
> an
> external library, such as Dask or CuPy. Ideally, it should be the module
> ``dask.array`` or ``cupy`` itself.
>
>
> Some questions about the above:
>
> - What happens if I call `library_function(dask_array)` without
> registering `da` as a backend first? Will `unp.asarray` try to instantiate
> a potentially 100GB array? This seems bad.
> - To get `library_function`, I presumably have to do `from
> fancy_array_library import library_function`. Can the code in
> `fancy_array_library` itself register backends, and if so, should/would
> fancy array libraries that want to maximise compatibility pre-register a
> bunch of backends so that users don't have to?
>
> Here are a couple of code snippets that I would *want* to "just work".
> Maybe it's unreasonable, but imho the NEP should provide these as use cases
> (specifically: how library_function should be written so that they work,
> and what dask.array and pytorch would need to do so that they work, OR, why
> the NEP doesn't solve them).
>
> 1.
> from dask import array as da
> from fancy_array_library import library_function # hopefully skimage one
> day ;)
>
> data = da.from_zarr('myfile.zarr')
> result = library_function(data) # result should still be dask, all things
> being equal
> result.to_zarr('output.zarr')
>
> 2.
> from dask import array as da
> from magic_library import pytorch_predict
>
> data = da.from_zarr('myfile.zarr')
> result = pytorch_predict(data) # normally here I would use e.g.
> data.map_overlap, but could this be done magically?
> result.to_zarr('output.zarr')
>
>
> There's probably a whole bunch of other "user stories" one can concoct,
> and no doubt many from the authors of the NEP themselves, but they don't
> come through in the NEP text. My apologies that I haven't read *all* the
> references: I understand that it is frustrating if the above are addressed
> there, but I think it's important to have this kind of context in the NEP
> itself.
>
> Thank you again, and I hope the above is helpful rather than feels like
> more unnecessary churn.
>
Thanks Juan, this feedbac
