[Numpy-discussion] NEP 42 and physical unit DType

[Lee Johnston]

NEP 42 mentions physical units as a possible use case for the new DType.
Having worked on `unyt`, which is an ndarray subclass, and other unit
system implementations based on the dispatch mechanism, I am quite familiar
with the challenges. One challenge is integration with Matplotlib that has
a unit conversion interface. This interface is invoked by a registry
mapping from unit array type, such as `unyt_array`, and its conversion
class. A custom DType will still result in an ndarray type - correct? If
so, Matplotlib will attempt to perform its internal calculations eventually
leading to an invalid operation when adding a physical quantity to a pure
number. This happens today with `unyt_array` and some of the plotting
functions that allow the subclass to leak through into the internals. Has
this been discussed? I can imagine other libraries will have the same
challenge.
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NEP 42 and physical unit DType

[Sebastian Berg]

On Fri, 2020-10-09 at 07:48 -0500, Lee Johnston wrote:
> NEP 42 mentions physical units as a possible use case for the new
> DType.
> Having worked on `unyt`, which is an ndarray subclass, and other unit
> system implementations based on the dispatch mechanism, I am quite
> familiar
> with the challenges. One challenge is integration with Matplotlib
> that has
> a unit conversion interface. This interface is invoked by a registry
> mapping from unit array type, such as `unyt_array`, and its
> conversion
> class. A custom DType will still result in an ndarray type - correct?

A custom DType will be attached to a normal NumPy array (or, hopefully
a pandas dataframe, etc.). I am not sure if that is what you meant, but
a custom DType will mean there is no array subclass involved (this is
different from how pandas approaches extension dtypes).
But, I am not sure that this makes much of a difference with respect to
the problem.

> If
> so, Matplotlib will attempt to perform its internal calculations
> eventually
> leading to an invalid operation when adding a physical quantity to a
> pure
> number. This happens today with `unyt_array` and some of the plotting

It sounds unlikely that this would solve the issue immediately. Rather,
we probably need to think about creating some kind of "protocol" to
drop units, such as:

   arr.astype(Unitless)

which would be a no-op for the NumPy numerical types. Or baking in
something similar to make working with units easier.

I would have hoped a bit that it is possible to avoid such issues e.g.
by dividing by a step/difference, but I do not know the details.

The most important question for me would be more whether there is
something in the basic infrastructure to consider that cannot be done
as an extension/protocol later (like the `arr.astype(Unitless)` cast,
which should be a straight forward extension).

Although, it would be interesting to figure out how a future Unit DType
would best look in this regard!

Cheers,

Sebastian


> functions that allow the subclass to leak through into the internals.
> Has
> this been discussed? I can imagine other libraries will have the same
> challenge.
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion



signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NEP 42 and physical unit DType

[Thomas Caswell]

>From the Matplotlib side we can definitely do dispatch based on the dtype
of the ndarray (not just that it is an array) which is how we handle arrays
of datetimes.

Matplotlib should (internally) never try to do operations on user data
before we do the unit conversion and my default position is that where we
let that leak is a bug in mpl.  I don't think bugs in Matplotlib
should feed back to the dtype design.

Tom

On Fri, Oct 9, 2020 at 1:08 PM Sebastian Berg 
wrote:

> On Fri, 2020-10-09 at 07:48 -0500, Lee Johnston wrote:
> > NEP 42 mentions physical units as a possible use case for the new
> > DType.
> > Having worked on `unyt`, which is an ndarray subclass, and other unit
> > system implementations based on the dispatch mechanism, I am quite
> > familiar
> > with the challenges. One challenge is integration with Matplotlib
> > that has
> > a unit conversion interface. This interface is invoked by a registry
> > mapping from unit array type, such as `unyt_array`, and its
> > conversion
> > class. A custom DType will still result in an ndarray type - correct?
>
> A custom DType will be attached to a normal NumPy array (or, hopefully
> a pandas dataframe, etc.). I am not sure if that is what you meant, but
> a custom DType will mean there is no array subclass involved (this is
> different from how pandas approaches extension dtypes).
> But, I am not sure that this makes much of a difference with respect to
> the problem.
>
> > If
> > so, Matplotlib will attempt to perform its internal calculations
> > eventually
> > leading to an invalid operation when adding a physical quantity to a
> > pure
> > number. This happens today with `unyt_array` and some of the plotting
>
> It sounds unlikely that this would solve the issue immediately. Rather,
> we probably need to think about creating some kind of "protocol" to
> drop units, such as:
>
>arr.astype(Unitless)
>
> which would be a no-op for the NumPy numerical types. Or baking in
> something similar to make working with units easier.
>
> I would have hoped a bit that it is possible to avoid such issues e.g.
> by dividing by a step/difference, but I do not know the details.
>
> The most important question for me would be more whether there is
> something in the basic infrastructure to consider that cannot be done
> as an extension/protocol later (like the `arr.astype(Unitless)` cast,
> which should be a straight forward extension).
>
> Although, it would be interesting to figure out how a future Unit DType
> would best look in this regard!
>
> Cheers,
>
> Sebastian
>
>
> > functions that allow the subclass to leak through into the internals.
> > Has
> > this been discussed? I can imagine other libraries will have the same
> > challenge.
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>


-- 
Thomas Caswell
tcasw...@gmail.com
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Documentation Team meeting - Monday October 12

[Melissa Mendonça]

Hi all!

This is a reminder that our next Documentation Team meeting will be on *Monday,
October 12* at 3PM UTC**.

If you wish to join on Zoom, **you need to use this NEW link**

https://zoom.us/j/96219574921?pwd=VTRNeGwwOUlrYVNYSENpVVBRRjlkZz09


Here's the permanent hackmd document with the meeting notes (still being
updated in the next few days!):

https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg


Hope to see you around!

** You can click this link to get the correct time at your timezone:
https://www.timeanddate.com/worldclock/fixedtime.html?msg=NumPy+Documentation+Team+Meeting&iso=20201012T15&p1=1440&ah=1

*** You can add the NumPy community calendar to your google calendar by
clicking this link:
https://calendar.google.com/calendar/r?cid=YmVya2VsZXkuZWR1X2lla2dwaWdtMjMyamJobGRzZmIyYzJqODFjQGdyb3VwLmNhbGVuZGFyLmdvb2dsZS5jb20

- Melissa
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion