[Numpy-discussion] Type annotations for NumPy
There's been growing interest in supporting PEP-484 style type annotations in NumPy: https://github.com/numpy/numpy/issues/7370 This would allow NumPy users to add type-annotations to their code that uses NumPy, which they could check with mypy, pycharm or pytype. For example: def f(x: np.ndarray) -> np.ndarray: """Identity function on a NumPy array.""" return x Eventually, we could include data types and potentially array shapes as part of the type. This gets quite a bit more complicated, and to do in a really satisfying way would require new features in Python's typing system. To help guide discussion, I wrote a doc describing use-cases and needs for typing array shapes in more detail: https://docs.google.com/document/d/1vpMse4c6DrWH5rq2tQSx3qwP_m_0lyn-Ij4WHqQqRHY Nathaniel Smith and I recently met with group in San Francisco interested in this topic, including several mypy/typeshed developers (Jelle Zijlstra and Ethan Smith). We discussed and came up with a plan for moving forward: 1. Release basic type stubs for numpy.ndarray without dtypes or shapes, as separate "numpy_stubs" package on PyPI per PEP 561. This will let us iterate rapidly on (experimental) type annotations without coupling to NumPy's release cycle. 2. Add support for dtypes in ndarray type-annotations. This might be as simple as writing np.ndarray[np.float64], but will need a decision about appropriate syntax for shape typing to ensure that this is forwards compatible with typing shapes. Note: this will likely require minor changes to NumPy itself, e.g., to add __class_getitem__ per PEP 560. 3. Add support for shapes in ndarray type-annotations, and define a broader standard for typing array shapes. This will require collaboration with type-checker developers on the required typing features (for details, see my doc above). Eventually, this may entail writing a PEP. I'm writing to gauge support for this general plan, and specifically to get support for step 1. Cheers, Stephan ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Type annotations for NumPy
Hi Stephan, A question of perhaps broader scope than what you were asking for, and more out of curiosity than anything else, but can one mix type annotations with others? E.g., in astropy, we have a decorator that looks for units in the annotations (not dissimilar from dtype, I guess). Could one mix annotations or does one have to stick with one purpose? All the best, Marten ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Type annotations for NumPy
On Sat, Nov 25, 2017 at 1:14 AM, Stephan Hoyer wrote: > There's been growing interest in supporting PEP-484 style type annotations > in NumPy: https://github.com/numpy/numpy/issues/7370 > > This would allow NumPy users to add type-annotations to their code that > uses NumPy, which they could check with mypy, pycharm or pytype. For > example: > > def f(x: np.ndarray) -> np.ndarray: > """Identity function on a NumPy array.""" > return x > > Eventually, we could include data types and potentially array shapes as > part of the type. This gets quite a bit more complicated, and to do in a > really satisfying way would require new features in Python's typing system. > To help guide discussion, I wrote a doc describing use-cases and needs for > typing array shapes in more detail: https://docs.google.com/document/d/ > 1vpMse4c6DrWH5rq2tQSx3qwP_m_0lyn-Ij4WHqQqRHY > > Nathaniel Smith and I recently met with group in San Francisco interested > in this topic, including several mypy/typeshed developers (Jelle Zijlstra > and Ethan Smith). We discussed and came up with a plan for moving forward: > 1. Release basic type stubs for numpy.ndarray without dtypes or shapes, as > separate "numpy_stubs" package on PyPI per PEP 561. This will let us > iterate rapidly on (experimental) type annotations without coupling to > NumPy's release cycle. > 2. Add support for dtypes in ndarray type-annotations. This might be as > simple as writing np.ndarray[np.float64], but will need a decision about > appropriate syntax for shape typing to ensure that this is forwards > compatible with typing shapes. Note: this will likely require minor changes > to NumPy itself, e.g., to add __class_getitem__ per PEP 560. > 3. Add support for shapes in ndarray type-annotations, and define a > broader standard for typing array shapes. This will require collaboration > with type-checker developers on the required typing features (for details, > see my doc above). Eventually, this may entail writing a PEP. > > Can you make a case for the usefulness numpy annotations? What benefits to you want to achieve and how will annotation aid in getting there. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Type annotations for NumPy
This is a complete outsider’s perspective but (a) it would be good if NumPy type annotations could include an “array_like” type that allows lists, tuples, etc. (b) I’ve always thought (since PEP561) that it would be cool for type annotations to replace compiler type annotations for e.g. Cython and Numba. Is this in the realm of possibility for the future? Juan. On 26 Nov 2017, 3:54 AM +1100, Charles R Harris , wrote: > > > > On Sat, Nov 25, 2017 at 1:14 AM, Stephan Hoyer wrote: > > > There's been growing interest in supporting PEP-484 style type > > > annotations in NumPy: https://github.com/numpy/numpy/issues/7370 > > > > > > This would allow NumPy users to add type-annotations to their code that > > > uses NumPy, which they could check with mypy, pycharm or pytype. For > > > example: > > > > > > def f(x: np.ndarray) -> np.ndarray: > > > """Identity function on a NumPy array.""" > > > return x > > > > > > Eventually, we could include data types and potentially array shapes as > > > part of the type. This gets quite a bit more complicated, and to do in a > > > really satisfying way would require new features in Python's typing > > > system. To help guide discussion, I wrote a doc describing use-cases and > > > needs for typing array shapes in more detail: > > > https://docs.google.com/document/d/1vpMse4c6DrWH5rq2tQSx3qwP_m_0lyn-Ij4WHqQqRHY > > > > > > Nathaniel Smith and I recently met with group in San Francisco interested > > > in this topic, including several mypy/typeshed developers (Jelle Zijlstra > > > and Ethan Smith). We discussed and came up with a plan for moving forward: > > > 1. Release basic type stubs for numpy.ndarray without dtypes or shapes, > > > as separate "numpy_stubs" package on PyPI per PEP 561. This will let us > > > iterate rapidly on (experimental) type annotations without coupling to > > > NumPy's release cycle. > > > 2. Add support for dtypes in ndarray type-annotations. This might be as > > > simple as writing np.ndarray[np.float64], but will need a decision about > > > appropriate syntax for shape typing to ensure that this is forwards > > > compatible with typing shapes. Note: this will likely require minor > > > changes to NumPy itself, e.g., to add __class_getitem__ per PEP 560. > > > 3. Add support for shapes in ndarray type-annotations, and define a > > > broader standard for typing array shapes. This will require collaboration > > > with type-checker developers on the required typing features (for > > > details, see my doc above). Eventually, this may entail writing a PEP. > > > > > > > Can you make a case for the usefulness numpy annotations? What benefits to > > you want to achieve and how will annotation aid in getting there. > > > > Chuck > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Type annotations for NumPy
On Sat, Nov 25, 2017 at 7:21 AM Marten van Kerkwijk < m.h.vankerkw...@gmail.com> wrote: > A question of perhaps broader scope than what you were asking for, and > more out of curiosity than anything else, but can one mix type > annotations with others? E.g., in astropy, we have a decorator that > looks for units in the annotations (not dissimilar from dtype, I > guess). Could one mix annotations or does one have to stick with one > purpose? > Hi Marten, I took a look at Astropy's units decorator: http://docs.astropy.org/en/stable/api/astropy.units.quantity_input.html Annotations for return values that "coerce" units would be hard to make compatible with typing, because type annotations are used to check programs, not change runtime semantics. But in principle, I think you could even make a physical units library that relies entirely on static type checking for correctness, imposing almost no run-time overhead at all. There are several examples for Haskell: https://wiki.haskell.org/Physical_units I don't see any obvious way to support to mixing of annotations for typing and runtime effects in the same function, though doing so in the same program might be possible. My guess is that the preferred way to do this would be to use decorators for runtime changes to arguments, and keep annotations for typing. The Python community seems to be standardizing on using annotations for typing: https://www.python.org/dev/peps/pep-0563/#non-typing-usage-of-annotations Cheers, Stephan ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Type annotations for NumPy
Can you make a case for the usefulness numpy annotations? What benefits to you want to achieve and how will annotation aid in getting there. 1. Error checking on large codebases with systems like MyPy 2. Hinting and error checking at code-writing time with systems like Jedi "Hey, this function expects a 2-d square array but you just passed in a 3d array with irregular sizes" 3. Supporting systems like the Cython compiler with type information, allowing them to speedup pure-python code without switching to the Cython language On Sat, Nov 25, 2017 at 6:12 PM, Stephan Hoyer wrote: > On Sat, Nov 25, 2017 at 7:21 AM Marten van Kerkwijk < > m.h.vankerkw...@gmail.com> wrote: > >> A question of perhaps broader scope than what you were asking for, and >> more out of curiosity than anything else, but can one mix type >> annotations with others? E.g., in astropy, we have a decorator that >> looks for units in the annotations (not dissimilar from dtype, I >> guess). Could one mix annotations or does one have to stick with one >> purpose? >> > > Hi Marten, > > I took a look at Astropy's units decorator: > http://docs.astropy.org/en/stable/api/astropy.units.quantity_input.html > > Annotations for return values that "coerce" units would be hard to make > compatible with typing, because type annotations are used to check > programs, not change runtime semantics. But in principle, I think you could > even make a physical units library that relies entirely on static type > checking for correctness, imposing almost no run-time overhead at all. > There are several examples for Haskell: > https://wiki.haskell.org/Physical_units > > I don't see any obvious way to support to mixing of annotations for typing > and runtime effects in the same function, though doing so in the same > program might be possible. My guess is that the preferred way to do this > would be to use decorators for runtime changes to arguments, and keep > annotations for typing. The Python community seems to be standardizing on > using annotations for typing: > https://www.python.org/dev/peps/pep-0563/#non-typing-usage-of-annotations > > Cheers, > Stephan > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Type annotations for NumPy
Thoughts on basing this on a more generic Array type rather than the np.ndarray? I can imagine other nd-array libraries (XArray, Tensorflow, Dask.array) wanting to reuse this work. For dask.array in particular we would want to copy this entirely, but we probably can't specify that dask.arrays are np.ndarrays. It would be nice to ensure that the container type was swappable. On Sat, Nov 25, 2017 at 6:31 PM, Matthew Rocklin wrote: > Can you make a case for the usefulness numpy annotations? What benefits to > you want to achieve and how will annotation aid in getting there. > > >1. Error checking on large codebases with systems like MyPy >2. Hinting and error checking at code-writing time with systems like >Jedi "Hey, this function expects a 2-d square array but you just passed in >a 3d array with irregular sizes" >3. Supporting systems like the Cython compiler with type information, >allowing them to speedup pure-python code without switching to the Cython >language > > > > On Sat, Nov 25, 2017 at 6:12 PM, Stephan Hoyer wrote: > >> On Sat, Nov 25, 2017 at 7:21 AM Marten van Kerkwijk < >> m.h.vankerkw...@gmail.com> wrote: >> >>> A question of perhaps broader scope than what you were asking for, and >>> more out of curiosity than anything else, but can one mix type >>> annotations with others? E.g., in astropy, we have a decorator that >>> looks for units in the annotations (not dissimilar from dtype, I >>> guess). Could one mix annotations or does one have to stick with one >>> purpose? >>> >> >> Hi Marten, >> >> I took a look at Astropy's units decorator: >> http://docs.astropy.org/en/stable/api/astropy.units.quantity_input.html >> >> Annotations for return values that "coerce" units would be hard to make >> compatible with typing, because type annotations are used to check >> programs, not change runtime semantics. But in principle, I think you could >> even make a physical units library that relies entirely on static type >> checking for correctness, imposing almost no run-time overhead at all. >> There are several examples for Haskell: >> https://wiki.haskell.org/Physical_units >> >> I don't see any obvious way to support to mixing of annotations for >> typing and runtime effects in the same function, though doing so in the >> same program might be possible. My guess is that the preferred way to do >> this would be to use decorators for runtime changes to arguments, and keep >> annotations for typing. The Python community seems to be standardizing on >> using annotations for typing: >> https://www.python.org/dev/peps/pep-0563/#non-typing-usage-of-annotations >> >> Cheers, >> Stephan >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> >> > ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Type annotations for NumPy
On Sat, Nov 25, 2017 at 3:09 PM, Juan Nunez-Iglesias wrote: > This is a complete outsider’s perspective but > > (a) it would be good if NumPy type annotations could include an “array_like” > type that allows lists, tuples, etc. I'm sure this will exist. > (b) I’ve always thought (since PEP561) that it would be cool for type > annotations to replace compiler type annotations for e.g. Cython and Numba. > Is this in the realm of possibility for the future? It turns out that the PEP 484 type system is *mostly* not useful for this. They're really designed for checking consistency across a large code-base, not for enabling compiler speedups. For example, if you annotate something as an int, that means "this object is a subclass of int". This is enough to let mypy catch your mistake if you accidentally pass in a float instead, but it's not enough to tell you anything at all about the object's behavior -- you could make a wacky int subclass that acts like a string or something. Probably there are some benefits that compilers can get from PEP 484 annotations, but you should think of them as largely an orthogonal thing. -n -- Nathaniel J. Smith -- https://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Type annotations for NumPy
On 26 Nov 2017, 12:27 PM +1100, Nathaniel Smith , wrote: > It turns out that the PEP 484 type system is *mostly* not useful for > this. They're really designed for checking consistency across a large > code-base, not for enabling compiler speedups. For example, if you > annotate something as an int, that means "this object is a subclass of > int". This is enough to let mypy catch your mistake if you > accidentally pass in a float instead, but it's not enough to tell you > anything at all about the object's behavior -- you could make a wacky > int subclass that acts like a string or something. But doesn’t Cython do all kinds of type conversions and implied equivalences that could be applied here? e.g. I’m going to annotate this as int, which might mean whatever in mypy, but if I pass this .py file to a newfangled Cython 0.35 compiler, the compiler will understand this to mean “actually really this is an int”? ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
