[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

[Steven Troxler]

> It would be nice if someone did some work and collected a list of tutorials 
> about type annotations that exist (especially the ones that are  discoverable 
> with a simple Bing query) and ranked them by quality.

I went with Google rather than Bing but here's what I found:
https://gist.github.com/stroxler/ade7977ed07e27448222a468796bc467

I did check a few Bing queries, the results are mostly a bit less helpful, I 
get more discussions of the `type` function and fewer about type annotations. 
But the Real Python tutorial still shows up on the first page most of the time.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AHJCF57LISD7G7DHLSQLKDKGN6FFHT3M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Adding an `inspect` function to return the signature of a type?

[Steven Troxler]

I've been wondering whether it would make sense to have a function in `inspect` 
that returns the signature of a type, rather than the signature of a specific 
callable object.

I'm not attached to any name for such a function, two ideas are 
`inspect.signature_of` or `inspect.signature_of_type`.

## Context: the behavior of `inspect.signature`

The existing `inspect.signature` function specified in PEP 362 [0] returns the 
call signature of a particular object, in other words given
```
class Dog:
def __init__(self, color: str):
self.color = color
def __call__(self, command: str):
print(command)
```
we will get
```
>>> import inspect
>>> inspect.signature(Dog)

>>> inspect.signature(Dog("brown"))

```

## Outline of the idea

In some cases it might be nice to be able to access the signature of an 
instance of `Dog` given the type `Dog`, without actually needing to instantiate 
it. 

Moreover, there are some cases where it isn't even possible to get a callable 
object, but it still might be nice to have access to an `inspect.Signature` 
object:
- protocols cannot be instantiated. This is particularly relevant to getting 
Signatures because of callback protocols [1] 
- typing.Callable types describe a signature, but again can't be instantiated

If if we had a function to operate on types, it would treat normal classes as 
if calling `inspect.signature` on the `__call__` method of an instance.

For callable types we'd want to do something similar, although this brings up 
some issues: 
- the existing Signature requires parameter names, but a `typing.Callable` 
doesn't have any.
  - One option would be to invent names `__0`, `__1,` etc. Otherwise we'd have 
to make the name optional.
- it's not obvious how to support PEP 612 ParamSpecs, and even less obvious how 
to support PEP 646 in full generality

I have some code snippets running through how simple cases might work, as well 
as the edge cases I've thought of thus far:
https://gist.github.com/stroxler/4760cbc5e49df2295cd0b524328b1c73

I'm undecided about whether the edge cases are an indication that this idea 
isn't worth considering.
It's worth noting that most of the edge cases involve PEP 612 / 646 generics 
which are pretty rare.

## Why did I start thinking about this?

In discussions of PEP 677 [2], a few folks suggested using `inspect.Signature` 
as inspiration for the runtime API (which we did).

This got me thinking that it might be nice for code that uses runtime types, 
regardless of what we decide about PEP 677, to have an API that makes accessing 
signature information from a type easier.

I don't personally have much need for this, but
- when I chatted with Carl Meyer from the static python team at Meta he thought 
it could be handy
- I could imagine it being useful for tools like ML frameworks, which I've 
worked on in the past, that interpret type annotations to auto-generate logic

--

[0] PEP 362 Function Signatures https://www.python.org/dev/peps/pep-0362/
[1] Callback protocols 
https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols
[2] PEP 677 Callable Type Syntax  https://www.python.org/dev/peps/pep-0677/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/47NLKQZYYQL6LKA7BQWGFT4UZJOIAR4M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 677 (Callable Type Syntax): Final Call for Comments

[Steven Troxler]

We just merged changes to the PEP with a detailed runtime API specification.

Highlights are that:
- The new type should have an `inspect.Signature`- inspired structured API
- It should have a backward-compatible interface with the `__args__` and 
`__params__`
- As with the PEP 604 union syntax, `__eq__` should treat `typing.Callable` and 
equivalent builtin callable types as equal
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UB6VLDVDSCHWOGEGTPZAOX6BQFVZRHEK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 677 (Callable Type Syntax): Final Call for Comments

[Steven Troxler]

Using an operator is an interesting idea, and we should probably call it out as 
an alternative in the PEP.
It's not a substitute for the current PEP from the standpoint of typing-sig for 
a few reasons:

(A) We care that the syntax be forward-compatible with supporting named 
arguments, as outlined in the "Extended Syntax" section of Rejected 
Alternatives and I don't think an operator-based syntax would extend to that.

(B) We  like supporting `(**P) -> bool` and `(int, **P) -> bool` for PEP 612 
ParamSpec and Concatenate; this makes it much easier to write types for 
decorators, which are a common use case. It would be tricky to do this using an 
arrow operator.

(C) I'd hesitate to add a dunder method on the tuple type, for reasons already 
mentioned here as well as the fact that it would force us to write unary 
callables - which are the most common kind - as `(int,) >> bool`

--

Separate from the question of arrow types, I wonder if an arrow operator would 
be worthwhile; I don't think it would have to be just for typing.

Discussion probably belongs in a separate thread and I'd be happy to kick one 
off, but it sounds like Steve has a potential use case and I've always found 
the use of `>>`, which feels very low-level because it has baggage as a 
bit-shift, in DSLs to be a little weird. Proposing an operator like `=>` or 
`|>` (my preference would be not to use `->`) might make sense.

The DSLs I'm thinking of (coming from data science) are all, loosely speaking, 
about some kind of pipelining
- dplython uses `>>` to mimic R's pipeline operator, which works well for 
certain data transformations
- At least at one point airflow supported creating DAG links using `>>` to 
represent control flow
- As far as I know Tensorflow doesn't currently use `>>`, but their curried API 
that would lend itself to this
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NIYHXMFKLN5C47ZIHBF7M4KSFJLZFDMQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 677 (Callable Type Syntax): Final Call for Comments

[Steven Troxler]

Good catch, thanks Nick!
It's been specified in a linked doc [0] but I forgot to move that into the PEP 
itself.
I'll aim to get that done today.
---
[0] 
https://docs.google.com/document/d/15nmTDA_39Lo-EULQQwdwYx_Q1IYX4dD5WPnHbFG71Lk/edit
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GLRQUR76I4UTH4YBTBNAB54CYEMFRD6W/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 677 (Callable Type Syntax): Final Call for Comments

[Steven Troxler]

Hello all,

We’d like to make a last call for discussion on PEP 677: Callable Type Syntax 
[1] before sending this to the Steering Council.

For context, PEP 677 proposes an arrow-like callable syntax for python:
```
(int, str) -> bool # equivalent to Callable[[int, str], bool]
```

Thanks everyone for comments on our earlier thread [2]. We made some changes to 
the PEP, but not the proposal itself, in response to that discussion:
 - More discussion about readability of return types and parentheses
 - Noting that complex code might be improved by using TypeAliases
 - Discussion of additional ideas in the Rejected Alternatives section

Many of the concerns in the previous thread are about whether new callable 
syntax is worth the implementation complexity, which is an important question. 
We’ve seen demand for this from regular typed Python users. And given that 
Callable is one of the most used and complicated type annotations, we at 
typing-sig favor new syntax.

We have considered the feedback here and in typing-sig and feel that the syntax 
in the PEP is a reasonable tradeoff between usability and simplicity.

One concern raised was that the `async` portion of the proposed syntax may not 
be worthwhile, even if the rest of the proposal is. We considered removing it, 
but several heavy users of async pointed out [3] that:
 - For authors of large, typed async codebases, this syntax would be very 
helpful
 - For everyone else there’s minimal readability cost, because async callables
   will almost never come up.

As a result, we decided to leave it in for now, but would be happy to remove it 
if the SC would prefer that.

If there are no other concerns, we'd like to move forward and submit it to the 
SC.

Cheers,
Steven

---
[1] PEP 677: https://www.python.org/dev/peps/pep-0677/
[2] Previous RFC thread in python-dev: 
https://mail.python.org/archives/list/python-dev@python.org/thread/OGACYN2X7RX2GHAUP2AKRPT6DP432VCN/
 
[3] Discussion in typing-sig about removing `async` keyword: 
https://mail.python.org/archives/list/typing-...@python.org/thread/RFZOSCJHDS66MOVQO6XMVVX43VEKJJBW/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GQB36XDSPIJBAPBQCYVNVPGXS4W2NZA2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Syntax PEP

[Steven Troxler]

In the example I was aiming for something easy to understand that produced a 
type illustrating potential problems of PEP 677, which is at its worst when 
there are callables in both argument and return position. I don't have a great 
real-world example of this worst-case, most of what I've seen involves simpler 
and the current PEP 677 proposal isn't as bad.

As for formatting I agree that I wouldn't hand-write the type as
```
((int) -> float, (str) -> bool) -> (int, str) -> tuple[float, bool]
```
but a lot of code formatters might fight me on this, so I think it's worthy of 
consideration, and I was trying to illustrate a readability problem and 
possible fix. Formatting the code nicely so that my proposal looks good would 
have meant not really engaging with the concern.

If you're looking for examples from real code where Callable is unweildy, 
Pradeep collected a few at [1] from typeshed although most of them look just 
fine with the current PEP 677 proposal. A couple examples:

Callable[[AnyStr, Callable[[AnyStr, AnyStr, AnyStr], AnyStr]], AnyStr]
Callable[[Optional[str], tuple[_Marshallable, ...]], Union[Fault, 
tuple[_Marshallable, ...]]]
Callable[[str, Callable[[Iterable[str], str], str]], None]

versus the same types written using the current PEP 677 syntax:

(AnyStr, (AnyStr, AnyStr, AnyStr) -> AnyStr) -> AnyStr
(Optional[str], tuple[_Marshallable, ...]) -> Union[Fault, tuple[_Marshallable, 
...]]
(str, (Iterable[str], str) -> str) -> None

versus with outer parentheses:

(AnyStr, (AnyStr, AnyStr, AnyStr -> AnyStr) -> AnyStr)
(Optional[str], tuple[_Marshallable, ...] -> Union[Fault, tuple[_Marshallable, 
...]])
(str, (Iterable[str], str -> str) -> None)

and another idea, requiring both outer parentheses and argument parentheses:

((AnyStr, ((AnyStr, AnyStr, AnyStr )-> AnyStr)) -> AnyStr)
((Optional[str], tuple[_Marshallable, ...]) -> Union[Fault, 
tuple[_Marshallable, ...]])
(str, (Iterable[str], str) -> str) -> None)

To me, these are convincing examples of where Callable is hard to read and an 
arrow syntax is easier, but that doesn't necessarily mean it's worth the price 
of new syntax.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KWQVEQDZFBZDQDLOGATXSFN7KSF4WVYJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Syntax PEP

[Steven Troxler]

I've been thinking about readability hard because I share many of your concerns 
about readability.

I'm starting to think parenthesizing the outside might work much better in 
practice:
```
(int, str -> bool)
```
instead of
```
(int, str) -> bool
```

This looks a bit less familiar, but it
- eliminates any situation
- helps the eye group together functions (and lets us use an editor's 
go-to-delimiter if we get lost!)


An example
=

The function


To get at what I mean, here's a nice simple function:
```
def zip(f, g):
def zipped(x: int, y: str):
return f(x), g(y)
return zipped
```



typing.Callable


Here's it's type using typing.Callable:
```
typing.Callable[
[typing.Callable[[int], bool], typing.Callable[[str], float]],
typing.Callable[[int, str], tuple[bool, float]
]
```
which seems ugly. It's actually not bad compared to a lot of production code, 
but this is the kind of thing that led to PEP 677.


current PEP 677 proposal
--


With the current PEP 677 proposal, writing the type on one line we get
```
((int) -> float, (str) -> bool) -> (int, str) -> tuple[float, bool]
```

I prefer this to the Callable version. But it’s dense, and I do agree it 
illustrates some readability issues with PEP 677.



Using (int, str -> bool) syntax

Here’s the type if we change the syntax to put the right parenthesis after the 
return type:
```
((int -> float), (str -> bool) -> (int, str -> tuple[float, bool])
```

To my eyes, most of the pain points are now eliminated.
- there’s never a double-arrow due to callable in return position
- even for argument types, to my eyes it’s now easier to read

An added bonus is we no longer have to think about double-arrows when a 
callable type is in the return position of a function, e.g.:
```
def f() -> (int, str -> bool): ...
```

And it solves another major usability problem we found - writing optional 
callables - because it’s now no problem at all to write
```
(int, str -> bool) | None
```
as the type of an optional callable argument.


New edge cases / readability issues?
-

I’m pretty sure *all* existing edge cases and readability issues are basically 
solved by this syntax because the grouping becomes obvious.

The only new edge case I can see is that we might not like `(-> bool)` for 
parameterless callables. I’d be okay with that, but I’d also be down with a 
hack like `(() -> bool)`.

I don’t actually think it matters much - by making this change we would trade 
multiple edge cases in *complicated* examples for a single edge case in the 
*simplest possible* example where readability isn’t a big problem.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AXRKGCPP2GTDRINQ6NQYM743Y6EH6FO5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Syntax PEP

[Steven Troxler]

In most of our discussions of this idea, we've assumed that we'd adopt the same 
semantics that callback protocols use.

If we do that, then only `lambda a: 3` will type check. In order to type check 
both you'd have to make `a` positional-only:
```
def IntToIntFunc(a: int, /) -> int:
...
```

This is one of the reasons I think functions-as-types could be a great idea but 
not a good substitute for better callable syntax.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6EB54EZXUVFR3HPTGLQK4AGJ7UUR5K4F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Syntax PEP

[Steven Troxler]

Thanks for the feedback. I have a few thoughts.


(1) Concerns about complexity of the syntax make sense to me, it's definitely 
possible to write confusing types with this syntax. Readability would be a good 
reason to reject this idea, but it does cut both ways because `Callable` can be 
hard to read today.

Typing-sig as a whole is confident that an arrow syntax would be an improvement 
but I think the submission process is a good time for wider opinions, I can see 
that it might not be a good idea overall even if it's a better type syntax.


(2) I do like the idea of allowing parameters in a tuple in the existing 
Callable type, that seems like a very clear easy win if we *don't* accept 
callable syntax. I agree that tuple-for-parameters combined with making 
`builtins.callable` subscriptable would address a chunk of the concerns.



I can look into getting more stats on async callables, I don't think they are 
terribly common in most projects, but in certain contexts like async webservers 
they can come up quite a lot.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BXX3IHTGXCBBL6XKSEGK366CFGFDNLHX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] RFC on Callable Syntax PEP

[Steven Troxler]

Hello all,

Thanks everyone for comments on our earlier thread [1] about callable type 
syntax. We now  have a draft PEP [2] proposing an arrow-based syntax for 
callable types, for example:

```
(int, str) -> bool # equivalent to Callable[[int, str], bool]
```

In support of the PEP we also have:
- a reference implementation of the parser [3] to ensure the grammar is correct 
 (tests [5], [6], [7])
- a detailed specification of planned runtime behavior [4], which is not yet in 
the reference implementation

We'd like to get your feedback about the PEP in general, and especially details 
and edge cases we need to consider regarding runtime behavior.

Cheers,
Steven Troxler

-
[1] Earlier python-dev thread 
https://mail.python.org/archives/list/python-dev@python.org/thread/VBHJOS3LOXGVU6I4FABM6DKHH65GGCUB/
[2] PEP 677: https://www.python.org/dev/peps/pep-0677/
[3] Reference implementation of Parser: 
https://github.com/stroxler/cpython/tree/callable-type-syntax--shorthand
[4] Details on the runtime behavior:  
https://docs.google.com/document/d/15nmTDA_39Lo-EULQQwdwYx_Q1IYX4dD5WPnHbFG71Lk/edit

[5] Ast tests for parser changes:
https://github.com/stroxler/cpython/blob/20eb59fdca0d6d8dbe4efa3b04038c7c22024654/Lib/test/test_ast.py#L359-L392
[6] Easy-read tests of examples from the PEP: 
https://github.com/stroxler/cpython/blob/callable-type-syntax--shorthand/Lib/test/test_callable_type_examples_for_pep.py
[7] Test sanity checking hundreds of examples pulled from typeshed:
https://github.com/stroxler/cpython/blob/callable-type-syntax--shorthand/Lib/test/test_callable_type_examples_for_pep.py
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OGACYN2X7RX2GHAUP2AKRPT6DP432VCN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Steven Troxler]

> Is this also why re-using an actual callable at a type was rejected?

The idea to use a function as a type (which Łukasz proposed at the PyCon typing 
summit) is still under consideration. Our thought was that it would probably be 
a separate PEP from a syntax change.

Personally, I think we should both adopt function-as-a-type and shorthand 
syntax, and reject both the XOR and hybrid syntax because function-as-a-type is 
convenient for expressing the same things (named, default-value, and variadic 
args).

We don't think function-as-a-type solves all cases for a few reasons:

(1) in existing code, Pradeep's data analysis shows that the use of named 
arguments in calls is quite rare, less than 5% of cases; positional arguments 
are at least 12 times more common.

As a result, we think it's important to have an option where positional 
arguments are convenient. We don't think the `/` or underscore conventions 
would be very convenient; `/` in particular would be easy to forget, which 
would lead to accidentally pinning argument names.

It's easy to imagine library authors not realizing they did this, particularly 
their unit tests use the same argument names.

(2)  Callable is one of the most heavily used types. So we do think the ability 
to inline it may be important.

(3) Around 20% of Callables in existing code Pradeep analyzed require a 
ParamSpec. That's 4 times the number of callables that made use of default 
arguments or named arguments. 

It is not easy to make use of ParamSpec in function-as-a-type, but it is easy 
to support with a new shorthand syntax.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MGJUXXBYLCZZUX3OTHYDOV5OUPGMEAQD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Steven Troxler]

We drew up a draft syntax to clarify the three styles that have been discussed 
in typing-sig:
- shorthand syntax, which supports exactly what Callable currently supports
- XOR syntax, which would allow a def-like style in addition to shorthand
- hybrid syntax, which extends shorthand to support named, default-value, and 
variadic args

https://docs.google.com/document/d/1oCRBAAPs3efTYoIRSUwWLtOr1AUVqc8OCEYn4vKYjGI/edit?usp=sharing

Within the discussion in typing-sig,
- everyone would like to see support for at least shorthand
- a little over half would like to have the more complex hybrid option
- I don't think anyone has XOR as their first choice, but several of us think 
it might be better than hybrid
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/P5R6P7NLFMCUOJHZL3K2VAQBFZMVGMR6/
Code of Conduct: http://python.org/psf/codeofconduct/