[issue45438] inspect not capturing type annotations created by __class_getitem__

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I have opened separate issues for different cases and a meta-issue45665 for 
general discussion.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
type:  -> behavior
versions: +Python 3.11, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Guido van Rossum]

Guido van Rossum  added the comment:

OK, OK, I see your point. I'm curious what Łukasz thinks.

--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

See also issue40296.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

> issubclass(x, list[int]) rejects the second argument for reasons explained in 
> the PEP.

1. One problem is that isinstance(x, type) != issubclass(type(x), type)
if x is list[int]. It is unprecedented, I cannot recall any other case
in which isinstance() and issubclas() are inconsistent. Ant it breaks
code because these two expressions are often considered equivalent and
interchangeable in refactoring.

2. Other problem is that isinstance(x, type) is used as a guard before
using x as a type. `isinstance(obj, type) and issubclass(obj,
SomeClass)` is a common idiom, because issubclass() raises an exception
if its first argument is not a type. It is now broken for list[int].


> What other places are there that are broken because of this?
$ find Lib -name '*.py' \! -path '*/test*' -exec egrep 'isinstance.*,
type\)' '{}' + | wc -l
55

In msg403826 I showed few examples from just two files, but there are
tens more potential examples. I'll show them all if I have enough of
spare time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Guido van Rossum]

Guido van Rossum  added the comment:

Can we close this?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[miss-islington]

miss-islington  added the comment:


New changeset 21150c6fa330f80747d698e4b883c7b4801a25bd by Miss Islington (bot) 
in branch '3.9':
bpo-45438: format of inspect.Signature with generic builtins (GH-29212)
https://github.com/python/cpython/commit/21150c6fa330f80747d698e4b883c7b4801a25bd


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[miss-islington]

miss-islington  added the comment:


New changeset ce7a6afb797d2ffde45e9e902516b8437c8f9e31 by Miss Islington (bot) 
in branch '3.10':
bpo-45438: format of inspect.Signature with generic builtins (GH-29212)
https://github.com/python/cpython/commit/ce7a6afb797d2ffde45e9e902516b8437c8f9e31


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[miss-islington]

Change by miss-islington :


--
pull_requests: +27518
pull_request: https://github.com/python/cpython/pull/29254

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 6.0 -> 7.0
pull_requests: +27517
pull_request: https://github.com/python/cpython/pull/29253

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Guido van Rossum]

Guido van Rossum  added the comment:


New changeset d02ffd1b5c0fd8dec6dd2f7e3f2b0cfae48b7899 by Martin Rueckl in 
branch 'main':
bpo-45438: format of inspect.Signature with generic builtins (#29212)
https://github.com/python/cpython/commit/d02ffd1b5c0fd8dec6dd2f7e3f2b0cfae48b7899


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Martin Rueckl]

Martin Rueckl  added the comment:

Just my two cents as a new contributor but long time user:

- isinstance(list[int], type) returning False seems incredibly un-intuitive to 
me. I always see generics (e.g. list without type parameter) as higher kinded 
types, where passing a type argument via [] turns the hkt into a concrete type.

- Backporting to 3.9 and 3.10 should be no issue

- I am not deep enough in pythons type system to judge whether my PR is 
consistent/good. I really just tried out what works and "feels consistent".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Guido van Rossum]

Guido van Rossum  added the comment:

> The issue is that this feature is internally inconsistent (isinstance() is 
> not consistent with issubclass()),

issubclass(x, list[int]) rejects the second argument for reasons explained in 
the PEP. I don't think from that reasoning you can infer that list[int] is not 
a type. Rather the contrary -- if it's not a type the PEP could just have said 
"it's not a type" but it doesn't, it says "it's unacceptable as second arg to 
issubclass()". In fact, the PEP uses language ("instances of parameterized 
collections") that makes me think in the PEP author's mind list[int] *is* a 
type.

> the C implementation of list[int] is not consistent with the Python 
> implementation of List[int],

Consistency with typing was not a goal of the PEP.

> and a lot of code was not tested with this feature,

What other places are there that are broken because of this?

> so we perhaps have a lot of bugs in the stdlib and in the third-party 
> libraries.

Without more evidence this sounds like speculation. *With* evidence you might 
well convince me.

> Are we going to backport changes for making code working with GenericAlias as 
> a type to 3.9 and 3.10?

I'm fine with backporting the fix in the PR, GH-29212.

> If not, we will actually add new features in 3.11 and keep 3.9 and 3.10 
> broken. If yes, these changes can have larger effect than just making 
> isinstance(list[int], type) returning False, and they can break more code.

Are there other fixes you already know about besides GH-29212? That PR seems 
100% beneficial.

> Note also that isinstance(List[int], type) was True before 3.7, and we 
> intentionally made it False in 3.7 (it was required significant rewriting of 
> the typing module and introducing __mro_entries__). Do we want to revert this 
> decision?

I don't recall why we changed that, I thought it was a side effect of making 
the implementation faster, not because of a decision that we didn't want these 
to be treated as types. I looked at inspect.py in 3.6, and it seems its 
formatannotation() has a special case for annotations that come from the typing 
module (like List[int]). I guess nobody thought to have a test for that, so the 
bug in inspect.py slipped by when it was introduced in 3.9 -- we're relying on 
Raymond's keen eye here. But if we had had such a test, and it had alerted us 
to the problem in 3.9 when types.GenericAlias was introduced, I expect that we 
would have fixed it just like martinitus does in his PR.

Anyway, I am willing to be swayed by evidence, but this bug in inspect.py isn't 
enough.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Two years is not so long for a bug. We fixed 8-year and 12-year bugs.

The issue is that this feature is internally inconsistent (isinstance() is not 
consistent with issubclass()), the C implementation of list[int] is not 
consistent with the Python implementation of List[int], and a lot of code was 
not tested with this feature, so we perhaps have a lot of bugs in the stdlib 
and in the third-party libraries. Are we going to backport changes for making 
code working with GenericAlias as a type to 3.9 and 3.10? If not, we will 
actually add new features in 3.11 and keep 3.9 and 3.10 broken. If yes, these 
changes can have larger effect than just making isinstance(list[int], type) 
returning False, and they can break more code.

Note also that isinstance(List[int], type) was True before 3.7, and we 
intentionally made it False in 3.7 (it was required significant rewriting of 
the typing module and introducing __mro_entries__). Do we want to revert this 
decision?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Guido van Rossum]

Guido van Rossum  added the comment:

I really don't recall if we even seriously considered what 
isinstance(list[int], type) should return. PEP 585 doesn't mention it. I 
presume it falls out of the way it's being tested and the way list[int] passes 
most attribute requests on to the origin (i.e., to list).

Given that this has returned True for two releases now I'm very reluctant to 
changing this now. So our best option is pursuing Serhiy's option 1, at least 
partially. The fix for the OP should be simple enough, right?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Ivan Levkivskyi]

Ivan Levkivskyi  added the comment:

> Was it mistake to make isinstance(list[int], type) returning True?

What was the motivation for this? At first glance returning True looks wrong.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

There are two ways to fix the larger issue.

1. Make issubclass(types.GenericAlias, type) returning True, and also make 
isinstance(typing.List[int], type) returning True and 
issubclass(typing._GenericAlias, type) returning True, and analyze every place 
in the interpreter and the stdlib which calls isinstance(..., type) or 
issubclass(..., type) and ensure that they work with types.GenericAlias and 
typing._GenericAlias and their instances as well as with ordinary types, fix 
them and add tests for them. And perhaps do the same for types.UnionType, 
typing._UnionGenericAlias, typing.TypeVar, typing.NewType, etc, etc.

2. Make isinstance(list[int], type) returning False. It would be nice to ad 
also tests for the same cases as in option 1, but it is not so urgent, and I 
expect that in most cases the current behavior which matches the status quo is 
expected.

I tried to implement option 1, but it is just too much places, so it would take 
a large amount of time which I do not have right now. First than invest my time 
in this I want to make sure which option is desirable in long term.

Guido, Ivan, Ken Jin, what would you say?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Martin Rueckl]

Martin Rueckl  added the comment:

I just created a PR and signed the contributor agreement. Waiting for it to 
update :-) Comments welcome!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Martin Rueckl]

Change by Martin Rueckl :


--
keywords: +patch
pull_requests: +27476
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/29212

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Martin Rueckl]

Martin Rueckl  added the comment:

Can confirm for 3.9.7 as well.

--
nosy: +martinitus

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

The cause is that isinstance(list[int], type) returns True. It can cause bugs 
in other parts of the code which test for instance of type. For example:

>>> types.resolve_bases((typing.List[int],))
(, )
>>> types.resolve_bases((list[int],))
(list[int],)

>>> types.prepare_class('A', (int,), {'metaclass': typing.Type[int]})
(typing.Type[int], {}, {})
>>> types.prepare_class('A', (int,), {'metaclass': type[int]})
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/serhiy/py/cpython/Lib/types.py", line 125, in prepare_class
meta = _calculate_meta(meta, bases)
   
  File "/home/serhiy/py/cpython/Lib/types.py", line 139, in _calculate_meta
if issubclass(base_meta, winner):
   ^
TypeError: issubclass() argument 2 cannot be a parameterized generic

>>> @functools.singledispatch
... def g(a): pass
... 
>>> @g.register
... def g2(a: typing.List[int]): pass
... 
Traceback (most recent call last):
  File "", line 2, in 
  File "/home/serhiy/py/cpython/Lib/functools.py", line 863, in register
raise TypeError(

TypeError: Invalid annotation for 'a'. typing.List[int] is not a class.
>>> @g.register(list[int])
... def g2(a): pass
... 
>>> @g.register
... def g3(a: typing.List[int]): pass
... 
Traceback (most recent call last):
  File "", line 2, in 
  File "/home/serhiy/py/cpython/Lib/functools.py", line 863, in register
raise TypeError(

TypeError: Invalid annotation for 'a'. typing.List[int] is not a class.
>>> @g.register
... def g3(a: list[int]): pass
... 

And many other examples, too many to show here.

Was it mistake to make isinstance(list[int], type) returning True?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Guido van Rossum]

Guido van Rossum  added the comment:

Sure. Waiting for your PR. :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Raymond Hettinger]

Raymond Hettinger  added the comment:

It looks like the error is in inspect.formatannotation().

For instances of type, that function incorrectly returns only 
annotation.__qualname__.

Instead, it should return repr(annotation) which would include the args.

=


def formatannotation(annotation, base_module=None):
if getattr(annotation, '__module__', None) == 'typing':
return repr(annotation).replace('typing.', '')
if isinstance(annotation, type):# <== Erroneous case
if annotation.__module__ in ('builtins', base_module):
return annotation.__qualname__
return annotation.__module__+'.'+annotation.__qualname__
return repr(annotation)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Guido van Rossum]

Guido van Rossum  added the comment:

Raymond, the bug must be in the Python code in inspect.py. Could you dig a 
little deeper there? I don't know much about it. Specifically I think the 
problem may just be in the repr() of class Parameter:

>>> inspect.signature(g).parameters['s'] 

>>> inspect.signature(g).parameters['s'].annotation
list[float]
>>>

--
nosy: +kj

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45438] inspect not capturing type annotations created by __class_getitem__

[Raymond Hettinger]

New submission from Raymond Hettinger :

In the example below, __annotations__ is correct but not the corresponding 
Signature object.

---

from typing import List

def f(s: List[float]) -> None: pass

def g(s: list[float]) -> None: pass

>>> inspect.signature(f)
 None>

>>> inspect.signature(g)
 None>

g.__annotations__
{'s': list[float], 'return': None}

--
components: Library (Lib)
messages: 403692
nosy: gvanrossum, levkivskyi, rhettinger
priority: normal
severity: normal
status: open
title: inspect not capturing type annotations created by __class_getitem__
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com