[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Łukasz Langa]

Łukasz Langa  added the comment:

Thanks! ✨  ✨

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset fa674bdea3bbb20ad6ccd95b3849fc4995bc37e0 by Ken Jin in branch 
'3.9':
[3.9] bpo-41249: Fix postponed annotations for TypedDict (GH-27017) (GH-27205)
https://github.com/python/cpython/commit/fa674bdea3bbb20ad6ccd95b3849fc4995bc37e0


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 480f29f913cff30329e7b425fd6669f83d6d8af8 by Miss Islington (bot) 
in branch '3.10':
bpo-41249: Fix postponed annotations for TypedDict (GH-27017) (#27204)
https://github.com/python/cpython/commit/480f29f913cff30329e7b425fd6669f83d6d8af8


--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Ken Jin]

Change by Ken Jin :


--
pull_requests: +25739
pull_request: https://github.com/python/cpython/pull/27205

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 6.0 -> 7.0
pull_requests: +25738
pull_request: https://github.com/python/cpython/pull/27204

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Guido van Rossum]

Guido van Rossum  added the comment:

Let’s both, since this feels like a real bug fix to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Ken Jin]

Ken Jin  added the comment:

> How far can/should we backport this?

It will work in 3.10 and 3.9 without issues. However, I don't remember if 
bugfixes for __future__ features require special treatment/are excluded from 
normal bugfix backports. I vaguely remember us not backporting from __future__ 
annotations very far back (since they usually broke backwards compatibility).

Maybe 3.10 is enough?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Guido van Rossum]

Guido van Rossum  added the comment:

How far can/should we backport this?

--
versions: +Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Guido van Rossum]

Guido van Rossum  added the comment:


New changeset 889036f7ef7290ef15b6c3373023f6a35387af0c by Germán Méndez Bravo 
in branch 'main':
bpo-41249: Fix postponed annotations for TypedDict (GH-27017)
https://github.com/python/cpython/commit/889036f7ef7290ef15b6c3373023f6a35387af0c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Germán Méndez Bravo]

Germán Méndez Bravo  added the comment:

I added a pull request with my fix here:
https://github.com/python/cpython/pull/27017

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Germán Méndez Bravo]

Change by Germán Méndez Bravo :


--
keywords: +patch
pull_requests: +25576
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/27017

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Germán Méndez Bravo]

Germán Méndez Bravo  added the comment:

Nils, unfortunately, fixing the MRO here won’t fix the issue because 
`TypedDict.__annotations__` in the class copies the annotations from the parent 
classes, and when the type evaluation is made, it’s made using the copied 
annotation found in the bottommost class (which is thus then expected to be a 
forward reference in the same module as the class that inherited them. This 
producing the exact same problem of missing type.

The most likely reason for incomplete MRO is that `TypeDict` extends a class’ 
`__annotations__` with all of it’s parent’s `__annotations__`, so the final 
class has the complete set of annotations of all of its parents, so having the 
full inheritance chain made less sense, after all the final dictionary class 
has all the annotations.

> On Jul 3, 2021, at 13:43, Nils Kattenbeck  wrote:
> 
> 
> Nils Kattenbeck  added the comment:
> 
>> The way I fixed this is I added `__forward_module__` to `typing.ForwardRef`, 
>> so that it can resolve the forward reference with the same globals as the 
>> ones specified by the module in `__forward_module__`. `TypedDict`'s 
>> metaclass should then pass the dictionary’s module name to the annotations’ 
>> forward references via the added `module`’s keyword argument in 
>> `typing._type_check()`. I can work in a pull request with this solution and 
>> discuss any potential problems.
> 
> While this seems like a good solution I would still like to figure out why 
> TypedDict do not preserve MRO. Because for now I have not found a reason nor 
> did someone on the typing-sig mailinglist have a clue. Should there (no 
> longer) be a reason for this then this problem has a trivial solution (just 
> re-add the MRO and use that).
> 
> --
> 
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Nils Kattenbeck]

Nils Kattenbeck  added the comment:

> The way I fixed this is I added `__forward_module__` to `typing.ForwardRef`, 
> so that it can resolve the forward reference with the same globals as the 
> ones specified by the module in `__forward_module__`. `TypedDict`'s metaclass 
> should then pass the dictionary’s module name to the annotations’ forward 
> references via the added `module`’s keyword argument in 
> `typing._type_check()`. I can work in a pull request with this solution and 
> discuss any potential problems.

While this seems like a good solution I would still like to figure out why 
TypedDict do not preserve MRO. Because for now I have not found a reason nor 
did someone on the typing-sig mailinglist have a clue. Should there (no longer) 
be a reason for this then this problem has a trivial solution (just re-add the 
MRO and use that).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Germán Méndez Bravo]

Germán Méndez Bravo  added the comment:

The way I fixed this is I added `__forward_module__` to `typing.ForwardRef`, so 
that it can resolve the forward reference with the same globals as the ones 
specified by the module in `__forward_module__`. `TypedDict`'s metaclass should 
then pass the dictionary’s module name to the annotations’ forward references 
via the added `module`’s keyword argument in `typing._type_check()`. I can work 
in a pull request with this solution and discuss any potential problems.

--
nosy: +Kronuz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Guido van Rossum]

Guido van Rossum  added the comment:

Note that this issue is now only a problem of you use `from __future__ import 
annotations` -- we rolled the default behavior for 3.10 back to what it was in 
3.9.

I am out of time to argue about why we chose this behavior, alas.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Nils Kattenbeck]

Nils Kattenbeck  added the comment:

> I believe it had something to do with TypedDict instances being instances of 
> dict at runtime, but I can't actually reconstruct the reason.

Hm that may be true.
My limited low-level Python knowledge leads me to believe that this could also 
be done using __new__ but I also read that most magic methods get called as 
type(Foo).__magic__(bar, ...) so that might not be possible.
(However also no methods can be declared on a TypedDict class so that might not 
be a problem?)

> Maybe it's written up in PEP 589, but I suspect not (I skimmed and couldn't 
> find it).

I read it completely and could not find anything

> If you ask on typing-sig maybe David Foster (who contributed the initial idea 
> and implementation) remembers.

I asked [here on 
typing-sig](https://mail.python.org/archives/list/typing-...@python.org/thread/RNFWPRLHTUTZES2FDSSMY472JFGMD4EW/)
 but did not yet get any responses.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Ken Jin]

Change by Ken Jin :


--
nosy: +kj

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Guido van Rossum]

Guido van Rossum  added the comment:

> What is/was the initial reason to not preserve the MRO for a TypedDict?

Hm, I can't say for sure. I believe it had something to do with TypedDict 
instances being instances of dict at runtime, but I can't actually reconstruct 
the reason. Maybe it's written up in PEP 589, but I suspect not (I skimmed and 
couldn't find it). If you ask on typing-sig maybe David Foster (who contributed 
the initial idea and implementation) remembers.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Nils Kattenbeck]

Nils Kattenbeck  added the comment:

What is/was the initial reason to not preserve the MRO for a TypedDict?
The only thing which came to my mind would be instantiation performance but as 
annotations are not evaluated by default and on the right-hide side of 
assignment most people will use dict literals I am not sure if this is still 
relevant. Otherwise it might even be simpler to just preserve the original 
bases in TypedDict but please correct me if I overlooked something

--
nosy: +Nils Kattenbeck

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Keith Blaha]

Keith Blaha  added the comment:

> TBH this is not very elegant, but I think you can go ahead with this (at 
> least as a quick fix) since I don't see a better solution yet.

Agreed, given that the current workaround of implementing them in the same 
module works I think I will stick with that while this is brainstormed further.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Ivan Levkivskyi]

Ivan Levkivskyi  added the comment:

> I was thinking that similarly to __required_keys__ and __optional_keys__, the 
> TypedDict could preserve its original bases in a new dunder attribute, and 
> get_type_hints could work off of that instead of MRO when it is dealing with 
> a TypedDict. I would be willing to contribute the PRs to implement this if 
> the design is acceptable

TBH this is not very elegant, but I think you can go ahead with this (at least 
as a quick fix) since I don't see a better solution yet.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Guido van Rossum]

Change by Guido van Rossum :


--
nosy: +gvanrossum, levkivskyi

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41249] TypedDict inheritance doesn't work with get_type_hints and postponed evaluation of annotations across modules

[Keith Blaha]

New submission from Keith Blaha :

Copied from https://github.com/python/typing/issues/737

I came across this issue while using inheritance to express required keys in a 
TypedDict, as is recommended by the docs.

It's most easily explained by a minimal example I cooked up. Let's say we have 
a module foo.py:

from __future__ import annotations

from typing import Optional
from typing_extensions import TypedDict

class Foo(TypedDict):
a: Optional[int]

And another module bar.py:

from __future__ import annotations

from typing import get_type_hints
from foo import Foo

class Bar(Foo, total=False):
b: int

print(get_type_hints(Bar))

Note that both foo.py and bar.py have adopted postponed evaluation of 
annotations (PEP 563) by using the __future__ import.

If we execute bar.py, we get the error message NameError: name 'Optional' is 
not defined.

This is due to the combination of:

get_type_hints relies on the MRO to resolve types: 
https://github.com/python/cpython/blob/3.7/Lib/typing.py#L970
TypedDict does not preserve the original bases, so Foo is not in the MRO 
for Bar:

typing/typing_extensions/src_py3/typing_extensions.py

Line 1652 in d79edde

 tp_dict = super(_TypedDictMeta, cls).__new__(cls, name, (dict,), ns) 

Thus, get_type_hints is unable to resolve the types for annotations that are 
only imported in foo.py.

I ran this example using typing_extensions 3.7.4.2 (released via #709) and 
Python 3.7.3, but it seems like this would be an issue using the current main 
branches of both repositories as well.

I'm wondering what the right approach is to tackling this issue. It is of 
course solvable by defining Bar in foo.py instead, but it isn't ideal or 
intuitive to always need to inherit from a TypedDict in the same module.

I was thinking that similarly to __required_keys__ and __optional_keys__, the 
TypedDict could preserve its original bases in a new dunder attribute, and 
get_type_hints could work off of that instead of MRO when it is dealing with a 
TypedDict. I would be willing to contribute the PRs to implement this if the 
design is acceptable, but am open to other ideas as well.

--
components: Library (Lib)
messages: 373360
nosy: keithblaha
priority: normal
severity: normal
status: open
title: TypedDict inheritance doesn't work with get_type_hints and postponed 
evaluation of annotations across modules
type: behavior
versions: Python 3.10, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com