Sorry it took me 3+ days to reply--I had a lot to think about here. But
I have good things to report!
On 1/11/21 8:42 PM, Guido van Rossum wrote:
On Mon, Jan 11, 2021 at 1:20 PM Larry Hastings <la...@hastings.org
<mailto:la...@hastings.org>> wrote:
PEP 563 states:
For code that uses type hints, the typing.get_type_hints(obj,
globalns=None, localns=None) function correctly evaluates
expressions back from its string form.
So, if you are passing in a localns argument that isn't None,
okay, but you're not using them "correctly" according to the
language. Also, this usage won't be compatible with static type
checkers.
I think you're misreading PEP 563 here. The mention of globalns=None,
localns=None refers to the fact that these parameters have defaults,
not that you must pass None. Note that the next paragraph in that PEP
mentions eval(ann, globals, locals) -- it doesn't say eval(ann, {}, {}).
I think that's misleading, then. The passage is telling you how to
"correctly evaluate[s] expressions", and how I read it was, it's telling
me I have to supply globalns=None and localns=None for it to work
correctly--which, I had to discover on my own, were the default values.
I don't understand why PEP 563 feels compelled to define a function that
it's not introducing, and in fact had already shipped with Python two
versions ago.
Later in that same section, PEP 563 points out a problem with
annotations that reference class-scoped variables, and claims that the
implementation would run into problems because methods can't "see" the
class scope. This is indeed a problem for PEP 563, but *you* can
easily generate correct code, assuming the containing class exists in
the global scope (and your solution requires that anyway). So in this case
```
class Outer:
class Inner:
...
def method(self, a: Inner, b: Outer) -> None:
...
```
The generated code for the `__annotations__` property could just have
a reference to `Outer.Inner` for such cases:
```
def __annotations__():
return {"a": Outer.Inner, "b": Outer, "return": None}
```
This suggestion was a revelation for me. Previously, a combination of
bad experiences early on when hacking on compile and symtable, and my
misunderstanding of exactly what was being asserted in the November 2017
thread, led me to believe that all I could support was globals. But
I've been turning this over in my head for several days now, and I
suspect I can support... just about anything.
I can name five name resolution scenarios I might encounter. I'll
discuss them below, in increasing order of difficulty.
*First* is references to globals / builtins. That's already working,
it's obvious how it works, and I need not elaborate further.
*Second* is local variables in an enclosing function scope:
def outer_fn():
class C: pass
def inner_fn(a:C=None): pass
return inner_fn
As you pointed out elsewhere in un-quoted text, I could make the
annotation a closure, so it could retain a reference to the value of
(what is from its perspective) the free variable "C".
*Third* is local variables in an enclosing class scope, as you describe
above:
class OuterCls:
class InnerCls:
def method(a:InnerCls=None): pass
If I understand what you're suggesting, I could notice inside the
compiler that Inner is being defined in a class scope, walk up the
enclosing scopes until I hit the outermost class, then reconstruct the
chain of pulling out attributes until it resolves globally. Thus I'd
rewrite this example to:
class OuterCls:
class InnerCls:
def method(a:OuterCls.InnerCls=None): pass
We've turned the local reference into a global reference, and we already
know globals work fine.
*Fourth* is local variables in an enclosing class scope, which are
themselves local variables in an enclosing function scope:
def outerfn():
class OuterCls:
class InnerCls:
def method(a:InnerCls=None): pass
return OuterCls.InnerCls
Even this is solvable, I just need to combine the "second" and "third"
approaches above. I walk up the enclosing scopes to find the outermost
class scope, and if that's a function scope, I create a closure and
retain a reference to /that/ free variable. Thus this would turn into
def outerfn():
class OuterCls:
class InnerCls:
def method(a:OuterCls.InnerCls=None): pass
and method.__co_annotations__ would reference the free variable
"OuterCls" defined in outerfn.
*Fifth* is the nasty one. Note that so far every definition we've
referred to in an annotation has been /before/ the definition of the
annotation. What if we want to refer to something defined /after/ the
annotation?
def outerfn():
class OuterCls:
class InnerCls:
def method(a:zebra=None): pass
...
We haven't seen the definition of "zebra" yet, so we don't know what
approach to take. It could be any of the previous four scenarios. What
do we do?
This is solvable too: we simply delay the compilation of
__co_annotations__ code objects until the very last possible moment.
First, at the time we bind the class or function, we generate a stub
__co_annotations__ object, just to give the compiler what it expects.
The compiler inserts it into the const table for the enclosing construct
(function / class / module), and we remember what index it went into.
Then, after we've finished processing the entire AST tree for this
module, but before we we exit the compiler, we reconstruct the required
context for evaluating each __co_annotations__ function--the nested
chain of symbol tables, the compiler blocks if needed, etc--and evaluate
the annotations for real. We assemble the correct __co_annotations__
code object and overwrite the stub in the const table with this
now-correct value.
I can't think of any more scenarios. So, I think I can handle basically
anything!
However, there are two scenarios where the behavior of evaluations will
change in a way the user might find surprising. The first is when they
redefine a variable used in an annotation:
x = str
def fn(a:x="345"): pass
x = int
With stock semantics, the annotation to "a" will be "str". With PEP 563
or my PEP, the annotation to "a" will be "int". (It gets even more
exciting if you said "del x".)
Similarly, delaying the annotations so that we make everything visible
means defining variables with the same name in multiple scopes may lead
to surprising behavior.
x = str
class Outer:
def method(a:x="345"): pass
x = int
Again, stock gets you an annotation of "str", but PEP 563 and my PEP
gets you "str", because they'll see the /final/ result of evaluating the
body of Outer.
Sadly this is the price you pay for delayed evaluation of annotations.
Delaying the evaluation of annotations is the goal, and the whole point
is to make changes, observable by the user, in how annotations are
evaluated. All we can do is document these behaviors and hope our users
forgive us.
I think this is a vast improvement over the first draft of my PEP, and
assuming nobody points out major flaws in this approach (and,
preferably, at least a little encouragement), I plan to redesign my
prototype along these lines. (Though not right away--I want to take a
break and attend to some other projects first.)
Thanks for the mind-blowing suggestions, Guido! I must say, you're
pretty good at this Python stuff.
Cheers,
//arry/
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/5447IRH33FDAAVIEAFIUPLZVNIVOW2EB/
Code of Conduct: http://python.org/psf/codeofconduct/
