Re: [Python-ideas] PEP 563 and expensive backwards compatibility

[Lukasz Langa]

> On Sep 13, 2017, at 9:44 PM, Nick Coghlan  wrote:
> 
> On 14 September 2017 at 09:43, Lukasz Langa  wrote:
>>> On Sep 13, 2017, at 6:37 PM, Nick Coghlan  wrote:
>>> That way, during the "from __future__ import lazy_annotations" period,
>>> folks will have clearer guidance on how to explicitly opt-in to eager
>>> evaluation via function and class decorators.
>> 
>> I like this idea! For classes it would have to be a function that you call 
>> post factum. The way class decorators are implemented, they cannot evaluate 
>> annotations that contain forward references. For example:
>> 
>> class Tree:
>>left: Tree
>>right: Tree
>> 
>>def __init__(self, left: Tree, right: Tree):
>>self.left = left
>>self.right = right
>> 
>> This is true today, get_type_hints() called from within a class decorator 
>> will fail on this class. However, a function performing postponed evaluation 
>> can do this without issue. If a class decorator knew what name a class is 
>> about to get, that would help. But that's a different PEP and I'm not 
>> writing that one ;-)
> 
> The class decorator case is indeed a bit more complicated, but there
> are a few tricks available to create a forward-reference friendly
> evaluation environment.

Using cls.__name__ and the ChainMap is clever, I like it. It might prove useful 
for Eric's data classes later. However, there's more to forward references than 
self-references:

class A:
b: B

class B:
...

In this scenario evaluation of A's annotations has to happen after the module 
is fully loaded. This is the general case. No magic decorator will solve this.

The general solution is running eval() later, when the namespace is fully 
populated. I do agree with you that a default implementation of a 
typing-agnostic variant of `get_type_hints()` would be nice. If anything, 
implementing this might better surface limitations of postponed annotations. 
That function won't be recursive though as your example. And I'll leave 
converting the function to a decorator as an exercise for the reader, 
especially given the forward referencing caveats.

- Ł



signature.asc
Description: Message signed with OpenPGP
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 563 and expensive backwards compatibility

[Nick Coghlan]

On 14 September 2017 at 09:43, Lukasz Langa  wrote:
>> On Sep 13, 2017, at 6:37 PM, Nick Coghlan  wrote:
>> That way, during the "from __future__ import lazy_annotations" period,
>> folks will have clearer guidance on how to explicitly opt-in to eager
>> evaluation via function and class decorators.
>
> I like this idea! For classes it would have to be a function that you call 
> post factum. The way class decorators are implemented, they cannot evaluate 
> annotations that contain forward references. For example:
>
> class Tree:
> left: Tree
> right: Tree
>
> def __init__(self, left: Tree, right: Tree):
> self.left = left
> self.right = right
>
> This is true today, get_type_hints() called from within a class decorator 
> will fail on this class. However, a function performing postponed evaluation 
> can do this without issue. If a class decorator knew what name a class is 
> about to get, that would help. But that's a different PEP and I'm not writing 
> that one ;-)

The class decorator case is indeed a bit more complicated, but there
are a few tricks available to create a forward-reference friendly
evaluation environment.

1. To get the right globals namespace, you can do:

global_ns = sys.modules[cls.__module__].__dict__

2. Define the evaluation locals as follows:

local_ns = collections.ChainMap({cls.__name__: cls}, cls.__dict__)

3. Evaluate the variable and method annotations using "eval(expr,
global_ns, local_ns)"

If you make the eager annotation evaluation recursive (so the
decorator can be applied to the outermost class, but also affects all
inner class definitions), then it would even be sufficient to allow
nested classes to refer to both the outer class as well as other inner
classes (regardless of definition order).

To prevent inadvertent eager evaluation of annotations on functions
and classes that are merely referenced from a class attribute, the
recursive descent would need to be conditional on "attr.__qualname__
== cls.__qualname__ + '.' + attr.__name__".

So something like:

def eager_class_annotations(cls):
global_ns = sys.modules[cls.__module__].__dict__
local_ns = collections.ChainMap({cls.__name__: cls}, cls.__dict__)
annotations = cls.__annotations__
for k, v in annotations.items():
annotations[k] = eval(v, global_ns, local_ns)
for attr in cls.__dict__.values():
name = getattr(attr, "__name__", None)
if name is None:
continue
qualname = getattr(attr, "__qualname__", None)
if qualname is None:
continue
if qualname != f"{cls.__qualname}.{name}":
continue
if isinstance(attr, type):
eager_class_annotations(attr)
else:
eager_annotations(attr)
return cls

You could also hide the difference between eager annotation evaluation
on a class or a function inside a single decorator:

def eager_annotations(obj):
if isinstance(obj, type):
_eval_class_annotations(obj) # Class
elif hasattr(obj, "__globals__"):
_eval_annotations(obj, obj.__globals__) # Function
else:
_eval_annotations(obj, obj.__dict__) # Module
return obj

Given the complexity of the class decorator variant, I now think it
would actually make sense for the PEP to propose *providing* these
decorators somewhere in the standard library (the lower level "types"
module seems like a reasonable candidate, but we've historically
avoided having that depend on the full collections module)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 563 and expensive backwards compatibility

[Lukasz Langa]

> On Sep 13, 2017, at 6:37 PM, Nick Coghlan  wrote:
> 
> I think it would be useful for the PEP to include a definition of an
> "eager annotations" decorator that did something like:
> 
>def eager_annotations(f):
>ns = f.__globals__
>annotations = f.__annotations__
>for k, v in annotations.items():
>annotations[k] = eval(v, ns)
>return f
> 
> And pointed out that you can create variants of that which also pass
> in the locals() namespace (or use sys._getframes() to access it
> dynamically).
> 
> That way, during the "from __future__ import lazy_annotations" period,
> folks will have clearer guidance on how to explicitly opt-in to eager
> evaluation via function and class decorators.

I like this idea! For classes it would have to be a function that you call post 
factum. The way class decorators are implemented, they cannot evaluate 
annotations that contain forward references. For example:

class Tree:
left: Tree
right: Tree

def __init__(self, left: Tree, right: Tree):
self.left = left
self.right = right

This is true today, get_type_hints() called from within a class decorator will 
fail on this class. However, a function performing postponed evaluation can do 
this without issue. If a class decorator knew what name a class is about to 
get, that would help. But that's a different PEP and I'm not writing that one 
;-)

- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 563 and expensive backwards compatibility

[Nick Coghlan]

On 14 September 2017 at 06:01, Jim J. Jewett  wrote:
> The "right time" is whenever they are currently evaluated.
> (Definition time, I think, but won't swear.)
>
> For example, the "annotation" might really be a call to a logger,
> showing the current environment, including names that will be rebound
> before the module finishes loading.
>
> I'm perfectly willing to agree that even needing this much control
> over timing is a code smell, but it is currently possible, and I would
> rather it not become impossible.
>
> At a minimum, it seems like "just run this typing function that you
> should already be using" should either save the right context, or the
> PEP should state explicitly that this functionality is being
> withdrawn.  (And go ahead and suggest a workaround, such as running
> the code before the method definition, or as a decorator.)

I think it would be useful for the PEP to include a definition of an
"eager annotations" decorator that did something like:

def eager_annotations(f):
ns = f.__globals__
annotations = f.__annotations__
for k, v in annotations.items():
annotations[k] = eval(v, ns)
return f

And pointed out that you can create variants of that which also pass
in the locals() namespace (or use sys._getframes() to access it
dynamically).

That way, during the "from __future__ import lazy_annotations" period,
folks will have clearer guidance on how to explicitly opt-in to eager
evaluation via function and class decorators.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 563 and expensive backwards compatibility

[Jelle Zijlstra]

2017-09-13 13:01 GMT-07:00 Jim J. Jewett :

> On Wed, Sep 13, 2017 at 3:12 PM, Lukasz Langa  wrote:
> > On Sep 13, 2017, at 2:56 PM, Jim J. Jewett  wrote:
>
> >> I am generally supportive of leaving the type annotations
> >> unprocessed by default, but there are use cases where
> >> they should be processed (and even cases where doing it
> >> at the right time matters, because of a side effect).
>
> > What is the "right time" you're speaking of?
>
> The "right time" is whenever they are currently evaluated.
> (Definition time, I think, but won't swear.)
>
> For example, the "annotation" might really be a call to a logger,
> showing the current environment, including names that will be rebound
> before the module finishes loading.
>
> I'm perfectly willing to agree that even needing this much control
> over timing is a code smell, but it is currently possible, and I would
> rather it not become impossible.
>

Is this just a theoretical concern? Unless there is significant real-world
code doing this sort of thing, I don't see much of a problem in deprecating
such code using the normal __future__-based deprecation cycle.


>
> At a minimum, it seems like "just run this typing function that you
> should already be using" should either save the right context, or the
> PEP should state explicitly that this functionality is being
> withdrawn.  (And go ahead and suggest a workaround, such as running
> the code before the method definition, or as a decorator.)
>
>
> >> (1)  The PEP suggests opting out with @typing.no_type_hints ...
>
> > This is already possible. PEP 484 specifies that
>
> > "A # type: ignore comment on a line by itself is equivalent to adding an
> > inline # type: ignore to each line until the end of the current indented
> > block. At top indentation level this has effect of disabling type
> checking
> > until the end of file."
>
> Great!  Please mention this as well as (or perhaps instead of)
> typing.no_type_check.
>
>
> >> It would be a bit messy (like the old coding cookie),
> >> but recognizing a module-wide
>
> >> # typing.no_type_hints
>
> >> comment and then falling back to the current behavior
> >> would be enough for me.
>
> > Do you know of any other per-module feature toggle of this kind?
>
> No, thus the comment about it being messy.  But it does offer one way
> to ensure that annotations are evaluated within the proper
> environment, even without having to save those environments.
>
> -jJ
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 563 and expensive backwards compatibility

[Jim J. Jewett]

On Wed, Sep 13, 2017 at 3:12 PM, Lukasz Langa  wrote:
> On Sep 13, 2017, at 2:56 PM, Jim J. Jewett  wrote:

>> I am generally supportive of leaving the type annotations
>> unprocessed by default, but there are use cases where
>> they should be processed (and even cases where doing it
>> at the right time matters, because of a side effect).

> What is the "right time" you're speaking of?

The "right time" is whenever they are currently evaluated.
(Definition time, I think, but won't swear.)

For example, the "annotation" might really be a call to a logger,
showing the current environment, including names that will be rebound
before the module finishes loading.

I'm perfectly willing to agree that even needing this much control
over timing is a code smell, but it is currently possible, and I would
rather it not become impossible.

At a minimum, it seems like "just run this typing function that you
should already be using" should either save the right context, or the
PEP should state explicitly that this functionality is being
withdrawn.  (And go ahead and suggest a workaround, such as running
the code before the method definition, or as a decorator.)


>> (1)  The PEP suggests opting out with @typing.no_type_hints ...

> This is already possible. PEP 484 specifies that

> "A # type: ignore comment on a line by itself is equivalent to adding an
> inline # type: ignore to each line until the end of the current indented
> block. At top indentation level this has effect of disabling type checking
> until the end of file."

Great!  Please mention this as well as (or perhaps instead of)
typing.no_type_check.


>> It would be a bit messy (like the old coding cookie),
>> but recognizing a module-wide

>> # typing.no_type_hints

>> comment and then falling back to the current behavior
>> would be enough for me.

> Do you know of any other per-module feature toggle of this kind?

No, thus the comment about it being messy.  But it does offer one way
to ensure that annotations are evaluated within the proper
environment, even without having to save those environments.

-jJ
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 563 and expensive backwards compatibility

[Lukasz Langa]

> On Sep 13, 2017, at 2:56 PM, Jim J. Jewett  wrote:
> 
> I am generally supportive of leaving the type annotations unprocessed
> by default, but there are use cases where they should be processed
> (and even cases where doing it at the right time matters, because of a
> side effect).

What is the "right time" you're speaking of?


> (1)  The PEP suggests opting out with @typing.no_type_hints ... The
> closest I could find was @typing.no_type_check, which has to be called
> on each object.

This was a typo on my part. Yes, no_type_check is what I meant.


> It should be possible to opt out for an entire module, and it should
> be possible to do so *without* first importing typing.
> 
> Telling type checkers to ignore scopes (including modules) with a
> 
> # typing.no_type_hints
> 
> comment would be sufficient for me.

This is already possible. PEP 484 specifies that

"A # type: ignore comment on a line by itself is equivalent to adding an inline 
# type: ignore to each line until the end of the current indented block. At top 
indentation level this has effect of disabling type checking until the end of 
file."


> (2)  Getting the annotations processed (preferably at the currently
> correct time) should also be possible on a module-wide basis, and
> should also not require importing the entire typing apparatus.

Again, what is the "correct time" you're speaking of?


> It would be a bit messy (like the old coding cookie), but recognizing
> a module-wide
> 
> # typing.no_type_hints
> 
> comment and then falling back to the current behavior would be enough for me.

Do you know of any other per-module feature toggle of this kind? __future__ 
imports are not feature toggles, they are timed deprecations.

Finally, the non-typing use cases that you're worried about, what are they? 
From the research I've done, none of the actual use cases in existence would be 
rendered impossible by postponed evaluation. So far the concerns about side 
effects and local scope in annotations aren't supported by any strong evidence 
that this change would be disruptive.

Don't get me wrong, I'm not being dismissive. I just don't think it's 
reasonable to get blocked on potential and obscure use cases that no real world 
code actually employs.

- Ł



signature.asc
Description: Message signed with OpenPGP
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP 563 and expensive backwards compatibility

[Guido van Rossum]

On Wed, Sep 13, 2017 at 11:56 AM, Jim J. Jewett 
wrote:

> It should be possible to opt out for an entire module, and it should
> be possible to do so *without* first importing typing.
>

PEP 484 has a notation for this -- put

  # type: ignore

at the top of your file and the file won't be type-checked. (Before you
test this, mypy doesn't yet support this. But it could.)

IIUC functions and classes will still have an __annotations__ attribute
(except when it would be empty) so even with the __future__ import (or in
Python 4.0) you could still make non-standard use of annotations pretty
easily -- you'd just get a string rather than an object. (And a simple
eval() will turn the string into an object -- the PEP has a lot of extra
caution because currently the evaluation happens in the scope where the
annotation is encountered, but if you don't care about that everything's
easy.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] PEP 563 and expensive backwards compatibility

[Jim J. Jewett]

I am generally supportive of leaving the type annotations unprocessed
by default, but there are use cases where they should be processed
(and even cases where doing it at the right time matters, because of a
side effect).

I am concerned that the backwards compatibility story for non-typing
cases be not just possible, but reasonable.

(1)  The PEP suggests opting out with @typing.no_type_hints ... The
closest I could find was @typing.no_type_check, which has to be called
on each object.

It should be possible to opt out for an entire module, and it should
be possible to do so *without* first importing typing.

Telling type checkers to ignore scopes (including modules) with a

# typing.no_type_hints

comment would be sufficient for me.

If that isn't possible, please at least create a nontyping or
minityping module so that the marker can be imported without the full
overhead of the typing module.

(2)  Getting the annotations processed (preferably at the currently
correct time) should also be possible on a module-wide basis, and
should also not require importing the entire typing apparatus.

It would be a bit messy (like the old coding cookie), but recognizing
a module-wide

# typing.no_type_hints

comment and then falling back to the current behavior would be enough for me.

Alternatively, it would be acceptable to run something like
typing.get_type_hints, if that could be done in a single pass at the
end of the module (callable from both within the module and from
outside) ... but again, such a cleanup function should be in a smaller
module that doesn't require loading all of typing.

-jJ
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/