Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations (Draft 3)

[Ivan Levkivskyi]

Congratulations, Łukasz!

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations (Draft 3)

[Guido van Rossum]

Łukasz,

I am hereby accepting your PEP. This will be a great improvement in the
experience of users annotating large complex codebases. Congrats on the
design and implementation and on your shepherding the PEP through the
discussion phase. Also a special thanks to Serhiy for thoroughly reviewing
and contributing to the ast-expr-stringification code.

--Guido

PS. I have some editorial quibbles (mostly suggestions to make the
exposition clearer in a few places) but they don't affect acceptance of the
PEP and I will contact you at a later time with these.


On Tue, Nov 21, 2017 at 4:26 PM, Lukasz Langa  wrote:

> Based on the feedback I gather in early November,
> I'm publishing the third draft for consideration on python-dev.
> I hope you like it!
>
> A nicely formatted rendering is available here:
> https://www.python.org/dev/peps/pep-0563/
>
> The full list of changes between this version and the previous draft
> can be found here:
> https://github.com/ambv/static-annotations/compare/
> python-dev1...python-dev2
>
> - Ł
>
>
>
> PEP: 563
> Title: Postponed Evaluation of Annotations
> Version: $Revision$
> Last-Modified: $Date$
> Author: Łukasz Langa 
> Discussions-To: Python-Dev 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 8-Sep-2017
> Python-Version: 3.7
> Post-History: 1-Nov-2017, 21-Nov-2017
> Resolution:
>
>
> Abstract
> 
>
> PEP 3107 introduced syntax for function annotations, but the semantics
> were deliberately left undefined.  PEP 484 introduced a standard meaning
> to annotations: type hints.  PEP 526 defined variable annotations,
> explicitly tying them with the type hinting use case.
>
> This PEP proposes changing function annotations and variable annotations
> so that they are no longer evaluated at function definition time.
> Instead, they are preserved in ``__annotations__`` in string form.
>
> This change is going to be introduced gradually, starting with a new
> ``__future__`` import in Python 3.7.
>
>
> Rationale and Goals
> ===
>
> PEP 3107 added support for arbitrary annotations on parts of a function
> definition.  Just like default values, annotations are evaluated at
> function definition time.  This creates a number of issues for the type
> hinting use case:
>
> * forward references: when a type hint contains names that have not been
>   defined yet, that definition needs to be expressed as a string
>   literal;
>
> * type hints are executed at module import time, which is not
>   computationally free.
>
> Postponing the evaluation of annotations solves both problems.
>
> Non-goals
> -
>
> Just like in PEP 484 and PEP 526, it should be emphasized that **Python
> will remain a dynamically typed language, and the authors have no desire
> to ever make type hints mandatory, even by convention.**
>
> This PEP is meant to solve the problem of forward references in type
> annotations.  There are still cases outside of annotations where
> forward references will require usage of string literals.  Those are
> listed in a later section of this document.
>
> Annotations without forced evaluation enable opportunities to improve
> the syntax of type hints.  This idea will require its own separate PEP
> and is not discussed further in this document.
>
> Non-typing usage of annotations
> ---
>
> While annotations are still available for arbitrary use besides type
> checking, it is worth mentioning that the design of this PEP, as well
> as its precursors (PEP 484 and PEP 526), is predominantly motivated by
> the type hinting use case.
>
> In Python 3.8 PEP 484 will graduate from provisional status.  Other
> enhancements to the Python programming language like PEP 544, PEP 557,
> or PEP 560, are already being built on this basis as they depend on
> type annotations and the ``typing`` module as defined by PEP 484.
> In fact, the reason PEP 484 is staying provisional in Python 3.7 is to
> enable rapid evolution for another release cycle that some of the
> aforementioned enhancements require.
>
> With this in mind, uses for annotations incompatible with the
> aforementioned PEPs should be considered deprecated.
>
>
> Implementation
> ==
>
> In Python 4.0, function and variable annotations will no longer be
> evaluated at definition time.  Instead, a string form will be preserved
> in the respective ``__annotations__`` dictionary.  Static type checkers
> will see no difference in behavior, whereas tools using annotations at
> runtime will have to perform postponed evaluation.
>
> The string form is obtained from the AST during the compilation step,
> which means that the string form might not preserve the exact formatting
> of the source.  Note: if an annotation was a string literal already, it
> will still be wrapped in a string.
>
> Annotations need to be syntactically valid Python expressions, also when
> passed as literal strings (i.e. 

[Python-Dev] PEP 563: Postponed Evaluation of Annotations (Draft 3)

[Lukasz Langa]

Based on the feedback I gather in early November,
I'm publishing the third draft for consideration on python-dev.
I hope you like it!

A nicely formatted rendering is available here:
https://www.python.org/dev/peps/pep-0563/

The full list of changes between this version and the previous draft
can be found here:
https://github.com/ambv/static-annotations/compare/python-dev1...python-dev2

- Ł



PEP: 563
Title: Postponed Evaluation of Annotations
Version: $Revision$
Last-Modified: $Date$
Author: Łukasz Langa 
Discussions-To: Python-Dev 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 8-Sep-2017
Python-Version: 3.7
Post-History: 1-Nov-2017, 21-Nov-2017
Resolution:


Abstract


PEP 3107 introduced syntax for function annotations, but the semantics
were deliberately left undefined.  PEP 484 introduced a standard meaning
to annotations: type hints.  PEP 526 defined variable annotations,
explicitly tying them with the type hinting use case.

This PEP proposes changing function annotations and variable annotations
so that they are no longer evaluated at function definition time.
Instead, they are preserved in ``__annotations__`` in string form.

This change is going to be introduced gradually, starting with a new
``__future__`` import in Python 3.7.


Rationale and Goals
===

PEP 3107 added support for arbitrary annotations on parts of a function
definition.  Just like default values, annotations are evaluated at
function definition time.  This creates a number of issues for the type
hinting use case:

* forward references: when a type hint contains names that have not been
  defined yet, that definition needs to be expressed as a string
  literal;

* type hints are executed at module import time, which is not
  computationally free.

Postponing the evaluation of annotations solves both problems.

Non-goals
-

Just like in PEP 484 and PEP 526, it should be emphasized that **Python
will remain a dynamically typed language, and the authors have no desire
to ever make type hints mandatory, even by convention.**

This PEP is meant to solve the problem of forward references in type
annotations.  There are still cases outside of annotations where
forward references will require usage of string literals.  Those are
listed in a later section of this document.

Annotations without forced evaluation enable opportunities to improve
the syntax of type hints.  This idea will require its own separate PEP
and is not discussed further in this document.

Non-typing usage of annotations
---

While annotations are still available for arbitrary use besides type
checking, it is worth mentioning that the design of this PEP, as well
as its precursors (PEP 484 and PEP 526), is predominantly motivated by
the type hinting use case.

In Python 3.8 PEP 484 will graduate from provisional status.  Other
enhancements to the Python programming language like PEP 544, PEP 557,
or PEP 560, are already being built on this basis as they depend on
type annotations and the ``typing`` module as defined by PEP 484.
In fact, the reason PEP 484 is staying provisional in Python 3.7 is to
enable rapid evolution for another release cycle that some of the
aforementioned enhancements require.

With this in mind, uses for annotations incompatible with the
aforementioned PEPs should be considered deprecated.


Implementation
==

In Python 4.0, function and variable annotations will no longer be
evaluated at definition time.  Instead, a string form will be preserved
in the respective ``__annotations__`` dictionary.  Static type checkers
will see no difference in behavior, whereas tools using annotations at
runtime will have to perform postponed evaluation.

The string form is obtained from the AST during the compilation step,
which means that the string form might not preserve the exact formatting
of the source.  Note: if an annotation was a string literal already, it
will still be wrapped in a string.

Annotations need to be syntactically valid Python expressions, also when
passed as literal strings (i.e. ``compile(literal, '', 'eval')``).
Annotations can only use names present in the module scope as postponed
evaluation using local names is not reliable (with the sole exception of
class-level names resolved by ``typing.get_type_hints()``).

Note that as per PEP 526, local variable annotations are not evaluated
at all since they are not accessible outside of the function's closure.

Enabling the future behavior in Python 3.7
--

The functionality described above can be enabled starting from Python
3.7 using the following special import::

from __future__ import annotations

A reference implementation of this functionality is available
`on GitHub `_.


Resolving Type Hints at Runtime
===

To resolve an 

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Lukasz Langa]

See my response to Mark's e-mail. I agree that special-casing outermost strings 
is not generic enough of a solution and will be removing this special handling 
from the PEP and the implementation.

- Ł

> On Nov 19, 2017, at 2:38 PM, Koos Zevenhoven  wrote:
> 
> On Mon, Nov 13, 2017 at 11:59 PM, Brett Cannon  > wrote:
> ​[..]​
> On Sun, Nov 12, 2017, 10:22 Koos Zevenhoven,  > wrote:​​
> 
> There's two thing I don't understand here:
> 
> * What does it mean to preserve the string verbatim? No matter how I read it, 
> I can't tell if it's with quotes or without.
> 
> Maybe I'm missing some context.
> 
> I believe the string passes through unchanged (i.e. no quotes). Think of the 
> PEP as simply turning all non-string annotations into string ones.
> 
> 
> ​Ok, maybe that was just wishful thinking on my part ;-).
> 
> More info in the other threads, for example:
> 
> https://mail.python.org/pipermail/python-dev/2017-November/150642.html 
> 
> https://mail.python.org/pipermail/python-dev/2017-November/150637.html 
> 
> 
> -- Koos
> 
> 
> --
> + Koos Zevenhoven + http://twitter.com/k7hoven  +
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/lukasz%40langa.pl



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Koos Zevenhoven]

On Mon, Nov 13, 2017 at 11:59 PM, Brett Cannon  wrote:
​[..]​

> On Sun, Nov 12, 2017, 10:22 Koos Zevenhoven,  wrote:
> ​​
>
>>
>> There's two thing I don't understand here:
>>
>> * What does it mean to preserve the string verbatim? No matter how I read
>> it, I can't tell if it's with quotes or without.
>>
>> Maybe I'm missing some context.
>>
>
> I believe the string passes through unchanged (i.e. no quotes). Think of
> the PEP as simply turning all non-string annotations into string ones.
>
>
​Ok, maybe that was just wishful thinking on my part ;-).

More info in the other threads, for example:

https://mail.python.org/pipermail/python-dev/2017-November/150642.html
https://mail.python.org/pipermail/python-dev/2017-November/150637.html

-- Koos


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Brett Cannon]

On Sun, Nov 12, 2017, 10:22 Koos Zevenhoven,  wrote:

> On Nov 12, 2017 19:10, "Guido van Rossum"  wrote:
>
> On Sun, Nov 12, 2017 at 4:14 AM, Koos Zevenhoven 
> wrote:
>
>> So actually my question is: What should happen when the annotation is
>> already a string literal?
>>
>
> The PEP answers that clearly (under Implementation):
>
> > If an annotation was already a string, this string is preserved
> > verbatim.
>
>
> Oh sorry, I was looking for a spec, so I somehow assumed I can ignore the
> gory implementation details just like I routinely ignore things like
> headers and footers of emails.
>
> There's two thing I don't understand here:
>
> * What does it mean to preserve the string verbatim? No matter how I read
> it, I can't tell if it's with quotes or without.
>
> Maybe I'm missing some context.
>

I believe the string passes through unchanged (i.e. no quotes). Think of
the PEP as simply turning all non-string annotations into string ones.

-brett


>
> -- Koos (mobile)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Koos Zevenhoven]

On Nov 12, 2017 19:10, "Guido van Rossum"  wrote:

On Sun, Nov 12, 2017 at 4:14 AM, Koos Zevenhoven  wrote:

> So actually my question is: What should happen when the annotation is
> already a string literal?
>

The PEP answers that clearly (under Implementation):

> If an annotation was already a string, this string is preserved
> verbatim.


Oh sorry, I was looking for a spec, so I somehow assumed I can ignore the
gory implementation details just like I routinely ignore things like
headers and footers of emails.

There's two thing I don't understand here:

* What does it mean to preserve the string verbatim? No matter how I read
it, I can't tell if it's with quotes or without.

Maybe I'm missing some context.


-- Koos (mobile)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Guido van Rossum]

On Sun, Nov 12, 2017 at 4:14 AM, Koos Zevenhoven  wrote:

> So actually my question is: What should happen when the annotation is
> already a string literal?
>

The PEP answers that clearly (under Implementation):

> If an annotation was already a string, this string is preserved
> verbatim.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Koos Zevenhoven]

On Sun, Nov 12, 2017 at 7:07 AM, Guido van Rossum  wrote:

> On Fri, Nov 10, 2017 at 11:02 PM, Nick Coghlan  wrote:
>
>> On 11 November 2017 at 01:48, Guido van Rossum  wrote:
>> > I don't mind the long name. Of all the options so far I really only like
>> > 'string_annotations' so let's go with that.
>>
>> +1 from me.
>>
>
> I'd like to reverse my stance on this. We had `from __future__ import
> division` for many years in Python 2, and nobody argued that it implied
> that Python 2 doesn't have division -- it just meant to import the future
> *version* of division. So I think the original idea, `from __future__
> import annotations` is fine. I don't expect there will be *other* things
> related to annotations that we'll be importing from the future.
>
>
Furthermore, *​nobody* expects the majority of programmers to look at
__annotations__ either. But those who do need to care about the
'implementation detail' of whether it's a string won't be surprised to find
nested strings like "'ForwardReferencedThing'". But one might fear that
those cases get ruthlessly converted into being equivalent to just
"ForwardReferencedThing".

So actually my question is: What should happen when the annotation is
already a string literal?

-- Koos
  ​
-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Guido van Rossum]

On Fri, Nov 10, 2017 at 11:02 PM, Nick Coghlan  wrote:

> On 11 November 2017 at 01:48, Guido van Rossum  wrote:
> > I don't mind the long name. Of all the options so far I really only like
> > 'string_annotations' so let's go with that.
>
> +1 from me.
>

I'd like to reverse my stance on this. We had `from __future__ import
division` for many years in Python 2, and nobody argued that it implied
that Python 2 doesn't have division -- it just meant to import the future
*version* of division. So I think the original idea, `from __future__
import annotations` is fine. I don't expect there will be *other* things
related to annotations that we'll be importing from the future.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 11 November 2017 at 01:48, Guido van Rossum  wrote:
> I don't mind the long name. Of all the options so far I really only like
> 'string_annotations' so let's go with that.

+1 from me.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Peter Ludemann via Python-Dev]

On 10 November 2017 at 19:17, Greg Ewing 
wrote:

> Ethan Furman wrote:
>
>> Contriwise, "annotation_strings" sounds like a different type of
>> annotation -- they are now being stored as strings, instead of something
>> else.
>>
>
> How about "annotations_as_strings"?


That feels unambiguous. "annotations_to_str" is shorter, given that "str"
is a type in Python, and "to" says that it's converting *to* string (it's
given *as* an expression).
​

>
>
> --
> Greg
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/pludemann
> %40google.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Greg Ewing]

Ethan Furman wrote:
Contriwise, "annotation_strings" sounds like a different type of 
annotation -- they are now being stored as strings, instead of something 
else.


How about "annotations_as_strings"?

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Lukasz Langa]

> On 10 Nov, 2017, at 4:48 PM, Guido van Rossum  wrote:
> 
> On Fri, Nov 10, 2017 at 1:20 AM, Lukasz Langa  > wrote:
> Alright, we're on bikeshed territory now. Finally! :-)
> 
> I was always thinking about this as "static annotations". The fact they're 
> strings at runtime is irrelevant for most people who will use this future. 
> They don't want string annotations, they want them to not be evaluated on 
> import time... they want them to be static. Also, "static typing" et al. I 
> think it has a nice vibe to it.
> 
> I admit "annotations" is too broad but "static_annotations" (or 
> "string_annotations" ¯\_(ツ)_/¯ ) will be the longest __future__ name so far. 
> That was my main motivation behind using the shorter name. And a bit of 
> megalomania I guess.
> 
> I don't mind the long name. Of all the options so far I really only like 
> 'string_annotations' so let's go with that.

Done.

- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Guido van Rossum]

On Fri, Nov 10, 2017 at 9:50 AM, Ethan Furman  wrote:

> On 11/10/2017 07:48 AM, Guido van Rossum wrote:
>
> I don't mind the long name. Of all the options so far I really only like
>> 'string_annotations' so let's go with that.
>>
>
> As someone else mentioned, we have function annotations and variable
> annotations already, which makes string_annotations sound like it's
> annotations for strings.
>

We can't strive to encode the full documentation in the future's name.


> Contriwise, "annotation_strings" sounds like a different type of
> annotation -- they are now being stored as strings, instead of something
> else.
>

Trust me, that can also be misinterpreted. Let's stop the bikeshedding.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Koos Zevenhoven]

On Fri, Nov 10, 2017 at 7:50 PM, Ethan Furman  wrote:

> On 11/10/2017 07:48 AM, Guido van Rossum wrote:
>
> I don't mind the long name. Of all the options so far I really only like
>> 'string_annotations' so let's go with that.
>>
>
> As someone else mentioned, we have function annotations and variable
> annotations already, which makes string_annotations sound like it's
> annotations for strings.
>
>
> Contriwise, "annotation_strings" sounds like a different type of
> annotation -- they are now being stored as strings, instead of something
> else.
>
>
​Or a step further (longer), with annotations_as_strings.

––Koos​


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Ethan Furman]

On 11/10/2017 07:48 AM, Guido van Rossum wrote:


I don't mind the long name. Of all the options so far I really only like 
'string_annotations' so let's go with that.


As someone else mentioned, we have function annotations and variable annotations already, which makes string_annotations 
sound like it's annotations for strings.



Contriwise, "annotation_strings" sounds like a different type of annotation -- they are now being stored as strings, 
instead of something else.


--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Guido van Rossum]

On Fri, Nov 10, 2017 at 1:20 AM, Lukasz Langa  wrote:

> Alright, we're on bikeshed territory now. Finally! :-)
>
> I was always thinking about this as "static annotations". The fact they're
> strings at runtime is irrelevant for most people who will use this future.
> They don't want string annotations, they want them to not be evaluated on
> import time... they want them to be static. Also, "static typing" et al. I
> think it has a nice vibe to it.
>
> I admit "annotations" is too broad but "static_annotations" (or
> "string_annotations" ¯\_(ツ)_/¯ ) will be the longest __future__ name so
> far. That was my main motivation behind using the shorter name. And a bit
> of megalomania I guess.
>

I don't mind the long name. Of all the options so far I really only like
'string_annotations' so let's go with that.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Koos Zevenhoven]

On Thu, Nov 9, 2017 at 9:51 PM, Guido van Rossum  wrote:

> If we have to change the name I'd vote for string_annotations -- "lazy"
> has too many other connotations (e.g. it might cause people to think it's
> the thunks). I find str_annotations too abbreviated, and
> stringify_annotations is too hard to spell.
>
>
​I can't say I disagree. ​And maybe importing string_annotations from the
__future__ doesn't sound quite as sad as importing something from the
__past__.

Anyway, it's not obvious to me that it is the module author that should
decide how the annotations are handled. See also this quote below:

(Quoted from the end of
https://mail.python.org/pipermail/python-ideas/2017-October/047311.html )

On Thu, Oct 12, 2017 at 3:59 PM, Koos Zevenhoven  wrote:

>
> ​​[*] Maybe somehow make the existing functionality a phantom easter
> egg––a blast from the past which you can import and use, but which is
> otherwise invisible :-). Then later give warnings and finally remove it
> completely.
>
> But we need better smooth upgrade paths anyway, maybe something like:
>
> from __compat__ import unintuitive_decimal_contexts
>
> with unintuitive_decimal_contexts:
> do_stuff()
>
> ​Now code bases can more quickly switch to new python versions and make
> the occasional compatibility adjustments more lazily, while already
> benefiting from other new language features.
>
>
> ––Koos​
>
>
>
-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 10 November 2017 at 19:20, Lukasz Langa  wrote:
> Alright, we're on bikeshed territory now. Finally! :-)
>
> I was always thinking about this as "static annotations". The fact they're
> strings at runtime is irrelevant for most people who will use this future.

It's highly relevant to anyone currently using annotations for a
purpose *other than* type hints, though - they're going to have to
work out how to cope with annotations being strings rather than
eagerly evaluated expressions.

It's also a hopefully useful mnemonic as to what the new runtime
semantics are: the feature flag makes it as if all your annotations
were quoted strings, just without the actual quote markers.

> They don't want string annotations, they want them to not be evaluated on
> import time... they want them to be static. Also, "static typing" et al. I
> think it has a nice vibe to it.

Getting folks to *not* call type hinting static typing is an ongoing
challenge though, so it doesn't seem like a good idea to encourage
that link to me.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Lukasz Langa]

Alright, we're on bikeshed territory now. Finally! :-)

I was always thinking about this as "static annotations". The fact they're 
strings at runtime is irrelevant for most people who will use this future. They 
don't want string annotations, they want them to not be evaluated on import 
time... they want them to be static. Also, "static typing" et al. I think it 
has a nice vibe to it.

I admit "annotations" is too broad but "static_annotations" (or 
"string_annotations" ¯\_(ツ)_/¯ ) will be the longest __future__ name so far. 
That was my main motivation behind using the shorter name. And a bit of 
megalomania I guess.

- Ł

> On 9 Nov, 2017, at 7:30 PM, Guido van Rossum  wrote:
> 
> So... Łukasz?
> 
> On Thu, Nov 9, 2017 at 6:11 PM, Nick Coghlan  > wrote:
> On 10 November 2017 at 05:51, Guido van Rossum  > wrote:
> > If we have to change the name I'd vote for string_annotations -- "lazy" has
> > too many other connotations (e.g. it might cause people to think it's the
> > thunks). I find str_annotations too abbreviated, and stringify_annotations
> > is too hard to spell.
> 
> Aye, I'd be fine with "from __future__ import string_annotations" -
> that's even more explicitly self-documenting than either of my
> suggestions.
> 
> --
> --Guido van Rossum (python.org/~guido )



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 10 November 2017 at 16:42, Victor Stinner  wrote:
> I didn't follow the discussion on the PEP but I was surprised to read "from
> __future__ import annotations" in an example. Annotations exist since Python
> 3.0, why would Python 3.7 require a future for them? Well, I was aware of
> the PEP, but I was confused anyway.
>
> I really prefer "from __future__ import string_annotations" !

At risk of complicating matters, I now see that this could be read as
"annotations on strings", just as variable annotations are annotations
on variable names, and function annotations are annotations on
functions.

If we decide we care about that possible misreading, then an
alternative would be to swap the word order and use "from __future__
import annotation_strings".

Cheers,
Nick.

P.S. I don't think this really matters either way, it just struck me
that the reversed order might be marginally clearer, so it seemed
worthwhile to mention it.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Victor Stinner]

I didn't follow the discussion on the PEP but I was surprised to read "from
__future__ import annotations" in an example. Annotations exist since
Python 3.0, why would Python 3.7 require a future for them? Well, I was
aware of the PEP, but I was confused anyway.

I really prefer "from __future__ import string_annotations" !

Victor

Le 10 nov. 2017 03:14, "Nick Coghlan"  a écrit :

> On 10 November 2017 at 05:51, Guido van Rossum  wrote:
> > If we have to change the name I'd vote for string_annotations -- "lazy"
> has
> > too many other connotations (e.g. it might cause people to think it's the
> > thunks). I find str_annotations too abbreviated, and
> stringify_annotations
> > is too hard to spell.
>
> Aye, I'd be fine with "from __future__ import string_annotations" -
> that's even more explicitly self-documenting than either of my
> suggestions.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> victor.stinner%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Terry Reedy]

On 11/9/2017 9:11 PM, Nick Coghlan wrote:

On 10 November 2017 at 05:51, Guido van Rossum  wrote:

If we have to change the name I'd vote for string_annotations -- "lazy" has
too many other connotations (e.g. it might cause people to think it's the
thunks). I find str_annotations too abbreviated, and stringify_annotations
is too hard to spell.


Aye, I'd be fine with "from __future__ import string_annotations" -
that's even more explicitly self-documenting than either of my
suggestions.


I think this is the best proposed so far.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Guido van Rossum]

So... Łukasz?

On Thu, Nov 9, 2017 at 6:11 PM, Nick Coghlan  wrote:

> On 10 November 2017 at 05:51, Guido van Rossum  wrote:
> > If we have to change the name I'd vote for string_annotations -- "lazy"
> has
> > too many other connotations (e.g. it might cause people to think it's the
> > thunks). I find str_annotations too abbreviated, and
> stringify_annotations
> > is too hard to spell.
>
> Aye, I'd be fine with "from __future__ import string_annotations" -
> that's even more explicitly self-documenting than either of my
> suggestions.
>

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 10 November 2017 at 05:51, Guido van Rossum  wrote:
> If we have to change the name I'd vote for string_annotations -- "lazy" has
> too many other connotations (e.g. it might cause people to think it's the
> thunks). I find str_annotations too abbreviated, and stringify_annotations
> is too hard to spell.

Aye, I'd be fine with "from __future__ import string_annotations" -
that's even more explicitly self-documenting than either of my
suggestions.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Guido van Rossum]

If we have to change the name I'd vote for string_annotations -- "lazy" has
too many other connotations (e.g. it might cause people to think it's the
thunks). I find str_annotations too abbreviated, and stringify_annotations
is too hard to spell.

On Thu, Nov 9, 2017 at 11:39 AM, Barry Warsaw  wrote:

> On Nov 8, 2017, at 23:57, Nick Coghlan  wrote:
>
> > Putting that quibble first: could we adjust the feature flag to be
> > either "from __future__ import lazy_annotations" or "from __future__
> > import str_annotations"?
> >
> > Every time I see "from __future__ import annotations" I think "But
> > we've had annotations since 3.0, why would they need a future
> > import?".
>
> +1 for lazy_annotations for the same reason.
>
> -Barry
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
> 40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Barry Warsaw]

On Nov 8, 2017, at 23:57, Nick Coghlan  wrote:

> Putting that quibble first: could we adjust the feature flag to be
> either "from __future__ import lazy_annotations" or "from __future__
> import str_annotations"?
> 
> Every time I see "from __future__ import annotations" I think "But
> we've had annotations since 3.0, why would they need a future
> import?".

+1 for lazy_annotations for the same reason.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Greg Ewing]

Guido van Rossum wrote:
I did not assume totally opaque -- but code objects are not very 
introspection friendly (and they have no strong compatibility guarantees).


If I understand the proposal correctly, there wouldn't be any
point in trying to introspect the lambdas/thunks/whatever.
They're only there to provide a level of lazy evaluation.
You would evaluate them and then introspect the returned
data structure.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

TL;DR version: I'm now +1 on a string-based PEP 563, with one
relatively small quibble regarding the future flag's name.

Putting that quibble first: could we adjust the feature flag to be
either "from __future__ import lazy_annotations" or "from __future__
import str_annotations"?

Every time I see "from __future__ import annotations" I think "But
we've had annotations since 3.0, why would they need a future
import?".

Adding the "lazy_" or "str_" prefix makes the feature flag
self-documenting: it isn't the annotations support that's new, it's
the fact the interpreter will avoid evaluating them at runtime by
treating them as implicitly quoted strings at compile time.

See inline comments for clarifications on what I was attempting to
propose in relation to thunks, and more details on why I changed my
mind :)

On 9 November 2017 at 14:16, Guido van Rossum  wrote:
> On Wed, Nov 8, 2017 at 5:49 PM, Nick Coghlan  wrote:
>>
>> On 8 November 2017 at 16:24, Guido van Rossum  wrote:
>> > I also don't like the idea that there's nothing you can do with a thunk
>> > besides calling it -- you can't meaningfully introspect it (not without
>> > building your own bytecode interpreter anyway).
>>
>> Wait, that wasn't what I was suggesting at all - with thunks exposing
>> their code object the same way a function does (i.e. as a `__code__`
>> attribute), the introspection functions in `dis` would still work on
>> them, so you'd be able to look at things like which variable names
>> they referenced, thus granting the caller complete control over *how*
>> they resolved those variable names (by setting them in the local
>> namespace passed to the call).
>
> I understood that they would be translated to `lambda: `. It seems you
> have a slightly more complex idea but if you're suggesting introspection
> through dis, that's too complicated for my taste.

Substituting in a lambda expression wouldn't work for the reasons you
gave when you objected to that idea (there wouldn't be any way for
typing.get_type_hints() to inject "vars(cls)" when evaluating the
annotations for method definitions, and enabling a cell-based
alternative would be a really intrusive change).

>> This is why they'd have interesting potential future use cases as
>> general purpose callbacks - every local, nonlocal, global, and builtin
>> name reference would implicitly be an optional parameter (or a
>> required parameter if the name couldn't be resolved as a nonlocal,
>> global, or builtin).
>
> Yeah, but that's scope creep for PEP 563. Łukasz and I are interested in
> gradually restricting the use of annotations to static typing with an
> optional runtime component. We're not interested in adding different use
> cases. (We're committed to backwards compatibility, but only until 4.0, with
> a clear deprecation path.)

Sorry, that was ambiguous wording on my part: the "potential future
use cases" there related to thunks in general, not their use for
annotations in particular.

APIs like pandas.query are a more meaningful example of where thunks
are potentially useful (and that's a problem I've been intermittently
pondering since Fernando Perez explained it to me at SciPy a few years
back - strings are an OK'ish workaround, but losing syntax
highlighting, precompiled code object caching, and other benefits of
real Python expressions means they *are* a workaround).

>> Instead, thunks would offer all the same introspection features as
>> lambda expressions do, they'd just differ in the following ways:
>>
>> * the parameter list on their code objects would always be empty
>> * the parameter list for their __call__ method would always be "ns=None"
>> * they'd be compiled without CO_OPTIMIZED (the same as a class namespace)
>> * they'd look up their closure references using LOAD_CLASSDEREF (the
>> same as a class namespace)
>
> I don't understand the __call__ with "ns-None" thing but I don't expect it
> matters.

It was an attempted shorthand for the way thunks could handle the
method annotations use case in a way that regular lambda expressions
can't: "thunk(vars(cls))" would be roughly equivalent to
"exec(thunk.__code__, thunk.__globals__, vars(cls))", similar to the
way class body evaluations works like "exec(body.__code__,
body.__globals__, mcl.__prepare__())"

That doesn't make a difference to your decision in relation to PEP 563, though.

>> That leaves the door open to a future PEP that proposes thunk-based
>> annotations as part of proposing thunks as a new low level delayed
>> evaluation primitive.
>
> Sorry, that's not a door I'd like to leave open.

At this point, I'd expect any successful PEP for the thunks idea to
offer far more compelling use cases than type annotations - the key
detail for me is that even if PEP 563 says "Lazy evaluation as strings
means that type annotations do not support lexical closures",
injecting attributes into class namespaces will still offer a way for
devs to emulate 

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Guido van Rossum]

On Wed, Nov 8, 2017 at 5:49 PM, Nick Coghlan  wrote:

> On 8 November 2017 at 16:24, Guido van Rossum  wrote:
> > I also don't like the idea that there's nothing you can do with a thunk
> > besides calling it -- you can't meaningfully introspect it (not without
> > building your own bytecode interpreter anyway).
>
> Wait, that wasn't what I was suggesting at all - with thunks exposing
> their code object the same way a function does (i.e. as a `__code__`
> attribute), the introspection functions in `dis` would still work on
> them, so you'd be able to look at things like which variable names
> they referenced, thus granting the caller complete control over *how*
> they resolved those variable names (by setting them in the local
> namespace passed to the call).
>

I understood that they would be translated to `lambda: `. It seems
you have a slightly more complex idea but if you're suggesting
introspection through dis, that's too complicated for my taste.

This is why they'd have interesting potential future use cases as
> general purpose callbacks - every local, nonlocal, global, and builtin
> name reference would implicitly be an optional parameter (or a
> required parameter if the name couldn't be resolved as a nonlocal,
> global, or builtin).
>

Yeah, but that's scope creep for PEP 563. Łukasz and I are interested in
gradually restricting the use of annotations to static typing with an
optional runtime component. We're not interested in adding different use
cases. (We're committed to backwards compatibility, but only until 4.0,
with a clear deprecation path.)


> > Using an AST instead of a string is also undesirable -- the AST changes
> in
> > each release, and the usual strong compatibility guarantees don't apply
> > here. And how are you going to do anything with it? If you've got a
> string
> > and you want an AST node, it's one call away. But if you've got an AST
> node
> > and you want either a string *or* the object to which that string would
> > evaluate, you've got a lot of work to do. Plus the AST takes up a lot
> more
> > space than the string, and we don't have a way to put an AST in a
> bytecode
> > file. (And as Inada-san pointed out a thunk *also* takes up more space
> than
> > a string.)
> >
> > Nick, please don't try to save the thunk proposal by carefully dissecting
> > every one of my objections. That will just prolong its demise.
>
> Just the one objection, since you seem to be rejecting something I
> didn't suggest (i.e. adding an opaque callable type that the dis and
> inspect modules didn't understand). I agree that would be a bad idea,
> but it's also not what I was suggesting we do.
>

I did not assume totally opaque -- but code objects are not very
introspection friendly (and they have no strong compatibility guarantees).

Instead, thunks would offer all the same introspection features as
> lambda expressions do, they'd just differ in the following ways:
>
> * the parameter list on their code objects would always be empty
> * the parameter list for their __call__ method would always be "ns=None"
> * they'd be compiled without CO_OPTIMIZED (the same as a class namespace)
> * they'd look up their closure references using LOAD_CLASSDEREF (the
> same as a class namespace)
>

I don't understand the __call__ with "ns-None" thing but I don't expect it
matters.


> That said, even without a full-fledged thunk based solution to
> handling lexical scoping I think there's a way to resolve the nested
> class problem in PEP 563 that works for both explicitly and implicitly
> quoted strings, while still leaving the door open to replacing
> implicitly quoted strings with thunks at a later date: stating that
> *if* users want such nested references to be resolvable at runtime,
> they need to inject a runtime reference to the outermost class into
> the inner class namespace.
>
> That is, if you want to take:
>
> class C:
> field = 1
> class D:
> def method(a: C.field):
> ...
>
> and move it inside a function, that would actually look like:
>
> def f():
> class C:
> field = 1
> class D:
> def method(a: C.field):
> ...
> C.D.C = C # Make annotations work at runtime
> return f
>
> That leaves the door open to a future PEP that proposes thunk-based
> annotations as part of proposing thunks as a new low level delayed
> evaluation primitive.
>

Sorry, that's not a door I'd like to leave open.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 8 November 2017 at 16:24, Guido van Rossum  wrote:
> I also don't like the idea that there's nothing you can do with a thunk
> besides calling it -- you can't meaningfully introspect it (not without
> building your own bytecode interpreter anyway).

Wait, that wasn't what I was suggesting at all - with thunks exposing
their code object the same way a function does (i.e. as a `__code__`
attribute), the introspection functions in `dis` would still work on
them, so you'd be able to look at things like which variable names
they referenced, thus granting the caller complete control over *how*
they resolved those variable names (by setting them in the local
namespace passed to the call).

This is why they'd have interesting potential future use cases as
general purpose callbacks - every local, nonlocal, global, and builtin
name reference would implicitly be an optional parameter (or a
required parameter if the name couldn't be resolved as a nonlocal,
global, or builtin).

> Using an AST instead of a string is also undesirable -- the AST changes in
> each release, and the usual strong compatibility guarantees don't apply
> here. And how are you going to do anything with it? If you've got a string
> and you want an AST node, it's one call away. But if you've got an AST node
> and you want either a string *or* the object to which that string would
> evaluate, you've got a lot of work to do. Plus the AST takes up a lot more
> space than the string, and we don't have a way to put an AST in a bytecode
> file. (And as Inada-san pointed out a thunk *also* takes up more space than
> a string.)
>
> Nick, please don't try to save the thunk proposal by carefully dissecting
> every one of my objections. That will just prolong its demise.

Just the one objection, since you seem to be rejecting something I
didn't suggest (i.e. adding an opaque callable type that the dis and
inspect modules didn't understand). I agree that would be a bad idea,
but it's also not what I was suggesting we do.

Instead, thunks would offer all the same introspection features as
lambda expressions do, they'd just differ in the following ways:

* the parameter list on their code objects would always be empty
* the parameter list for their __call__ method would always be "ns=None"
* they'd be compiled without CO_OPTIMIZED (the same as a class namespace)
* they'd look up their closure references using LOAD_CLASSDEREF (the
same as a class namespace)

That said, even without a full-fledged thunk based solution to
handling lexical scoping I think there's a way to resolve the nested
class problem in PEP 563 that works for both explicitly and implicitly
quoted strings, while still leaving the door open to replacing
implicitly quoted strings with thunks at a later date: stating that
*if* users want such nested references to be resolvable at runtime,
they need to inject a runtime reference to the outermost class into
the inner class namespace.

That is, if you want to take:

class C:
field = 1
class D:
def method(a: C.field):
...

and move it inside a function, that would actually look like:

def f():
class C:
field = 1
class D:
def method(a: C.field):
...
C.D.C = C # Make annotations work at runtime
return f

That leaves the door open to a future PEP that proposes thunk-based
annotations as part of proposing thunks as a new low level delayed
evaluation primitive.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Guido van Rossum]

To top off today's set of hopeful thread closures I want to put this one to
rest.

While I'm not ready to accept PEP 563 yet (I'm waiting for the first cut of
the implementation) I *am* hereby nixing the thunk idea.

When I said I was more worried about that than about the stringification
that wasn't about compatibility so much as about conceptual simplicity.
Right now the contents of __annotations__ can be one of two things: an
object created by evaluating an annotation, or a string literal. With
stringification these two possibilities remain the only two possibilities
(and they will still both occur in files that don't use the __future__
import). With thunks, there would be a third option, a 0-argument callable.

I've always disliked APIs (like Django templates, IIRC) that
"transparently" support callables by calling them and treat all other
values as-is. (What if I have a value that just *happens* to be callable?)
I don't want to add such an API.

I also don't like the idea that there's nothing you can do with a thunk
besides calling it -- you can't meaningfully introspect it (not without
building your own bytecode interpreter anyway).

Using an AST instead of a string is also undesirable -- the AST changes in
each release, and the usual strong compatibility guarantees don't apply
here. And how are you going to do anything with it? If you've got a string
and you want an AST node, it's one call away. But if you've got an AST node
and you want either a string *or* the object to which that string would
evaluate, you've got a lot of work to do. Plus the AST takes up a lot more
space than the string, and we don't have a way to put an AST in a bytecode
file. (And as Inada-san pointed out a thunk *also* takes up more space than
a string.)

Nick, please don't try to save the thunk proposal by carefully dissecting
every one of my objections. That will just prolong its demise.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 7 November 2017 at 09:20, Lukasz Langa  wrote:
>
>
>> On Nov 5, 2017, at 11:28 PM, Nick Coghlan  wrote:
>>
>> On 6 November 2017 at 16:36, Lukasz Langa  wrote:
>>
>> - compile annotations like a small nested class body (but returning
>> the expression result, rather than None)
>> - emit MAKE_THUNK instead of the expression's opcodes
>> - emit STORE_ANNOTATION as usual
>>
>
> Is the motivation behind creating thunks vs. reusing lambdas just the 
> difference in handling class-level scope? If so, would it be possible to just 
> modify lambdas to behave thunk-like there? It sounds like this would strictly 
> broaden the functionality of lambdas, in other words, wouldn't create 
> backwards incompatibility for existing code.
>
> Reusing lambdas (with extending them to support class-level scoping) would be 
> a less scary endeavor than introducing a brand new language construct.

I want to say "yes", but it's more "sort of", and at least arguably
"no" (while you'd be using building blocks that already exist inside
the compiler and eval loop, you'd be putting them together in a
slightly new way that's a hybrid of the way lambdas work and the way
class bodies work).

For the code execution part, class creation currently uses
MAKE_FUNCTION (just like lambdas and def statements), and is able to
inject the namespace returned by __prepare__ as the code execution
namespace due to differences in the way the function's code object
gets compiled. These are *exactly* the semantics you'd want for
deferred annotations, but the way they're currently structured
internally is inconvenient for your use case.

Compilation details here:
https://github.com/python/cpython/blob/master/Python/compile.c#L1895
Code execution details here:
https://github.com/python/cpython/blob/master/Python/bltinmodule.c#L167

It's the combination of using COMPILER_SCOPE_CLASS at compilation time
with the direct call to "PyEval_EvalCodeEx" in __build_class__ that
means you can't just treat annotations as a regular lambda function -
lambdas are compiled with CO_OPTIMIZED set, and that means they won't
read local variable values from the provided locals namespace, which
in turn means you wouldn't be able to easily inject "vars(cls)" to
handle the method annotation case.

So that's where the idea of finally adding a "thunk" object came from:
it would be to a class body as a lambda expression is to a function
body, except that instead of relying on a custom call to
PyEval_EvalCodeEx in __build_class__ the way class bodies do, it would
instead define a suitable implementation of tp_call that accepted the
locals namespace to use as a parameter.

The nice part of this approach is that even though it would
technically be a new execution primitive, it's still one with
well-established name resolution semantics: the behaviour we already
use for class bodies.

> With my current understanding I still think stringification is both easier to 
> implement and understand by end users.

No matter how you slice it, PEP 563 *is* defining a new delayed
execution primitive as part of the core language syntax. The question
is whether we define it as a fully compiler integrated primitive, with
clearly specified lexical name resolution semantics that align with
other existing constructs, or something bolted on to the side of the
language without integrating it properly, which we then have to live
with forever.

"This isn't visibly quoted, but it's a string anyway, so the compiler
won't check it for syntax errors" isn't easy to understand. Neither is
the complex set of rules you're proposing for what people will need to
do in order to actually evaluate those strings and turn them back into
runtime objects.

By contrast, "parameter: annotation" can be explained as "it's like
'lambda: expression', but instead of being a regular function with an
explicit parameter list, the annotation is a deferred expression that
accepts a locals namespace to use when called".

> The main usability win of thunks/lambdas is not very significant: evaluating 
> them is as easy as calling them whereas strings require 
> typing.get_type_hints(). I still think being able to access function-local 
> state at time of definition is only theoretically useful.

Your current proposal means that this code will work:

class C:
field = 1
def method(a: C.field):
pass

But this will fail:

def make_class():
class C:
field = 1
def method(a: C.field):
pass

Dropping the "C." prefix would make the single class case work, but
wouldn't help with the nested classes case:

def make_class():
class C:
field = 1
class D:
def method(a: C.field):
pass

Confusingly, though, this would still work:

def make_class():
class C:
field = 1
class D:
field2 = C.field
 

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[INADA Naoki]

As memory footprint and import time point of view, I prefer string to thunk.

We can intern strings, but not lambda.
Dict containing only strings is not tracked by GC,
dict containing lambdas is tracked by GC.
INADA Naoki  


On Tue, Nov 7, 2017 at 8:20 AM, Lukasz Langa  wrote:
>
>
>> On Nov 5, 2017, at 11:28 PM, Nick Coghlan  wrote:
>>
>> On 6 November 2017 at 16:36, Lukasz Langa  wrote:
>>
>> - compile annotations like a small nested class body (but returning
>> the expression result, rather than None)
>> - emit MAKE_THUNK instead of the expression's opcodes
>> - emit STORE_ANNOTATION as usual
>>
>
> Is the motivation behind creating thunks vs. reusing lambdas just the 
> difference in handling class-level scope? If so, would it be possible to just 
> modify lambdas to behave thunk-like there? It sounds like this would strictly 
> broaden the functionality of lambdas, in other words, wouldn't create 
> backwards incompatibility for existing code.
>
> Reusing lambdas (with extending them to support class-level scoping) would be 
> a less scary endeavor than introducing a brand new language construct.
>
> With my current understanding I still think stringification is both easier to 
> implement and understand by end users. The main usability win of 
> thunks/lambdas is not very significant: evaluating them is as easy as calling 
> them whereas strings require typing.get_type_hints(). I still think being 
> able to access function-local state at time of definition is only 
> theoretically useful.
>
> What would be significant though is if thunk/lambdas helped fixing forward 
> references in general. But I can't really see how that could work.
>
> - Ł
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Lukasz Langa]

> On Nov 5, 2017, at 11:28 PM, Nick Coghlan  wrote:
> 
> On 6 November 2017 at 16:36, Lukasz Langa  wrote:
> 
> - compile annotations like a small nested class body (but returning
> the expression result, rather than None)
> - emit MAKE_THUNK instead of the expression's opcodes
> - emit STORE_ANNOTATION as usual
> 

Is the motivation behind creating thunks vs. reusing lambdas just the 
difference in handling class-level scope? If so, would it be possible to just 
modify lambdas to behave thunk-like there? It sounds like this would strictly 
broaden the functionality of lambdas, in other words, wouldn't create backwards 
incompatibility for existing code.

Reusing lambdas (with extending them to support class-level scoping) would be a 
less scary endeavor than introducing a brand new language construct.

With my current understanding I still think stringification is both easier to 
implement and understand by end users. The main usability win of thunks/lambdas 
is not very significant: evaluating them is as easy as calling them whereas 
strings require typing.get_type_hints(). I still think being able to access 
function-local state at time of definition is only theoretically useful.

What would be significant though is if thunk/lambdas helped fixing forward 
references in general. But I can't really see how that could work.

- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Antoine Pitrou]

On Sun, 5 Nov 2017 20:18:07 -0800
Lukasz Langa  wrote:
> 
> Interestingly enough, at Facebook we found out that using f-strings is 
> *faster* at runtime than the lazy form of logging.log("format with %s and 
> %d", arg1, arg2), including for cases when the log message is not emitted.

I suspect this depends on how complex your f-strings are (or the
interpolated data).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 6 November 2017 at 16:36, Lukasz Langa  wrote:
>
> On 5 Nov, 2017, at 9:55 PM, Nick Coghlan  wrote:
>
> Python's name resolution rules are already ridiculously complicated,
> and PEP 563 is proposing to make them *even worse*, purely for the
> sake of an optional feature primarily of interest to large enterprise
> users.
>
>
> Solving forward references in type annotations is one of the two explicit
> goals of the PEP. That alone changes how name resolution works. It sounds
> like you're -1 on that idea alone?

That's just lazy evaluation, and no different from the way name
resolution works for globals and builtins in functions, and for
globals, builtins, and closure references in classes.

So while I do expect that's going to be confusing, it's also entirely
independent of how the lazy evaluation is implemented, and I think the
basic goal of deferring annotation evaluation is a good one.

[snip]

> Since your example that illustrates the problem shows that those things fail
> for regular attribute lookup, too, that can be simply fixed in the PEP. I
> did that here:
> https://github.com/ambv/static-annotations/blob/master/pep-0563.rst#backwards-compatibility

Unfortunately, it isn't quite that trivial to fix. To see the
remaining problem, take your nested class example from the PEP and
move it inside a function.

Today, that makes no difference, since "C" will transparently switch
from being accessed via LOAD_NAME to instead being accessed with
LOAD_CLASSDEREF.

By contrast, without the ability to access the outer "C" class
definition via a closure reference, you'll no longer have a generally
applicable way to evaluate *any* of the annotations that reference it,
since you won't have access to C from the module namespace any more.

I've been persuaded that a nested *function* isn't the right answer
for type annotations (since it doesn't let you tinker with locals()
prior to execution, which in turn means you can't readily allow access
to any class attributes at execution time), but that still leaves the
more exec-friendly logic of class body compilation available.

The difference between this and class body creation is that instead of
passing the compiled code object to MAKE_FUNCTION, and then passing
the resulting function to __build_class__, we'd instead introduce a
new MAKE_THUNK opcode that implemented __call__ differently from the
way regular functions implement it.

That way, the compiler changes would be limited to:

- compile annotations like a small nested class body (but returning
the expression result, rather than None)
- emit MAKE_THUNK instead of the expression's opcodes
- emit STORE_ANNOTATION as usual

>From a name resolution perspective, the new things folks would need to
learn are:

- annotations are now lazily evaluated (just like functions)
- but name resolution works the same way it does in class bodies
(unlike lambda expressions)

To allow typing.get_type_hints() to provide access to attributes
defined in the class, you'd need one final piece of the puzzle:

- rather than accepting regular function arguments (since thunks won't
have parameter lists) thunk.__call__ would instead accept an optional
pre-populated locals() namespace to use

With those changes, blindly calling annotations would usually just
work - the one case that couldn't be handled that way would be
annotations that implicitly accessed class level attrbutes, which
would require passing in "vars(class)" when calling the thunk.

Cheers,
Nick.

P.S. Back when I made the implicit scope change for list
comprehensions, I tried all sorts of potential tweaks to the
compiler's name resolution logic before finally giving up and deciding
that using a real nested function was the only way to make sure I
avoided introducing any weird new edge cases. Lexically nested
closures are generally great, but they make it *really* hard to
emulate Python's name resolution logic without direct assistance from
the compiler at the point where the name reference appears.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Lukasz Langa]

> On 5 Nov, 2017, at 9:55 PM, Nick Coghlan  wrote:
> 
> Python's name resolution rules are already ridiculously complicated,
> and PEP 563 is proposing to make them *even worse*, purely for the
> sake of an optional feature primarily of interest to large enterprise
> users.

Solving forward references in type annotations is one of the two explicit goals 
of the PEP. That alone changes how name resolution works. It sounds like you're 
-1 on that idea alone?


> That's enough to leave nested classes as the main problematic case,
> since they can't see each other's attributes by default, and the
> proposed injected locals semantics in PEP 563 don't get this right
> either (they only account for MRO-based name resolution, not lexical
> nesting, even though the PEP claims the latter is expected to work)


You're right! I went too far here. I meant to be as compatible as possible with 
how regular attribute access works for nested classes. Originally I didn't want 
to support any locals at all for simplicity. I was convinced this is important 
(documented in "Rejected ideas") but overdid the example.

Since your example that illustrates the problem shows that those things fail 
for regular attribute lookup, too, that can be simply fixed in the PEP. I did 
that here:
https://github.com/ambv/static-annotations/blob/master/pep-0563.rst#backwards-compatibility
 


Note that I am still including the "def method(self) -> D.field2:" example as 
valid as including a class' own name in the locals() chain map provided by 
get_type_hints() is going to be trivial. In fact, I think I'll add this to 
get_type_hints() independently of this PEP since users of string literal 
forward references probably don't expect it to fail for nested classes.

- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 6 November 2017 at 14:40, Lukasz Langa  wrote:
> On 4 Nov, 2017, at 6:32 PM, Nick Coghlan  wrote:
>> The only workaround I can see for that breakage is that instead of
>> using strings, we could instead define a new "thunk" type that
>> consists of two things:
>
>> 1. A code object to be run with eval()
>> 2. A dictionary mapping from variable names to closure cells (or None
>> for not yet resolved references to globals and builtins)
>
> This is intriguing.
>
> 1. Would that only be used for type annotations? Any other interesting
> things we could do with them?

Yes, they'd have the potential to replace strings for at least some
data analysis use cases, where passing in lambdas is too awkward
syntactically, since you have to spell out all the parameters.

The pandas.DataFrame.query operation is a reasonable example of that
kind of thing: 
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.query.html
(Not an exact example, since Pandas uses a python-like expression
language, rather than specifically Python)

Right now, folks tend to use strings for this kind of use case, which
has the same performance problem that pre-f-string string formatting
does: it defers the expression parsing and compilation step until
runtime, rather than being able to do it once and then cache the
result in __pycache__.

> 2. It feels to me like that would make annotations *heavier* at runtime
> instead of leaner, since now we're forcing the relevant closures to stay in
> memory.

Cells are pretty cheap (they're just a couple of pointers), and if
they're references to module or class attributes, the object
referenced by the cell would have remained alive regardless.

Even for nonlocal variable references (which a solely string-based
approach would disallow), the referenced objects will already be
getting kept alive anyway by way of the typing machinery.

> 3. This form of lazy evaluation seems pretty implicit to me for the reader.
> Peter Ludemann's example of magic logging.debug() is a case in point here.

One of the biggest advantages though is that just like functions, all
of the necessary logic for doing the delayed evaluation can be
captured in a __call__ method, rather than via elaborate instructions
on how to appropriately invoke eval() based on knowledge of where the
annotation came from.

This is especially important if typing gets taken out of the standard
library: you'll need a replacement for typing.get_type_hints() in PEP
563, and a thunk.__call__() method would be a good spelling for that.

> All in all, unless somebody else is ready to step up and write the PEP on
> this subject (and its implementation) right now, I think this idea will miss
> Python 3.7.

As long as we don't argue for that being an adequate excuse to rush
into "We're using plain strings with ill-defined name resolution
semantics because we couldn't be bothered coming up with a proper
thunk-based design to evaluate", I'd be fine with that. None of this
is urgent, and it's mainly of interest to large organisations that
will see a direct economic benefit from implementing it, so the entire
idea can easily be delayed to 3.8 if they're not prepared to fund a
proper evaluation of the available design options over the next 3
months.

Python's name resolution rules are already ridiculously complicated,
and PEP 563 is proposing to make them *even worse*, purely for the
sake of an optional feature primarily of interest to large enterprise
users. If delayed evaluation of type annotations is deemed important
enough to burden every future Pythonista with learning a second set of
name resolution semantics purely for type annotations, then it's
important enough to postpone implementing it until someone invests the
time in coming up with a competing thunk-based alternative that is
able to rely entirely on the *existing* name resolution semantics.

Exploring that potential thunk-based approach a bit further:

1. We'd continue to eagerly compile annotations (as we do today), but
treat them like a nested class body with a single expression. Unlike
an implicit lambda, this compilation mode will allow the resulting
code object to be used with the two-argument form of the exec builtin
2. That code object would be the main item stored on the thunk object
3. If __classcell__ is defined in the current namespace and names from
the current namespace are referenced, then that can be captured on the
thunk, giving its __call__ method access to any class attributes
needed for name resolution
4. Closure references would be captured automatically, but class
bodies already allow locals to override nonlocals (for compatibility
with pre-populated namespaces returned from __prepare__)
5. A thunk's __globals__ reference would be implicitly captured the
same way it is for a regular function

That's enough to leave nested classes as the main problematic case,
since they can't see each other's attributes by default, and the

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Lukasz Langa]

> On 4 Nov, 2017, at 6:32 PM, Nick Coghlan  wrote:
> 
> The PEP's current attitude towards this is "Yes, it will break, but
> that's OK, because it doesn't matter for the type annotation use case,
> since static analysers will still understand it". Adopting such a
> cavalier approach towards backwards compatibility with behaviour that
> has been supported since Python 3.0 *isn't OK*, since it would mean we
> were taking the step from "type annotations are the primary use case"
> to "Other use cases for function annotations are no longer supported".

Well, this is what the PEP literally says in "Deprecation policy":

> In Python 4.0 this will become the default behavior. Use of annotations 
> incompatible with this PEP is no longer supported.

The rationale here is that type annotations as defined by PEP 484 and others is 
the only notable use case. Note that "type annotations" includes things like 
data classes, auto_attribs in attrs, the dependency injection frameworks 
mentioned before, etc. Those are compatible with PEP 484. So, despite the open 
nature of annotations since Python 3.0, no alternative use case emerged that 
requires eager evaluation and access to local state. PEP 563 is addressing the 
pragmatic issue of improving usability of type annotations, instead of worrying 
about some unknown theoretically possible use case.

While function annotations were open to arbitrary use, typing was increasingly 
hinted (pun not intended) as *the* use case for them:

1. From Day 1, type checking is listed as the number one intended use case in 
PEP 3107 (and most others listed there are essentially type annotations by any 
other name).
2. PEP 484 says "We do hope that type hints will eventually become the sole use 
for annotations", and that "In order for maximal compatibility with offline 
type checking it may eventually be a good idea to change interfaces that rely 
on annotations to switch to a different mechanism, for example a decorator."
3. Variable annotations in PEP 526 were designed with type annotations as the 
sole stated purpose.

PEP 563 simply brings this multi-PEP dance to its logical conclusion, stating 
in "Rationale and Goals" that "uses for annotations incompatible with the 
aforementioned PEPs should be considered deprecated." The timeline for full 
deprecation is Python 4.0.


> The only workaround I can see for that breakage is that instead of
> using strings, we could instead define a new "thunk" type that
> consists of two things:
> 
> 1. A code object to be run with eval()
> 2. A dictionary mapping from variable names to closure cells (or None
> for not yet resolved references to globals and builtins)

This is intriguing.

1. Would that only be used for type annotations? Any other interesting things 
we could do with them?
2. It feels to me like that would make annotations *heavier* at runtime instead 
of leaner, since now we're forcing the relevant closures to stay in memory.
3. This form of lazy evaluation seems pretty implicit to me for the reader. 
Peter Ludemann's example of magic logging.debug() is a case in point here.

All in all, unless somebody else is ready to step up and write the PEP on this 
subject (and its implementation) right now, I think this idea will miss Python 
3.7.


> Now, even without the introduction of the IndirectAttributeCell
> concept, this is amenable to a pretty simple workaround:
> 
> A = Optional[int]
> class C:
>field: A = 1
>def method(self, arg: A) -> None: ...
>C.A = A
>del A

This is a poor workaround, worse in fact than using a string literal as a 
forward reference. This is more verbose and error-prone. Decorators address the 
same construct and their wild popularity suggests that this notation is 
inferior.


> But I genuinely can't see how breaking annotation evaluation at class
> scope can be seen as a deal-breaker for the implicit lambda based
> approach without breaking annotation evaluation for nested functions
> also being seen as a deal-breaker for the string based approach.

The main reason to use type annotations is readability, just like decorators. 
While there's nothing stopping the programmer to write:

class C: ...
def method(self, arg1): ...
method.__annotations__ = {'arg1': str, 'return': int}
C.__annotations__ = {'attribute1': ...}

...this notation doesn't fit the bill. Since nested classes and types embedded 
as class attributes are popular among type hinting users, supporting this case 
is a no-brainer. On the other hand, if you have a factory function that 
generates some class or function, then you either:

1. Use annotations in the generated class/function for type checking; OR
2. Add annotations in the generated class/function for them to be preserved in 
__annotations__ for some future runtime use.

In the former case, you are unlikely to use local state. But even if you were, 
that doesn't matter since the static type checker 

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Lukasz Langa]

> On 4 Nov, 2017, at 11:43 AM, Peter Ludemann via Python-Dev 
>  wrote:
> 
> If type annotations are treated like implicit lambdas, then that's a first 
> step to something similar to Lisp's "special forms". A full generalization of 
> that would allow, for example, logging.debug to not evaluate its args unless 
> debugging is turned on (I use a logging.debug wrapper that allows lambdas as 
> args, and evaluates them if debugging is turned on).

Interestingly enough, at Facebook we found out that using f-strings is *faster* 
at runtime than the lazy form of logging.log("format with %s and %d", arg1, 
arg2), including for cases when the log message is not emitted.

- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 5 November 2017 at 02:42, Guido van Rossum  wrote:
> I'm very worried about trying to come up with a robust implementation of
> this in under 12 weeks. By contrast, the stringification that Łukasz is
> proposing feels eminently doable.

I'm far from confident about that, as the string proposal inherently
breaks runtime type annotation evaluation for nested function and
class definitions, since those lose access to nonlocal variable
references (since the compiler isn't involved in their name resolution
any more).

https://www.python.org/dev/peps/pep-0563/#resolving-type-hints-at-runtime
is essentially defining a completely new type annotation specific
scheme for name resolution, and takes us back to a Python 1.x era
"locals and globals only" approach with no support for closure
variables.

Consider this example from the PEP:

def generate():
A = Optional[int]
class C:
field: A = 1
def method(self, arg: A) -> None: ...
return C
X = generate()

The PEP's current attitude towards this is "Yes, it will break, but
that's OK, because it doesn't matter for the type annotation use case,
since static analysers will still understand it". Adopting such a
cavalier approach towards backwards compatibility with behaviour that
has been supported since Python 3.0 *isn't OK*, since it would mean we
were taking the step from "type annotations are the primary use case"
to "Other use cases for function annotations are no longer supported".

The only workaround I can see for that breakage is that instead of
using strings, we could instead define a new "thunk" type that
consists of two things:

1. A code object to be run with eval()
2. A dictionary mapping from variable names to closure cells (or None
for not yet resolved references to globals and builtins)

Correctly evaluating the code object in its original context would
then be possible by reading the "cell_contents" attributes of the
cells stored in the mapping and injecting them into the globals
namespace used to run the code.

This would actually be a pretty cool new primitive to have available
(since it also leaves the consuming code free to *ignore* the closure
cells, which is what you'd want for use cases like callback functions
with implicitly named parameters), and retains the current eager
compilation behaviour (so we'd be storing compiled code objects as
constants instead of strings).

If PEP 563 were updated to handle closure references properly using a
scheme like the one above, I'd be far more supportive of the proposal.

Alternatively, in a lambda based proposal that compiled code like the
above as equivalent to the following code today:

def generate():
A = Optional[int]
class C:
field: A = 1
def method(self, arg: (lambda: A)) -> None: ...
return C
X = generate()

Then everything's automatically fine, since the compiler would
correctly resolve the nonlocal reference to A and inject the
appropriate closure references.

In such a lambda based implementation, the *only* tricky case is this
one, where the typevar is declared at class scope:

class C:
A = Optional[int]
field: A = 1
def method(self, arg: A) -> None: ...

Now, even without the introduction of the IndirectAttributeCell
concept, this is amenable to a pretty simple workaround:

 A = Optional[int]
 class C:
field: A = 1
def method(self, arg: A) -> None: ...
C.A = A
del A

But I genuinely can't see how breaking annotation evaluation at class
scope can be seen as a deal-breaker for the implicit lambda based
approach without breaking annotation evaluation for nested functions
also being seen as a deal-breaker for the string based approach.

Either way, there are going to be changes needed to the compiler in
order for it to still generate suitable references at compile time -
the only question would then be whether they're existing cells stored
in a new construct (a thunk to be executed with eval rather than via a
regular function call), or a new kind of cell stored on a regular
function object (implicit access to class attributes from implicitly
defined scopes in the class body).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Peter Ludemann via Python-Dev]

If type annotations are treated like implicit lambdas, then that's a first
step to something similar to Lisp's "special forms". A full generalization
of that would allow, for example, logging.debug to not evaluate its args
unless debugging is turned on (I use a logging.debug wrapper that allows
lambdas as args, and evaluates them if debugging is turned on).

Maybe a better question is whether we want "special forms" in Python. It
complicates some things but simplifies others. But things that satisfy Lisp
programmers might not make Python programmers happy. ;)


On 4 November 2017 at 09:42, Guido van Rossum  wrote:

> I'm very worried about trying to come up with a robust implementation of
> this in under 12 weeks. By contrast, the stringification that Łukasz is
> proposing feels eminently doable.
>
> On Sat, Nov 4, 2017 at 6:51 AM, Nick Coghlan  wrote:
>
>> On 4 November 2017 at 00:40, Guido van Rossum  wrote:
>> > IMO the inability of referencing class-level definitions from
>> annotations on
>> > methods pretty much kills this idea.
>>
>> If we decided we wanted to make it work, I think the key runtime
>> building block we would need is a new kind of cell reference: an
>> IndirectAttributeCell.
>>
>> Those would present the same interface as a regular nonlocal cell (so
>> it could be stored in __closure__ just as regular cells are, and
>> accessed the same way when the function body is executed), but
>> internally it would hold two references:
>>
>> - one to another cell object (__class__ for this use case)
>> - an attribute name on the target object that get/set/del operations
>> on the indirect cell's value should affect
>>
>> As Python code:
>>
>> class IndirectAttributeCell:
>> def __new__(cls, cell, attr):
>> self._cell = cell
>> self._attr = attr
>>
>> @property
>> def cell_contents(self):
>> return getattr(self._cell.cell_contents, self._attr)
>>
>> @cell_contents.setter
>> def cell_contents(self, value):
>> setattr(self._cell.cell_contents, self._attr, value)
>>
>> @cell_contents.deleter
>> def cell_contents(self):
>> delattr(self._cell.cell_contents, self._attr)
>>
>> The class body wouldn't be able to evaluate the thunks (since
>> `__class__` wouldn't be set yet), but `__init_subclass__`
>> implementations could, as could class decorators.
>>
>> It would require some adjustment in the compiler as well (in order to
>> pass the class level attribute definitions down to these implicitly
>> defined scopes as a new kind of accessible external namespace during
>> the symbol analysis pass, as well as to register the use of
>> "__class__" if one of the affected names was referenced), but I think
>> it would work at least at a technical level (by contrast, every other
>> idea I came up with back when I was working on the list comprehension
>> change was sufficiently flawed that it fell apart within a few hours
>> of starting to tinker with the idea).
>>
>> As an added bonus, we could potentially also extend the same
>> permissive name resolution semantics to the implicit scopes used in
>> comprehensions, such that it was only the explicitly defined scopes
>> (i.e. lambda expressions, function definitions, and nested classes)
>> that lost implicit access to the class level variables.
>>
>> Cheers,
>> Nick.
>>
>> P.S. If we subsequently decided to elevate expression thunks to a
>> first class language primitive, they shouldn't need any further
>> semantic enhancements beyond that one, since the existing scoping
>> rules already give the desired behaviour at module and function scope.
>>
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> pludemann%40google.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Guido van Rossum]

I'm very worried about trying to come up with a robust implementation of
this in under 12 weeks. By contrast, the stringification that Łukasz is
proposing feels eminently doable.

On Sat, Nov 4, 2017 at 6:51 AM, Nick Coghlan  wrote:

> On 4 November 2017 at 00:40, Guido van Rossum  wrote:
> > IMO the inability of referencing class-level definitions from
> annotations on
> > methods pretty much kills this idea.
>
> If we decided we wanted to make it work, I think the key runtime
> building block we would need is a new kind of cell reference: an
> IndirectAttributeCell.
>
> Those would present the same interface as a regular nonlocal cell (so
> it could be stored in __closure__ just as regular cells are, and
> accessed the same way when the function body is executed), but
> internally it would hold two references:
>
> - one to another cell object (__class__ for this use case)
> - an attribute name on the target object that get/set/del operations
> on the indirect cell's value should affect
>
> As Python code:
>
> class IndirectAttributeCell:
> def __new__(cls, cell, attr):
> self._cell = cell
> self._attr = attr
>
> @property
> def cell_contents(self):
> return getattr(self._cell.cell_contents, self._attr)
>
> @cell_contents.setter
> def cell_contents(self, value):
> setattr(self._cell.cell_contents, self._attr, value)
>
> @cell_contents.deleter
> def cell_contents(self):
> delattr(self._cell.cell_contents, self._attr)
>
> The class body wouldn't be able to evaluate the thunks (since
> `__class__` wouldn't be set yet), but `__init_subclass__`
> implementations could, as could class decorators.
>
> It would require some adjustment in the compiler as well (in order to
> pass the class level attribute definitions down to these implicitly
> defined scopes as a new kind of accessible external namespace during
> the symbol analysis pass, as well as to register the use of
> "__class__" if one of the affected names was referenced), but I think
> it would work at least at a technical level (by contrast, every other
> idea I came up with back when I was working on the list comprehension
> change was sufficiently flawed that it fell apart within a few hours
> of starting to tinker with the idea).
>
> As an added bonus, we could potentially also extend the same
> permissive name resolution semantics to the implicit scopes used in
> comprehensions, such that it was only the explicitly defined scopes
> (i.e. lambda expressions, function definitions, and nested classes)
> that lost implicit access to the class level variables.
>
> Cheers,
> Nick.
>
> P.S. If we subsequently decided to elevate expression thunks to a
> first class language primitive, they shouldn't need any further
> semantic enhancements beyond that one, since the existing scoping
> rules already give the desired behaviour at module and function scope.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 4 November 2017 at 00:40, Guido van Rossum  wrote:
> IMO the inability of referencing class-level definitions from annotations on
> methods pretty much kills this idea.

If we decided we wanted to make it work, I think the key runtime
building block we would need is a new kind of cell reference: an
IndirectAttributeCell.

Those would present the same interface as a regular nonlocal cell (so
it could be stored in __closure__ just as regular cells are, and
accessed the same way when the function body is executed), but
internally it would hold two references:

- one to another cell object (__class__ for this use case)
- an attribute name on the target object that get/set/del operations
on the indirect cell's value should affect

As Python code:

class IndirectAttributeCell:
def __new__(cls, cell, attr):
self._cell = cell
self._attr = attr

@property
def cell_contents(self):
return getattr(self._cell.cell_contents, self._attr)

@cell_contents.setter
def cell_contents(self, value):
setattr(self._cell.cell_contents, self._attr, value)

@cell_contents.deleter
def cell_contents(self):
delattr(self._cell.cell_contents, self._attr)

The class body wouldn't be able to evaluate the thunks (since
`__class__` wouldn't be set yet), but `__init_subclass__`
implementations could, as could class decorators.

It would require some adjustment in the compiler as well (in order to
pass the class level attribute definitions down to these implicitly
defined scopes as a new kind of accessible external namespace during
the symbol analysis pass, as well as to register the use of
"__class__" if one of the affected names was referenced), but I think
it would work at least at a technical level (by contrast, every other
idea I came up with back when I was working on the list comprehension
change was sufficiently flawed that it fell apart within a few hours
of starting to tinker with the idea).

As an added bonus, we could potentially also extend the same
permissive name resolution semantics to the implicit scopes used in
comprehensions, such that it was only the explicitly defined scopes
(i.e. lambda expressions, function definitions, and nested classes)
that lost implicit access to the class level variables.

Cheers,
Nick.

P.S. If we subsequently decided to elevate expression thunks to a
first class language primitive, they shouldn't need any further
semantic enhancements beyond that one, since the existing scoping
rules already give the desired behaviour at module and function scope.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Barry Warsaw]

On Nov 3, 2017, at 17:09, Luca Sbardella  wrote:
> 
> Impressive stats! I didn’t know this command, thanks!

Neither did I until a day or so ago.  I already only want to use Python 3.7.  :)

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Luca Sbardella]

Impressive stats! I didn’t know this command, thanks!

On Fri, 3 Nov 2017 at 20:47, Barry Warsaw  wrote:

> On Nov 2, 2017, at 23:22, Nick Coghlan  wrote:
> > Another point worth noting is that merely importing the typing module
> > is expensive:
> >
> > $ python -m perf timeit -s "from importlib import reload; import
> > typing" "reload(typing)"
> > .
> > Mean +- std dev: 10.6 ms +- 0.6 ms
> >
> > 10 ms is a *big* chunk out of a CLI application's startup time budget.
>
> Far and away so, except for the re module.
>
> % ./python.exe -X importtime -c "import typing"
> import time: self [us] | cumulative | imported package
> import time:72 | 72 | _codecs
> import time:   625 |696 |   codecs
> import time:   354 |354 |   encodings.aliases
> import time:   713 |   1762 | encodings
> import time:   198 |198 | encodings.utf_8
> import time:98 | 98 | _signal
> import time:   233 |233 | encodings.latin_1
> import time:   353 |353 | _weakrefset
> import time:   264 |617 |   abc
> import time:   402 |   1018 | io
> import time:   136 |136 |   _stat
> import time:   197 |333 | stat
> import time:   227 |227 |   genericpath
> import time:   377 |604 | posixpath
> import time:  2812 |   2812 | _collections_abc
> import time:   787 |   4534 |   os
> import time:   315 |315 |   _sitebuiltins
> import time:   336 |336 |   sitecustomize
> import time:   114 |114 |   usercustomize
> import time:  1064 |   6361 | site
> import time:   160 |160 |   _operator
> import time:  1412 |   1571 | operator
> import time:   371 |371 | keyword
> import time:   817 |817 |   _heapq
> import time:   762 |   1579 | heapq
> import time:   272 |272 | itertools
> import time:   635 |635 | reprlib
> import time:99 | 99 | _collections
> import time:  3580 |   8104 |   collections
> import time:   112 |112 |   _functools
> import time:   781 |892 | functools
> import time:  1774 |   2666 |   contextlib
> import time:   272 |272 |   types
> import time:   861 |   1132 | enum
> import time:76 | 76 |   _sre
> import time:   426 |426 | sre_constants
> import time:   446 |872 |   sre_parse
> import time:   414 |   1361 | sre_compile
> import time:79 | 79 | _locale
> import time:   190 |190 | copyreg
> import time: 17200 |  19961 |   re
> import time:   374 |374 |   collections.abc
> import time: 15124 |  46226 | typing
>
> -Barry
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/luca.sbardella%40gmail.com
>
-- 
http://lucasbardella.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Barry Warsaw]

On Nov 2, 2017, at 23:22, Nick Coghlan  wrote:
> Another point worth noting is that merely importing the typing module
> is expensive:
> 
> $ python -m perf timeit -s "from importlib import reload; import
> typing" "reload(typing)"
> .
> Mean +- std dev: 10.6 ms +- 0.6 ms
> 
> 10 ms is a *big* chunk out of a CLI application's startup time budget.

Far and away so, except for the re module.

% ./python.exe -X importtime -c "import typing"
import time: self [us] | cumulative | imported package
import time:72 | 72 | _codecs
import time:   625 |696 |   codecs
import time:   354 |354 |   encodings.aliases
import time:   713 |   1762 | encodings
import time:   198 |198 | encodings.utf_8
import time:98 | 98 | _signal
import time:   233 |233 | encodings.latin_1
import time:   353 |353 | _weakrefset
import time:   264 |617 |   abc
import time:   402 |   1018 | io
import time:   136 |136 |   _stat
import time:   197 |333 | stat
import time:   227 |227 |   genericpath
import time:   377 |604 | posixpath
import time:  2812 |   2812 | _collections_abc
import time:   787 |   4534 |   os
import time:   315 |315 |   _sitebuiltins
import time:   336 |336 |   sitecustomize
import time:   114 |114 |   usercustomize
import time:  1064 |   6361 | site
import time:   160 |160 |   _operator
import time:  1412 |   1571 | operator
import time:   371 |371 | keyword
import time:   817 |817 |   _heapq
import time:   762 |   1579 | heapq
import time:   272 |272 | itertools
import time:   635 |635 | reprlib
import time:99 | 99 | _collections
import time:  3580 |   8104 |   collections
import time:   112 |112 |   _functools
import time:   781 |892 | functools
import time:  1774 |   2666 |   contextlib
import time:   272 |272 |   types
import time:   861 |   1132 | enum
import time:76 | 76 |   _sre
import time:   426 |426 | sre_constants
import time:   446 |872 |   sre_parse
import time:   414 |   1361 | sre_compile
import time:79 | 79 | _locale
import time:   190 |190 | copyreg
import time: 17200 |  19961 |   re
import time:   374 |374 |   collections.abc
import time: 15124 |  46226 | typing

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Guido van Rossum]

IMO the inability of referencing class-level definitions from annotations
on methods pretty much kills this idea.

On Thu, Nov 2, 2017 at 11:27 PM, Nick Coghlan  wrote:

> On 3 November 2017 at 04:39, Jukka Lehtosalo  wrote:
> >> > * forward references: when a type hint contains names that have not
> been
> >> >   defined yet, that definition needs to be expressed as a string
> >> >   literal;
> >>
> >> After all the discussion, I still don't see why this is an issue.
> >> Strings makes perfectly fine forward references. What is the problem
> >> that needs solving? Is this about people not wanting to type the leading
> >> and trailing ' around forward references?
> >
> >
> > Let's make a thought experiment. What if every forward reference would
> > require special quoting? Would Python programmers be happy with this?
> Say,
> > let's use ! as a suffix to mark a forward reference. They make perfectly
> > fine forward references. They are visually pretty unobtrusive (I'm not
> > suggesting $ or other ugly perlisms):
> >
> > def main():
> > args = parse_args!()  # A forward reference
> > do_stuff!(args)  # Explicit is better than implicit
> >
> > def parse_args():
> > ...
> >
> > def do_stuff(args):
> > ...
> >
> > Of course, I'm not seriously proposing this, but this highlights the fact
> > that in normal code forward references "just work" (at least usually),
> and
> > if we'd require a special quoting mechanism to use them anywhere, Python
> > would look uglier and more inconsistent. Nobody would be happy with this
> > change, even though you'd only have to type a single ! character extra --
> > that's not a lot work, right?
> >
> > I think that the analogy is reasonable.
>
> I think it also makes a pretty decent argument that pushing function
> annotations into implicit lambda expressions will be easier to explain
> to people than converting them into strings, and then having to
> explain an entirely new complex set of name resolution rules.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 3 November 2017 at 04:39, Jukka Lehtosalo  wrote:
>> > * forward references: when a type hint contains names that have not been
>> >   defined yet, that definition needs to be expressed as a string
>> >   literal;
>>
>> After all the discussion, I still don't see why this is an issue.
>> Strings makes perfectly fine forward references. What is the problem
>> that needs solving? Is this about people not wanting to type the leading
>> and trailing ' around forward references?
>
>
> Let's make a thought experiment. What if every forward reference would
> require special quoting? Would Python programmers be happy with this? Say,
> let's use ! as a suffix to mark a forward reference. They make perfectly
> fine forward references. They are visually pretty unobtrusive (I'm not
> suggesting $ or other ugly perlisms):
>
> def main():
> args = parse_args!()  # A forward reference
> do_stuff!(args)  # Explicit is better than implicit
>
> def parse_args():
> ...
>
> def do_stuff(args):
> ...
>
> Of course, I'm not seriously proposing this, but this highlights the fact
> that in normal code forward references "just work" (at least usually), and
> if we'd require a special quoting mechanism to use them anywhere, Python
> would look uglier and more inconsistent. Nobody would be happy with this
> change, even though you'd only have to type a single ! character extra --
> that's not a lot work, right?
>
> I think that the analogy is reasonable.

I think it also makes a pretty decent argument that pushing function
annotations into implicit lambda expressions will be easier to explain
to people than converting them into strings, and then having to
explain an entirely new complex set of name resolution rules.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Nick Coghlan]

On 3 November 2017 at 03:00, Brett Cannon  wrote:
> The cost of constructing some of the objects used as type hints can be very
> expensive and make importing really expensive (this has been pointed out by
> Lukasz previously as well as Inada-san). By making Python itself not have to
> construct objects from e.g. the 'typing' module at runtime, you then don't
> pay a runtime penalty for something you're almost never going to use at
> runtime anyway.

Another point worth noting is that merely importing the typing module
is expensive:

$ python -m perf timeit -s "from importlib import reload; import
typing" "reload(typing)"
.
Mean +- std dev: 10.6 ms +- 0.6 ms

10 ms is a *big* chunk out of a CLI application's startup time budget.

So I think to be truly effective in achieving its goals, the PEP will
also need to promote TYPE_CHECKING to a builtin, such that folks can
write:

from __future__ import lazy_annotations # More self-explanatory name
if TYPE_CHECKING:
import typing

and be able to express their type annotations in a way that a static
type checker will understand, while incurring near-zero runtime
overhead.

[snip]

Regarding the thunk idea: the compiler can pretty easily rewrite all
annotations from being "" to "lambda: ".

This has a couple of very nice properties:

* rendering them later is purely a matter of calling them, since the
compiler will take care of capturing all the right namespaces and name
references (including references from method declarations to the type
defining them)
* quoted and unquoted annotations will reliably render differently
(since one will return a string when called, and the other won't)

The big downside to this approach is that it makes simple annotations
*more* expensive rather than less expensive.

Baseline (mentioning a builtin):
$ python -m perf timeit "str"
.
Mean +- std dev: 27.1 ns +- 1.4 ns

String constant (~4x speedup):
$ python -m perf timeit "'str'"
.
Mean +- std dev: 7.84 ns +- 0.22 ns

Lambda expression (~2x slowdown):
$ python -m perf timeit "lambda: str"
.
Mean +- std dev: 62.3 ns +- 1.4 ns

That said, I'll also point out the following:

* for application startup purposes, if you save 10 ms by not importing
the typing module, then that buys you time for around 285 *thousand*
implicit lambda declarations before your startup actually gets slower
(assuming the relative timings on my machine are typical)
* for nested functions, the overhead of the function call is enough
that the dramatic 4x speedup vs 2x slowdown ratio disappears (see P.S.
for numbers)
* I don't believe we've really invested much time in optimising the
creation of zero-argument lambdas yet, so there may be options for
bringing the numbers down for the lambda based approach

The other key downside to the lambda based approach is that it hits
the same backwards compatibility problem we hit when list
comprehensions were given their own nested scope: if we push
annotations down into a new scope, they won't be able to see class
level attributes any more. For comprehensions, we could partially
mitigate that by evaluating the outermost iterable expression in the
class scope, but there's no equivalent to that available for
annotations (since the annotation's lambda expression may never be
called at all).

Cheers,
Nick.

P.S. Relative performance of the annotation styles in a nested
function definition

Baseline:
$ python -m perf timeit -s "def f(): str" "f()"
.
Mean +- std dev: 103 ns +- 3 ns

String constant (~1.25x speedup):
$ python -m perf timeit -s "def f(): 'str'" "f()"
.
Mean +- std dev: 77.0 ns +- 1.7 ns

Lambda expression (~1.5x slowdown):
$ python -m perf timeit -s "def f(): (lambda: str)" "f()"
.
Mean +- std dev: 149 ns +- 6 ns

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[INADA Naoki]

I'm 100% agree with Łukasz and Brett.
+1 and thanks for writing this PEP.
INADA Naoki  


On Fri, Nov 3, 2017 at 2:00 AM, Brett Cannon  wrote:
>
>
> On Thu, 2 Nov 2017 at 08:46 Steven D'Aprano  wrote:
>>
>> On Wed, Nov 01, 2017 at 03:48:00PM -0700, Lukasz Langa wrote:
>>
>> > PEP: 563
>> > Title: Postponed Evaluation of Annotations
>>
>> > This PEP proposes changing function annotations and variable annotations
>> > so that they are no longer evaluated at function definition time.
>> > Instead, they are preserved in ``__annotations__`` in string form.
>>
>> This means that now *all* annotations, not just forward references, are
>> no longer validated at runtime and will allow arbitrary typos and
>> errors:
>>
>> def spam(n:itn):  # now valid
>> ...
>>
>> Up to now, it has been only forward references that were vulnerable to
>> that sort of thing. Of course running a type checker should pick those
>> errors up, but the evaluation of annotations ensures that they are
>> actually valid (not necessarily correct, but at least a valid name),
>> even if you happen to not be running a type checker. That's useful.
>>
>> Are we happy to live with that change?
>
>
> I would say "yes" for two reasons. One, if you're bothering to provide type
> hints then you should be testing those type hints. So as you pointed out,
> Steve, that will be caught at that point.
>
> Two, code editors with auto-completion will help prevent this kind of typo.
> Now I would never suggest that we design Python with expectations of what
> sort of tooling people have available, but in this instance it will help. It
> also feeds into a question you ask below...
>
>>
>>
>>
>> > Rationale and Goals
>> > ===
>> >
>> > PEP 3107 added support for arbitrary annotations on parts of a function
>> > definition.  Just like default values, annotations are evaluated at
>> > function definition time.  This creates a number of issues for the type
>> > hinting use case:
>> >
>> > * forward references: when a type hint contains names that have not been
>> >   defined yet, that definition needs to be expressed as a string
>> >   literal;
>>
>> After all the discussion, I still don't see why this is an issue.
>> Strings makes perfectly fine forward references. What is the problem
>> that needs solving? Is this about people not wanting to type the leading
>> and trailing ' around forward references?
>
>
> I think it's mainly about the next point you ask about...
>
>>
>>
>>
>> > * type hints are executed at module import time, which is not
>> >   computationally free.
>>
>> True; but is that really a performance bottleneck? If it is, that should
>> be stated in the PEP, and state what typical performance improvement
>> this change should give.
>>
>> After all, if we're going to break people's code in order to improve
>> performance, we should at least be sure that it improves performance :-)
>
>
> The cost of constructing some of the objects used as type hints can be very
> expensive and make importing really expensive (this has been pointed out by
> Lukasz previously as well as Inada-san). By making Python itself not have to
> construct objects from e.g. the 'typing' module at runtime, you then don't
> pay a runtime penalty for something you're almost never going to use at
> runtime anyway.
>
>>
>>
>>
>> > Postponing the evaluation of annotations solves both problems.
>>
>> Actually it doesn't. As your PEP says later:
>>
>> > This PEP is meant to solve the problem of forward references in type
>> > annotations.  There are still cases outside of annotations where
>> > forward references will require usage of string literals.  Those are
>> > listed in a later section of this document.
>>
>> So the primary problem this PEP is designed to solve, isn't actually
>> solved by this PEP.
>
>
> I think the performance bit is really the big deal here.
>
> And as I mentioned earlier, if you turn all of your type hints into strings,
> you lose auto-completion/intellisense which is a shame.
>
> I think there's also a benefit here of promoting the fact that type hints
> are not a runtime thing, they are a static analysis thing. By requiring the
> extra step to convert from a string to an actual object, it helps get the
> point across that type hints are just bits of metadata for tooling and not
> something you're expected really interact with at runtime unless you have a
> really good reason to.
>
> So I'm +1 on the idea, but the __future__ statement is a bit too generic for
> me. I would prefer something like `from __future__ import
> annotation_strings` or `annotations_as_strings`.
>
> -Brett
>
>>
>>
>> (See Guido's comments, quoted later.)
>>
>>
>>
>> > Implementation
>> > ==
>> >
>> > In Python 4.0, function and variable annotations will no longer be
>> > evaluated at definition time.  Instead, a string form will be preserved
>> > in the respective ``__annotations__`` dictionary. 

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Lukasz Langa]

> On Nov 2, 2017, at 1:13 PM, Stéfane Fermigier  wrote:
> 
> Another common use case is dependency injection / IoC:
> 
> - Injector (https://github.com/alecthomas/injector 
> ):
> - Flask-Injector (ok it's the same underlying injector):

Pretty cool! This is already using `get_type_hints()` so it's perfectly 
compatible with PEP 563:

https://github.com/alecthomas/injector/blob/master/injector.py#L915

> - Apistar components (https://github.com/encode/apistar#components 
>  ):

This is using `inspect` directly so will have to migrate to `get_type_hints()` 
later. But, as the PEP mentions, it should already be using `get_type_hints()` 
since people are free to use forward references even today.

Thanks for flagging those use cases!


- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Stéfane Fermigier]

On Thu, Nov 2, 2017 at 7:39 PM, Jukka Lehtosalo 
wrote:

>
> As type checking has become the main use case for annotations, using
> annotations without a type checker is fast becoming a marginal use case.
> Type checkers can easily and reliably validate that names in annotations 
> aren't
> misspelled.
>

Another common use case is dependency injection / IoC:

Examples include:

- Injector (https://github.com/alecthomas/injector):

>>> class Outer:
... @inject
... def __init__(self, inner: Inner):
... self.inner = inner


- Flsk-Injector (ok it's the same underlying injector):

# Route with injection
@app.route("/foo")
def foo(db: sqlite3.Connection):
users = db.execute('SELECT * FROM users').all()
return render("foo.html")

- Apistar components (https://github.com/encode/apistar#components ):

def say_hello(user: User):
return {'hello': user.username}

=> In each of the examples, the type annotation are used at runtime by the
IoC container to inject an object of the appropriate type, based on some
specifications.

They may or may not be used by a typechecker too, but that's secondary.

  S.

-- 
Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier -
http://linkedin.com/in/sfermigier
Founder & CEO, Abilian - Enterprise Social Software -
http://www.abilian.com/
Chairman, Free Group / Systematic Cluster -
http://www.gt-logiciel-libre.org/
Co-Chairman, National Council for Free & Open Source Software (CNLL) -
http://cnll.fr/
Founder & Organiser, PyData Paris - http://pydata.fr/
---
“You never change things by fighting the existing reality. To change
something, build a new model that makes the existing model obsolete.” — R.
Buckminster Fuller
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Ivan Levkivskyi]

On 1 November 2017 at 23:48, Lukasz Langa  wrote:

> Runtime annotation resolution and class decorators
> --
>
> Metaclasses and class decorators that need to resolve annotations for
> the current class will fail for annotations that use the name of the
> current class.  Example::
>
> def class_decorator(cls):
> annotations = get_type_hints(cls)  # raises NameError on 'C'
> print(f'Annotations for {cls}: {annotations}')
> return cls
>
> @class_decorator
> class C:
> singleton: 'C' = None
>
> This was already true before this PEP.  The class decorator acts on
> the class before it's assigned a name in the current definition scope.
>
>
Just a random idea: maybe this can be resolved by just updating the localns
before calling get_type_hints() like this:

  localns = locals().copy()
  localns.update({cls.__name__: cls})


In general I like how the PEP is written now. I maybe would add examples
for this

> the cases listed above might be worked around by placing the usage
> in a if TYPE_CHECKING: block.
> ...
> For named tuples, using the new class definition syntax introduced in
Python 3.6 solves the issue.

actually showing something like

  if TYPE_CHECKING:
  Alias = List[Tuple[int, SomeClass]]

  class NT(NamedTuple):
  one: SomeClass
  other: Alias

  class SomeClass:
  ...

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Ivan Levkivskyi]

On 2 November 2017 at 18:00, Brett Cannon  wrote:

>
>
> On Thu, 2 Nov 2017 at 08:46 Steven D'Aprano  wrote:
>
>> On Wed, Nov 01, 2017 at 03:48:00PM -0700, Lukasz Langa wrote:
>> [...snip...]
>
> I think the performance bit is really the big deal here.
>
>
I don't think so. Although subscripting generics is indeed very expensive,
this is heavily optimised
by various caches. I think PEP 560 might actually have comparable (or even
bigger) performance effects.

IIUC performance is listed second in the PEP for a reason. I am OK with
using quotes for forward references, but I have seen
many people complaining about this (especially novices). I think Jukka is
right here.

We can't allow all unquoted forward references but those that will still be
quoted appear in more advanced situations
like bounded type variables and derived generics. But in these cases it is
clear that a forward reference appears
in runtime context.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Jukka Lehtosalo]

On Thu, Nov 2, 2017 at 3:45 PM, Steven D'Aprano  wrote:

> On Wed, Nov 01, 2017 at 03:48:00PM -0700, Lukasz Langa wrote:
>
> > This PEP proposes changing function annotations and variable annotations
> > so that they are no longer evaluated at function definition time.
> > Instead, they are preserved in ``__annotations__`` in string form.
>
> This means that now *all* annotations, not just forward references, are
> no longer validated at runtime and will allow arbitrary typos and
> errors:
>
> def spam(n:itn):  # now valid
> ...
>
> Up to now, it has been only forward references that were vulnerable to
> that sort of thing. Of course running a type checker should pick those
> errors up, but the evaluation of annotations ensures that they are
> actually valid (not necessarily correct, but at least a valid name),
> even if you happen to not be running a type checker. That's useful.
>
> Are we happy to live with that change?
>

Within functions misspellings won't be caught until you invoke a function:

def spam(s):
return itn(s)  # no error unless spam() is called

We've lived with this for a long time and generally people seem to be happy
with it. The vast majority of code in non-trivial programs (where type
annotations are useful) tends to be within functions, so this will only
slightly increase the number of things that won't be caught without running
tests (or running the type checker).

As type checking has become the main use case for annotations, using
annotations without a type checker is fast becoming a marginal use case.
Type checkers can easily and reliably validate that names in annotations aren't
misspelled.

> * forward references: when a type hint contains names that have not been
> >   defined yet, that definition needs to be expressed as a string
> >   literal;
>
> After all the discussion, I still don't see why this is an issue.
> Strings makes perfectly fine forward references. What is the problem
> that needs solving? Is this about people not wanting to type the leading
> and trailing ' around forward references?
>

Let's make a thought experiment. What if every forward reference would
require special quoting? Would Python programmers be happy with this? Say,
let's use ! as a suffix to mark a forward reference. They make perfectly
fine forward references. They are visually pretty unobtrusive (I'm not
suggesting $ or other ugly perlisms):

def main():
args = parse_args!()  # A forward reference
do_stuff!(args)  # Explicit is better than implicit

def parse_args():
...

def do_stuff(args):
...

Of course, I'm not seriously proposing this, but this highlights the fact
that in normal code forward references "just work" (at least usually), and
if we'd require a special quoting mechanism to use them anywhere, Python
would look uglier and more inconsistent. Nobody would be happy with this
change, even though you'd only have to type a single ! character extra --
that's not a lot work, right?

I think that the analogy is reasonable. In type checked code annotations
are one of most widely used language features -- it's quite possible to
have annotations for almost every function in a code base. This is not a
marginal feature, and people expect commonly used features to feel polished
and usable, not inconsistent and hacky. It's quite possible that the first
type annotated experiment a user writes requires the use of forward
references, and this gives a pretty bad first impression -- not unlike how
the ! forward reference would make the first impression of using
non-type-checked Python pretty awkward.

Here are more arguments why literal escapes are a usability problem:

1) It's hard to predict when string quotes are needed. Real-world large
code bases tend to have a lot of import cycles, and string literal escapes
are often needed within import cycles. However, they aren't always needed.
To annotate code correctly, you frequently need to understand how the file
you are editing is related to other modules in terms of import cycle
structure. In large code bases this can be very difficult to keep in your
head, so basically adding forward references becomes a matter of
tweak-until-it-works. So either each time you write an annotation, you can
look at how imports are structured -- to see whether a particular type
needs to be quoted -- or you can guess and hope for the best. This is a
terrible user experience and increases cognitive load significantly. Our
goal should not be to just have something that technically 'works', as this
is a very low standard. I want Python to be easy to use, intuitive and
elegant. I don't expect that anybody who has annotated large code bases
could consider string literal forward references to be any of those.

2) It's one of the top complaints from users. Even a non-user with a basic
understanding of mypy told me what amounts to "Python doesn't have real
static typing; forward references make it obvious that types are just an

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Brett Cannon]

On Thu, 2 Nov 2017 at 08:46 Steven D'Aprano  wrote:

> On Wed, Nov 01, 2017 at 03:48:00PM -0700, Lukasz Langa wrote:
>
> > PEP: 563
> > Title: Postponed Evaluation of Annotations
>
> > This PEP proposes changing function annotations and variable annotations
> > so that they are no longer evaluated at function definition time.
> > Instead, they are preserved in ``__annotations__`` in string form.
>
> This means that now *all* annotations, not just forward references, are
> no longer validated at runtime and will allow arbitrary typos and
> errors:
>
> def spam(n:itn):  # now valid
> ...
>
> Up to now, it has been only forward references that were vulnerable to
> that sort of thing. Of course running a type checker should pick those
> errors up, but the evaluation of annotations ensures that they are
> actually valid (not necessarily correct, but at least a valid name),
> even if you happen to not be running a type checker. That's useful.
>
> Are we happy to live with that change?
>

I would say "yes" for two reasons. One, if you're bothering to provide type
hints then you should be testing those type hints. So as you pointed out,
Steve, that will be caught at that point.

Two, code editors with auto-completion will help prevent this kind of typo.
Now I would never suggest that we design Python with expectations of what
sort of tooling people have available, but in this instance it will help.
It also feeds into a question you ask below...


>
>
> > Rationale and Goals
> > ===
> >
> > PEP 3107 added support for arbitrary annotations on parts of a function
> > definition.  Just like default values, annotations are evaluated at
> > function definition time.  This creates a number of issues for the type
> > hinting use case:
> >
> > * forward references: when a type hint contains names that have not been
> >   defined yet, that definition needs to be expressed as a string
> >   literal;
>
> After all the discussion, I still don't see why this is an issue.
> Strings makes perfectly fine forward references. What is the problem
> that needs solving? Is this about people not wanting to type the leading
> and trailing ' around forward references?
>

I think it's mainly about the next point you ask about...


>
>
> > * type hints are executed at module import time, which is not
> >   computationally free.
>
> True; but is that really a performance bottleneck? If it is, that should
> be stated in the PEP, and state what typical performance improvement
> this change should give.
>
> After all, if we're going to break people's code in order to improve
> performance, we should at least be sure that it improves performance :-)
>

The cost of constructing some of the objects used as type hints can be very
expensive and make importing really expensive (this has been pointed out by
Lukasz previously as well as Inada-san). By making Python itself not have
to construct objects from e.g. the 'typing' module at runtime, you then
don't pay a runtime penalty for something you're almost never going to use
at runtime anyway.


>
>
> > Postponing the evaluation of annotations solves both problems.
>
> Actually it doesn't. As your PEP says later:
>
> > This PEP is meant to solve the problem of forward references in type
> > annotations.  There are still cases outside of annotations where
> > forward references will require usage of string literals.  Those are
> > listed in a later section of this document.
>
> So the primary problem this PEP is designed to solve, isn't actually
> solved by this PEP.
>

I think the performance bit is really the big deal here.

And as I mentioned earlier, if you turn all of your type hints into
strings, you lose auto-completion/intellisense which is a shame.

I think there's also a benefit here of promoting the fact that type hints
are not a runtime thing, they are a static analysis thing. By requiring the
extra step to convert from a string to an actual object, it helps get the
point across that type hints are just bits of metadata for tooling and not
something you're expected really interact with at runtime unless you have a
really good reason to.

So I'm +1 on the idea, but the __future__ statement is a bit too generic
for me. I would prefer something like `from __future__ import
annotation_strings` or `annotations_as_strings`.

-Brett


>
> (See Guido's comments, quoted later.)
>
>
>
> > Implementation
> > ==
> >
> > In Python 4.0, function and variable annotations will no longer be
> > evaluated at definition time.  Instead, a string form will be preserved
> > in the respective ``__annotations__`` dictionary.  Static type checkers
> > will see no difference in behavior,
>
> Static checkers don't see __annotations__ at all, since that's not
> available at edit/compile time. Static checkers see only the source
> code. The checker (and the human reader!) will no longer have the useful
> clue that something is a forward reference:
>
> # before
>  

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Steven D'Aprano]

On Wed, Nov 01, 2017 at 03:48:00PM -0700, Lukasz Langa wrote:

> PEP: 563
> Title: Postponed Evaluation of Annotations

> This PEP proposes changing function annotations and variable annotations
> so that they are no longer evaluated at function definition time.
> Instead, they are preserved in ``__annotations__`` in string form.

This means that now *all* annotations, not just forward references, are 
no longer validated at runtime and will allow arbitrary typos and 
errors:

def spam(n:itn):  # now valid
...

Up to now, it has been only forward references that were vulnerable to 
that sort of thing. Of course running a type checker should pick those 
errors up, but the evaluation of annotations ensures that they are 
actually valid (not necessarily correct, but at least a valid name), 
even if you happen to not be running a type checker. That's useful.

Are we happy to live with that change?


> Rationale and Goals
> ===
> 
> PEP 3107 added support for arbitrary annotations on parts of a function
> definition.  Just like default values, annotations are evaluated at
> function definition time.  This creates a number of issues for the type
> hinting use case:
> 
> * forward references: when a type hint contains names that have not been
>   defined yet, that definition needs to be expressed as a string
>   literal;

After all the discussion, I still don't see why this is an issue. 
Strings makes perfectly fine forward references. What is the problem 
that needs solving? Is this about people not wanting to type the leading 
and trailing ' around forward references?


> * type hints are executed at module import time, which is not
>   computationally free.

True; but is that really a performance bottleneck? If it is, that should 
be stated in the PEP, and state what typical performance improvement 
this change should give.

After all, if we're going to break people's code in order to improve 
performance, we should at least be sure that it improves performance :-)


> Postponing the evaluation of annotations solves both problems.

Actually it doesn't. As your PEP says later:

> This PEP is meant to solve the problem of forward references in type
> annotations.  There are still cases outside of annotations where
> forward references will require usage of string literals.  Those are
> listed in a later section of this document.

So the primary problem this PEP is designed to solve, isn't actually 
solved by this PEP.

(See Guido's comments, quoted later.)



> Implementation
> ==
> 
> In Python 4.0, function and variable annotations will no longer be
> evaluated at definition time.  Instead, a string form will be preserved
> in the respective ``__annotations__`` dictionary.  Static type checkers
> will see no difference in behavior,

Static checkers don't see __annotations__ at all, since that's not 
available at edit/compile time. Static checkers see only the source 
code. The checker (and the human reader!) will no longer have the useful 
clue that something is a forward reference:

# before
class C:
def method(self, other:'C'): 
...

since the quotes around C will be redundant and almost certainly left 
out. And if they aren't left out, then what are we to make of the 
annotation? Will the quotes be stripped out, or left in?

In other words, will method's __annotations__ contain 'C' or "'C'"? That 
will make a difference when the type hint is eval'ed.


> If an annotation was already a string, this string is preserved
> verbatim.

That's ambiguous. See above.


> Annotations can only use names present in the module scope as postponed
> evaluation using local names is not reliable (with the sole exception of
> class-level names resolved by ``typing.get_type_hints()``).

Even if you call get_type_hints from inside the function defining the 
local names?

def function():
A = something()
def inner(x:A)->int:
...
d = typing.get_type_hints(inner)
return (d, inner)

I would expect that should work. Will it?


> For code which uses annotations for other purposes, a regular
> ``eval(ann, globals, locals)`` call is enough to resolve the
> annotation.

Let's just hope nobody doing that has allowed any tainted strings to 
be stuffed into __annotations__.


> * modules should use their own ``__dict__``.

Which is better written as ``vars()`` with no argument, I believe. Or 
possibly ``globals()``.


> If a function generates a class or a function with annotations that
> have to use local variables, it can populate the given generated
> object's ``__annotations__`` dictionary directly, without relying on
> the compiler.

I don't understand this paragraph.


> The biggest controversy on the issue was Guido van Rossum's concern
> that untokenizing annotation expressions back to their string form has
> no precedent in the Python programming language and feels like a hacky
> workaround.  He said:
> 
> One thing that comes to mind is that 

[Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Lukasz Langa]

Based on positive feedback on python-ideas back in September,
I'm publishing the second draft for consideration on python-dev.
I hope you like it!

A nicely formatted rendering is available here:
https://www.python.org/dev/peps/pep-0563/

(Just make sure you're looking at the version that has
"Post-History: 1-Nov-2017" as you might have an older version
cached by your browser, or you might be seeing a newer
version if you get to this e-mail in the future.)

- Ł



PEP: 563
Title: Postponed Evaluation of Annotations
Version: $Revision$
Last-Modified: $Date$
Author: Łukasz Langa 
Discussions-To: Python-Dev 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 8-Sep-2017
Python-Version: 3.7
Post-History: 1-Nov-2017
Resolution:


Abstract


PEP 3107 introduced syntax for function annotations, but the semantics
were deliberately left undefined.  PEP 484 introduced a standard meaning
to annotations: type hints.  PEP 526 defined variable annotations,
explicitly tying them with the type hinting use case.

This PEP proposes changing function annotations and variable annotations
so that they are no longer evaluated at function definition time.
Instead, they are preserved in ``__annotations__`` in string form.

This change is going to be introduced gradually, starting with a new
``__future__`` import in Python 3.7.


Rationale and Goals
===

PEP 3107 added support for arbitrary annotations on parts of a function
definition.  Just like default values, annotations are evaluated at
function definition time.  This creates a number of issues for the type
hinting use case:

* forward references: when a type hint contains names that have not been
  defined yet, that definition needs to be expressed as a string
  literal;

* type hints are executed at module import time, which is not
  computationally free.

Postponing the evaluation of annotations solves both problems.

Non-goals
-

Just like in PEP 484 and PEP 526, it should be emphasized that **Python
will remain a dynamically typed language, and the authors have no desire
to ever make type hints mandatory, even by convention.**

This PEP is meant to solve the problem of forward references in type
annotations.  There are still cases outside of annotations where
forward references will require usage of string literals.  Those are
listed in a later section of this document.

Annotations without forced evaluation enable opportunities to improve
the syntax of type hints.  This idea will require its own separate PEP
and is not discussed further in this document.

Non-typing usage of annotations
---

While annotations are still available for arbitrary use besides type
checking, it is worth mentioning that the design of this PEP, as well
as its precursors (PEP 484 and PEP 526), is predominantly motivated by
the type hinting use case.

With Python 3.7, PEP 484 graduates from provisional status.  Other
enhancements to the Python programming language like PEP 544, PEP 557,
or PEP 560, are already being built on this basis as they depend on
type annotations and the ``typing`` module as defined by PEP 484.

With this in mind, uses for annotations incompatible with the
aforementioned PEPs should be considered deprecated.


Implementation
==

In Python 4.0, function and variable annotations will no longer be
evaluated at definition time.  Instead, a string form will be preserved
in the respective ``__annotations__`` dictionary.  Static type checkers
will see no difference in behavior, whereas tools using annotations at
runtime will have to perform postponed evaluation.

If an annotation was already a string, this string is preserved
verbatim.  In other cases, the string form is obtained from the AST
during the compilation step, which means that the string form preserved
might not preserve the exact formatting of the source.

Annotations need to be syntactically valid Python expressions, also when
passed as literal strings (i.e. ``compile(literal, '', 'eval')``).
Annotations can only use names present in the module scope as postponed
evaluation using local names is not reliable (with the sole exception of
class-level names resolved by ``typing.get_type_hints()``).

Note that as per PEP 526, local variable annotations are not evaluated
at all since they are not accessible outside of the function's closure.

Enabling the future behavior in Python 3.7
--

The functionality described above can be enabled starting from Python
3.7 using the following special import::

from __future__ import annotations


Resolving Type Hints at Runtime
===

To resolve an annotation at runtime from its string form to the result
of the enclosed expression, user code needs to evaluate the string.

For code that uses type hints, the
``typing.get_type_hints(obj, globalns=None, localns=None)`` function
correctly