[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[miss-islington]

miss-islington  added the comment:


New changeset 90d584a2ab91cc0f30e90a0a99f8c1447eb51302 by Miss Islington (bot) 
in branch '3.10':
bpo-42725: mention the changes on what's new (GH-25974)
https://github.com/python/cpython/commit/90d584a2ab91cc0f30e90a0a99f8c1447eb51302


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 8.0 -> 9.0
pull_requests: +24640
pull_request: https://github.com/python/cpython/pull/25988

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
pull_requests: +24630
pull_request: https://github.com/python/cpython/pull/25974

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

> Batuhan, can this issue be closed?

No, I still need to take care of what's new entry.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Batuhan, can this issue be closed?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:


New changeset ad106c68eb00f5e4af2f937107baff6141948cee by Batuhan Taskaya in 
branch 'master':
bpo-42725: Render annotations effectless on symbol table with PEP 563 (GH-25583)
https://github.com/python/cpython/commit/ad106c68eb00f5e4af2f937107baff6141948cee


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
keywords: +patch
pull_requests: +24303
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/25583

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

>  I just thought having a whole “expr-without-walrus-or-yield” subgrammar 
> would be tedious.

Yeah I have experimented with something similar in the past and this becomes 
out of hand super quickly and leafs to a lot of duplication.

Similarly, a flag that conditionally accept some rules can be quite tricky to 
hey right with the backtracking and those require some extra metagrammar so 
adding the flag is not super unreadable. 

In my humble opinion, I think is far simpler to reject this after the parsing 
is done, either in the compiler or the symbol table.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

There is no “expr-without-await-or-async-for” subgrammar, but "await" and 
asynchronous comprehensions are invalid in synchronous functions. There should 
be similar flag for annotations in symtable.c.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Guido van Rossum]

Guido van Rossum  added the comment:

I don’t really care how you do it, I just thought having a whole 
“expr-without-walrus-or-yield” subgrammar would be tedious.

If there is a way to have the “in-an-annotation” flag set during parsing that 
may be better, I don’t know.

Maybe Lysandros has an idea?

--
nosy: +Guido.van.Rossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

> That can be done after the PEG grammar has accepted them

But wouldn't we still end up with maintaining a custom flag to see if we are in 
annotation (e.g a: Something((yield) + 2)) and act upon that which would seem 
to do a bit messy in the grammar actions rather than just throwing syntax 
errors from switch/cases inside of symbol table?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Guido van Rossum]

Guido van Rossum  added the comment:

So shouldn't we just rule out some specific bits of syntax in annotations? That 
can be done after the PEG grammar has accepted them, otherwise we'd end up 
having a whole new expression-like grammar section that excludes them. I think 
this is the list we should exclude, right?

- yield [from]
- walrus
- await (?)

I agree await technically doesn't need to be in this list, so maybe we 
shouldn't explicitly exclude it -- it's no different than writing

def f(x: open(filename).read()):
...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Larry Hastings]

Larry Hastings  added the comment:

Wouldn't it be easier to just throw an exception in the stringizing code?  I 
note that there are dedicated functions to generate walrus, yield, yield from, 
and await in Python/ast_unparse.c.  In theory this is a general-purpose 
facility, but in practice it's not part of the limited API, and it's called 
from literally one place, the "stringize this annotation" call site inside 
compile.c.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

> This was easy in my 649 branch; walrus adds locals, and yield / yield from 
> make it a generator.  So the code raises an error if the generated 
> annotations code object has locals or is a generator.  I don't think I had to 
> do anything special to prohibit await, because that's only valid in a 
> function declared "async def", which annotations code objects are not.

The implementation I have right now just adds a new state to the symbol table, 
and raises error in the visitors (like switch Yield, YieldFrom etc).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Larry Hastings]

Larry Hastings  added the comment:

I think stringized annotations should prohibit the same things PEP 649 
prohibits: walrus, yield / yield from, and await.

This was easy in my 649 branch; walrus adds locals, and yield / yield from make 
it a generator.  So the code raises an error if the generated annotations code 
object has locals or is a generator.  I don't think I had to do anything 
special to prohibit await, because that's only valid in a function declared 
"async def", which annotations code objects are not.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

I've posted an entry on python-dev to collect comments about how we should act 
(whether just act them as strings on the symbol table or just forbid them 
completely): 
https://mail.python.org/archives/list/python-...@python.org/thread/5NQH5GURAYW6XINPBO6VSF45CWLQOHUQ/

Since we are almost days away from the cut, I will probably just submit a 
proper patch this weekend where we can debate more through reviews but first I 
need to know what kind of action we intend to take. (this all assumes PR 23952 
is goes in).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Guido van Rossum]

Guido van Rossum  added the comment:

I'm not keen on prohibiting nonlocal variable reference in annotations,
since I can imagine uses for this (e.g. in tests). I'm not too worried
about keeping the cell alive "forever", since with Larry's PEP 649 their
lifetime would end when the object containing the annotation ends.

Someone should look into how Batuhan's super()-in-annotation example would
behave under Larry's PEP 649. (Using super() in the magical function Larry
proposes to generate might not work anyway, since super()'s own magical
powers don't work in a nested function.)

Re: issue42737, not sure what to do for that, since neither PEP 563 nor PEP
649 gives a way to access those annotations. Then again, that's also true
for annotations on local variables. So maybe we can treat them the same?
Type checkers won't care, and runtime uses of annotations (whether as type
hints or otherwise) won't have access..

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

> The use of nonlocal variables in an annotation will be a syntax error in 
> Python 3.10

What is the reasoning for forbidding nonlocal variables 
(https://bugs.python.org/msg383659), can't we just treat them like regular 
variables and leave the value to whom needed to deal with resolving the scope? 

Also, you should preferably clarify other cases regarding the symbol table 
interaction of annotations. For example this case:

Here is one;
class T:
def foo(self):
a: super().bar() = x
print(T.foo.__closure__)

And if we are on the road to writing a PEP, maybe we should somehow squeeze 
issue 42737 somewhere, since the annotations for complex targets are still 
evaluated and this makes everything a bit problematic and inconsistent. If you 
need collaboration on the PEP, just let me know (isidenti...@gmail.com) (I have 
a patch ready to go for the symbol table to both make annotations ineffective 
and forbid this stuff but was waiting for issue 42737)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Guido van Rossum]

Guido van Rossum  added the comment:

It'll probably be quicker if you leave me out of the loop (feel free to quote 
me though). I am currently quite overwhelmed with PEP-sized discussions. I 
expect that the SC can rule on this quickly (just use their tracker to send 
them the PEP).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Mark Shannon]

Mark Shannon  added the comment:

Draft PEP here
https://github.com/markshannon/peps/blob/annotation-syntax-errors/pep-.rst

Guido, would you like to be co-author as it was your idea to make these things 
a syntax error?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Guido van Rossum]

Guido van Rossum  added the comment:

Oh, you’re right about walrus. And I don’t actually understand async generator 
expressions.

This suggests that we definitely need a PEP. :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Does not walrus affect the meaning of variable? And await affects the meaning 
of generator expression.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Guido van Rossum]

Guido van Rossum  added the comment:

I wouldn’t have thought you’d need a PEP for this but if you want to write one 
that sounds like the right thing.

Re: async and walrus: I think those are different, their presence doesn’t 
affect the meaning of the function like yield. We can’t hope to prevent side 
effects syntactically.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Should not "await" and "async for" (in comprehesions) and ":=" be forbidden too?

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Mark Shannon]

Mark Shannon  added the comment:

What's the process for making a decision on whether to make 'yield' in an 
annotation a syntax error?

As a language change it should have a PEP, IMO.
The PEP will be short, and shouldn't need a long-winded acceptance process.
I just think that a PEP is more visible to the community than the bug tracker.

I'm happy to write the PEP.

--
nosy: +larry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Guido van Rossum]

Guido van Rossum  added the comment:

Yield in an annotation should be a syntax error.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Mark Shannon]

Mark Shannon  added the comment:

I've also opened #42837 which is about fixing the symbol table, so that it is 
correct w.r.t. to current behavior.

I'd like to fix it ASAP as the compiler should be able to rely on the symbol 
table being correct.

Of course, once we have decided what the behavior should be, then it may need 
to be updated again.

I'm inclined to agree that 'yield' in an annotation should probably be a syntax 
error, but I haven't put much thought into to.

--
nosy: +Mark.Shannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

One thing to note here, currently Pablo and I are trying to bring annotation 
unparsing from the compiler to the parser in issue 41967. If we do so, the 
annotations won't cause any side effects on the symbol table generation.

--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

I have a patch ready to go, but I'd prefer to block this issue until issue 
42737 is resolved/decided.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Guido van Rossum]

Guido van Rossum  added the comment:

Okay, let's do it. It should probably a post-parse check (before we invoke the 
bytecode compiler but after we have the AST in hand).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Lysandros Nikolaou]

Lysandros Nikolaou  added the comment:

I concur with Guido. I feel that making this an error is the best alternative 
we have.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Guido van Rossum]

Guido van Rossum  added the comment:

If we simply ignored yield in the annotation, wouldn't we have the problem that

def f(a: (yield)): pass

silently changes from being a generator (in 3.9) to not being a generator (in 
3.10)? That would be bad. I'd rather make this an error still. (But for 
nonlocals, not processing sounds fine.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

> So I don't feel as strongly about that.

Just to note, since I believe the solution for all this might be the same (not 
processing annotations at all, since they will be compiled to strings in the 
later stage). If we go down on that route, it will be simpler but we won't be 
able to raise SyntaxError's for "a: (yield)" / "b: (await/yield from x)". 

By the way, this is what happens if you try to use get_type_hints on a function 
where an argument is (yield):

>>> typing.get_type_hints(a)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/typing.py", line 537, in __init__
code = compile(arg, '', 'eval')
  File "", line 1
SyntaxError: 'yield' outside function

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python3.10/typing.py", line 1490, in get_type_hints
value = ForwardRef(value)
  File "/usr/local/lib/python3.10/typing.py", line 539, in __init__
raise SyntaxError(f"Forward reference must be an expression -- got {arg!r}")
SyntaxError: Forward reference must be an expression -- got '(yield)'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Guido van Rossum]

Guido van Rossum  added the comment:

The difference is that that just causes a slight inefficiency, while processing 
'yield' makes the function a generator. So I don't feel as strongly about that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

This is another side effect of processing annotations (at the symbol table 
construction stage) (and I would assume there are a few more cases like this);

def foo():
outer_var = 1

def bar():
inner_var: outer_var = T

return bar

inner = foo()
print(inner.__closure__)

In theory, there shouldn't be any cells / references to the variables from 
outer scope, but since we process the entry for the annotation and record 
`outer_var` as a free var it is listed here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Guido van Rossum]

Change by Guido van Rossum :


--
nosy: +lys.nikolaou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Guido van Rossum]

Guido van Rossum  added the comment:

Ouch.  I think we should generate a SyntaxError if yield [from] is used in an 
annotation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42725] PEP 563: Should the behavior change for yield/yield from's

[Batuhan Taskaya]

New submission from Batuhan Taskaya :

Since the annotations are processed just like all other expressions in the 
symbol table, the generated entries for functions etc. This would result with 

def foo():
for number in range(5):
foo: (yield number)
return number

foo()

returning a generator / coroutine (depending on yield/yield from/await usage). 
Is this something we want to keep or maybe tweak the symbol table generator to 
not to handle annotations (since there are also more subtle issues regarding 
analysis of cell / free vars).

--
messages: 383647
nosy: BTaskaya, gvanrossum
priority: normal
severity: normal
status: open
title: PEP 563: Should the behavior change for yield/yield from's

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com