[issue10544] yield expression inside generator expression does nothing

Serhiy Storchaka added the comment: New changeset 07ca9afaa8768b44baf816b4998d209ed3e0088f by Serhiy Storchaka in branch 'master': bpo-10544: Disallow "yield" in comprehensions and generator expressions. (GH-4564)

[issue10544] yield expression inside generator expression does nothing

Change by Serhiy Storchaka : -- status: open -> closed ___ Python tracker ___

[issue10544] yield expression inside generator expression does nothing

Serhiy Storchaka added the comment: New changeset 65d1887170fb278c10a836e9e4319cae4707f524 by Serhiy Storchaka in branch '2.7': [2.7] bpo-10544: Deprecate "yield" in comprehensions and generator expressions in Py3k mode. (GH-4579) (#4676)

[issue10544] yield expression inside generator expression does nothing

Serhiy Storchaka added the comment: PR 4676 backports warnings to 2.7 in Py3k mode. -- status: closed -> open ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Change by Serhiy Storchaka : -- pull_requests: +4584 ___ Python tracker ___ ___

[issue10544] yield expression inside generator expression does nothing

Nick Coghlan added the comment: With Serhiy's patch merged, I'm marking this as resolved. Thanks all! https://bugs.python.org/issue32189 is the follow-up issue to turn the warning into an unconditional SyntaxError in 3.8. -- resolution: -> fixed stage: patch

[issue10544] yield expression inside generator expression does nothing

Nick Coghlan added the comment: New changeset 73a7e9b10b2ec9636e3c6396cf7b3695f8ed1856 by Nick Coghlan (Serhiy Storchaka) in branch 'master': bpo-10544: Deprecate "yield" in comprehensions and generator expressions. (GH-4579)

[issue10544] yield expression inside generator expression does nothing

Guido van Rossum added the comment: OK, great. -- ___ Python tracker ___ ___

[issue10544] yield expression inside generator expression does nothing

Nick Coghlan added the comment: As far as I'm aware, there's nothing that specifically promises these constructs will do anything in Py3 at all - the existing behaviour is just an accident of implementation arising from the way nested scopes and yield expressions interact

[issue10544] yield expression inside generator expression does nothing

Guido van Rossum added the comment: After the tiresome debate I am happy to see this just as a "what's new" entry rather than soliciting more debate with a PEP. (However there may be some existing PEP that suggests it should work? That PEP should be amended with a note that

[issue10544] yield expression inside generator expression does nothing

Serhiy Storchaka added the comment: Note that the DeprecationWarning exception is replaced with a SyntaxError to get a more accurate error report. $ cat yield-gen.py def f(): return ((yield x) for x in range(3)) $ ./python -Wd yield-gen.py yield-gen.py:2:

[issue10544] yield expression inside generator expression does nothing

Nick Coghlan added the comment: Guido, should we write this change up as a PEP, or are you happy to just cover it as a section in the What's New document for 3.7? -- ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Change by Serhiy Storchaka : -- pull_requests: +4506 ___ Python tracker ___ ___

[issue10544] yield expression inside generator expression does nothing

Guido van Rossum added the comment: Great. It should be a DeprecationWarning, since we're planning to disallow it completely, right? IIRC SyntaxWarning is for syntax that we can't deprecate. -- ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Nick Coghlan added the comment: Cool, if you're OK with that behaviour, it actually makes this a lot easier, since it means: 1. Serhiy's patch is already sufficient for the final hard compatibility break 2. It can be readily adapted to emit either DeprecationWarning or

[issue10544] yield expression inside generator expression does nothing

Guido van Rossum added the comment: To clarify, the outermost iterator is marked here: [func(x, y) for x in for y in ] and it is evaluated before the comprehension proper is started. This is just part of how the language works (it's a similar rule as "in an assignment

[issue10544] yield expression inside generator expression does nothing

Guido van Rossum added the comment: No. On Nov 25, 2017 18:01, "Nick Coghlan" wrote: > > Nick Coghlan added the comment: > > Serhiy's PR now implements the "Prohibit yield & yield from in generator > expressions and

[issue10544] yield expression inside generator expression does nothing

Nick Coghlan added the comment: Serhiy's PR now implements the "Prohibit yield & yield from in generator expressions and comprehensions" approach discussed on python-dev (currently as a hard SyntaxError, but it could be amended to be a warning instead without too much

[issue10544] yield expression inside generator expression does nothing

Change by Serhiy Storchaka : -- pull_requests: +4492 stage: needs patch -> patch review ___ Python tracker ___

[issue10544] yield expression inside generator expression does nothing

Change by Ivan Levkivskyi : -- nosy: -levkivskyi ___ Python tracker ___ ___

[issue10544] yield expression inside generator expression does nothing

Serhiy Storchaka added the comment: Here is a sample of the implementation of the Nick's idea. -- keywords: +patch Added file: https://bugs.python.org/file47296/yield-in-comprehensions.diff ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Guido van Rossum added the comment: No to both. See python-dev. -- ___ Python tracker ___

[issue10544] yield expression inside generator expression does nothing

Nick Coghlan added the comment: I realised that even without modifying the compiler first, I could illustrate the proposed `yield from` based resolution for the comprehension case by way of explicit yield from clauses: ``` def get_gen_result(gen, inputs): try:

[issue10544] yield expression inside generator expression does nothing

Nick Coghlan added the comment: Given the direction of the python-dev thread, should we split this question into two issues? Issue 1: a yield expression inside a comprehension changes the type of the expression result (returning a generator-iterator instead of the

[issue10544] yield expression inside generator expression does nothing

Change by Armin Rigo : -- nosy: -arigo ___ Python tracker ___ ___

[issue10544] yield expression inside generator expression does nothing

Ivan Levkivskyi added the comment: > How about not posting about this topic for 24 hours. OK, makes sense :-) -- ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Guido van Rossum added the comment: I think we all need to calm down a bit. How about not posting about this topic for 24 hours. -- ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Ivan Levkivskyi added the comment: Guido, I am not sure about the complete removal, this is probably to radical. What I think we are missing more detailed docs that would be clear about the corner cases. (I already mentioned this in https://bugs.python.org/issue32113)

[issue10544] yield expression inside generator expression does nothing

Yury Selivanov added the comment: > Yury OK, sorry then this is a misunderstanding from my side. NP. Again, sorry if I sounded that way to you. -- ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Ivan Levkivskyi added the comment: Yury OK, sorry then this is a misunderstanding from my side. -- ___ Python tracker ___

[issue10544] yield expression inside generator expression does nothing

Yury Selivanov added the comment: >> Do you understand the difference? > Yury, sorry, but what is your problem? Have I said something about that this > is bad. Your tone is clearly insulting, and this is not the first time. Maybe > it makes sense to have some respect?

[issue10544] yield expression inside generator expression does nothing

Ivan Levkivskyi added the comment: > Do you understand the difference? Yury, sorry, but what is your problem? Have I said something about that this is bad. Your tone is clearly insulting, and this is not the first time. Maybe it makes sense to have some respect?

[issue10544] yield expression inside generator expression does nothing

Yury Selivanov added the comment: > Well, after all I am thinking maybe it is indeed makes sense to ban `yield` > inside both sync/async and both comprehensions/generator expressions. I agree, as I can't imagine a real use case for a = ((yield a) for a in as) which

[issue10544] yield expression inside generator expression does nothing

Guido van Rossum added the comment: I honestly think we went too far here. Asynchronous generators are still provisional. Though the PEP is silent about this, the acceptance notice is clear: https://mail.python.org/pipermail/python-dev/2016-September/146267.html. I propose to

[issue10544] yield expression inside generator expression does nothing

Ivan Levkivskyi added the comment: Well, after all I am thinking maybe it is indeed makes sense to ban `yield` inside both sync/async and both comprehensions/generator expressions. Since we already have a smörgåsbord of intuitive behaviors. This, _together_ with

[issue10544] yield expression inside generator expression does nothing

Yury Selivanov added the comment: > ... but [await x for x in xs] is still valid _only_ inside async def. Yes, because it's computed right where it is defined. a = [x for x in xs] # `a` is a list object a = (x for x in xs) # `a` is a generator Do you understand the

[issue10544] yield expression inside generator expression does nothing

Ivan Levkivskyi added the comment: ... but [await x for x in xs] is still valid _only_ inside async def. -- ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Yury Selivanov added the comment: It's also important to note, that in 3.7, this is legal: def foo(): return (await x for x in xs) Note that 'foo' is a regular function, not 'async def'. -- ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Yury Selivanov added the comment: > Yury explained why this happens in https://bugs.python.org/issue32113 It happens because '(x for x in xs)' creates a synchronous generator. So when there's an 'await' in it, it creates an asynchronous generator. --

[issue10544] yield expression inside generator expression does nothing

Ivan Levkivskyi added the comment: (Of course there should be not [1, 2] in the argument, but a list of some awaitables, otherwise there will be an error later.) -- ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Ivan Levkivskyi added the comment: > Remind me what happens when you use `await` in a generator expression that > survives the async function's scope? Awaiting on f([1, 2]) will result in an async generator (even though yield never appears here). Yury explained why this

[issue10544] yield expression inside generator expression does nothing

Guido van Rossum added the comment: Remind me what happens when you use `await` in a generator expression that survives the async function's scope? async def f(xs): return (await x for x in xs) -- ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Serhiy Storchaka added the comment: I concur with Ivan. It would be nice to me to disallow "yield" in comprehensions and generator expressions, but the fact that "await" in comprehensions already works right, and this behavior is intentional and useful, make me

[issue10544] yield expression inside generator expression does nothing

Serhiy Storchaka added the comment: Consider this as a sort of optimization. A function inlining. In general case we can't inline a function because it can be rebinded at any time, but in this particular case we call a just created function that doesn't have any

[issue10544] yield expression inside generator expression does nothing

Ivan Levkivskyi added the comment: Nick, > Yury took all this into account when designing the interaction between > `await` and comprehensions (which is why that's in a better state), but for > `yield` we inherited the existing behaviour of any other nested function.

[issue10544] yield expression inside generator expression does nothing

Serhiy Storchaka added the comment: This is straightforward. __result = []; __i = None try: for __i in range(2, 5): __result2 = []; __x = __y = None try: for __x in range(2): for __y in range(1, __i):

[issue10544] yield expression inside generator expression does nothing

Nick Coghlan added the comment: Sure, you can technically do that, but you'll have to rewrite most of the symtable pass in the compiler, since you won't be able to re-use the current recursive descent algorithms (which are designed to handle function scopes). That ran

[issue10544] yield expression inside generator expression does nothing

Nick Coghlan added the comment: Another slight variant to that test case to make sure the inner comprehension actually generates a closure reference in the current implementation: >>> [[x+y for x in range(2) for y in range(1, i)] for i in range(2, 5)] [[1, 2], [1,

[issue10544] yield expression inside generator expression does nothing

Serhiy Storchaka added the comment: I see nothing special in `[[x for x in range(1, i)] for i in range(2, 5)]`. This should be equivalent to: __result = []; __i = None try: for __i in range(2, 5): __result2 = []; __x = None try: for

[issue10544] yield expression inside generator expression does nothing

Nick Coghlan added the comment: Also see https://bugs.python.org/issue1660500 for the original Python 3.0 change to hide the iteration variable. While the test suite already covers some interesting scoping edge cases as result of that initial patch, I think one we're

[issue10544] yield expression inside generator expression does nothing

Nick Coghlan added the comment: "Just fix the issue" is easier said than done for the same reason that comprehensions were implemented the way they are now: lambda expressions still have to work. That is, we need to maintain the invariant that: [x for x in iterable]

[issue10544] yield expression inside generator expression does nothing

Change by Nick Coghlan : -- nosy: +ncoghlan ___ Python tracker ___ ___ Python-bugs-list

[issue10544] yield expression inside generator expression does nothing

Change by Ivan Levkivskyi : -- assignee: levkivskyi -> ___ Python tracker ___ ___

[issue10544] yield expression inside generator expression does nothing

Change by Serhiy Storchaka : -- nosy: +serhiy.storchaka versions: +Python 3.6, Python 3.7 -Python 3.5 ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Changes by Ivan Levkivskyi : -- assignee: -> levkivskyi ___ Python tracker ___ ___

[issue10544] yield expression inside generator expression does nothing

Changes by Jim Fasarakis-Hilliard : -- nosy: +Jim Fasarakis-Hilliard ___ Python tracker ___

[issue10544] yield expression inside generator expression does nothing

Armin Rigo added the comment: Let's see if the discussion goes anywhere or if this issue remains in limbo for the next 7 years. In the meantime, if I may humbly make a suggestion: whether the final decision is to give SyntaxError or change the semantics, one or a few intermediate versions

[issue10544] yield expression inside generator expression does nothing

Ivan Levkivskyi added the comment: > How about fixing CPython to raise SyntaxWarning or even SyntaxError? I think it is better to just fix the issue, i.e. make comprehensions be equivalent to for-loops even if they contain `yield`. (In particular this will lead to [(yield i) for i in

[issue10544] yield expression inside generator expression does nothing

Armin Rigo added the comment: Just to add my comment to this 7-years-old never-resolved issue: in PyPy 3.5, which behaves like Python 3.x in this respect, I made the following constructions give a warning. def wrong_listcomp(): return [(yield 42) for i in j] def

[issue10544] yield expression inside generator expression does nothing

Glyph Lefkowitz added the comment: OK, cool. So, long term, there will be a way to do this (suspend within a generator expression). Thanks for the pointer. -- ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Glyph Lefkowitz added the comment: (As far as awaiting on int, yes, I know how await works, I was focusing on the syntax.) -- ___ Python tracker ___

[issue10544] yield expression inside generator expression does nothing

Ivan Levkivskyi added the comment: Python 3.5 does not support this, you should use Python 3.6 (plus await x will fail when you will run the coroutine, since you cannot await on int). -- ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Glyph Lefkowitz added the comment: >>> async def foo(): ... bar = [await x for x in range(10)] File "", line 2 SyntaxError: 'await' expressions in comprehensions are not supported -- ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Ivan Levkivskyi added the comment: > Is the fact that 'await' produces a syntax error in this context the same bug > or a new one? What kind of SyntaxError? await outside an async function is prohibited, bare await is also prohibited. -- ___

[issue10544] yield expression inside generator expression does nothing

Glyph Lefkowitz added the comment: Is the fact that 'await' produces a syntax error in this context the same bug or a new one? -- nosy: +glyph ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Changes by A. Jesse Jiryu Davis je...@emptysquare.net: -- nosy: +emptysquare ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10544 ___ ___

[issue10544] yield expression inside generator expression does nothing

levkivskyi added the comment: I would like to add that since the introduction of asyncio module that heavily uses yield from syntax, binding of yield inside comprehensions/generator expressions could lead to unexpected results/confusing behavior. See for example this question on SO:

[issue10544] yield expression inside generator expression does nothing

Changes by Yury Selivanov yselivanov...@gmail.com: -- versions: +Python 3.5 -Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10544 ___

[issue10544] yield expression inside generator expression does nothing

Changes by Terry J. Reedy tjre...@udel.edu: -- nosy: -terry.reedy versions: -Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10544 ___

[issue10544] yield expression inside generator expression does nothing

Changes by Meador Inge mead...@gmail.com: -- stage: - needs patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10544 ___ ___ Python-bugs-list

[issue10544] yield expression inside generator expression does nothing

Changes by Daniel Shahaf pyt...@danielsh.fastmail.net: -- nosy: +danielsh ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10544 ___ ___

[issue10544] yield expression inside generator expression does nothing

Changes by Brett Cannon br...@python.org: -- nosy: -brett.cannon versions: +Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10544 ___

[issue10544] yield expression inside generator expression does nothing

Changes by Brett Cannon br...@python.org: -- versions: +Python 3.4 -Python 3.1 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10544 ___ ___

[issue10544] yield expression inside generator expression does nothing

Changes by Ed Campbell drescampb...@gmail.com: -- nosy: +esc24 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10544 ___ ___ Python-bugs-list

[issue10544] yield expression inside generator expression does nothing

Terry J. Reedy tjre...@udel.edu added the comment: #3267 did not expose endless loop possibility and was closed as won't fix. Rather than reopen that and close this and move nosy list back, I added to nosy list here. -- nosy: +brett.cannon, erickt, terry.reedy

[issue10544] yield expression inside generator expression does nothing

Georg Brandl ge...@python.org added the comment: FWIW, the endless loop possibility is not of issue here, and is simply an artifact of the specific generator function the OP uses. -- ___ Python tracker rep...@bugs.python.org

[issue10544] yield expression inside generator expression does nothing

Georg Brandl ge...@python.org added the comment: While the behavior is confusing, I don't think yield inside comprehensions should be disallowed. Rather, the fact that comprehensions have their own scope should be stated clearer. -- nosy: +georg.brandl

[issue10544] yield expression inside generator expression does nothing

Benjamin Peterson benja...@python.org added the comment: I think I can probably fix it, but it's debatable whether it should be done, since it'd make list comps more of quasi functions. -- nosy: +benjamin.peterson ___ Python tracker

[issue10544] yield expression inside generator expression does nothing

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: This discussion should probably be moved to python-dev. With tools like Twisted's inlineDefer or the Monocle package, there is a growing need to be able to use yield in complex expressions. Yet, that goes against the trend

[issue10544] yield expression inside generator expression does nothing

Guido van Rossum gu...@python.org added the comment: I think it is definitely wrong the way it works in 3.x. (Especially since it works as expected in 2.x.) I agree with Inyeol's preference of fixes: (1) make it work properly for listcomps as well as genexps, (2) if that's not possible,

[issue10544] yield expression inside generator expression does nothing

Guido van Rossum gu...@python.org added the comment: PS. Wasn't there a similar issue with something inside a genexp that raises StopIteration? Did we ever solve that? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10544

[issue10544] yield expression inside generator expression does nothing

Changes by Guido van Rossum gu...@python.org: -- assignee: gvanrossum - ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10544 ___ ___

[issue10544] yield expression inside generator expression does nothing

Changes by Alexander Belopolsky belopol...@users.sourceforge.net: -- nosy: +belopolsky ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10544 ___ ___

[issue10544] yield expression inside generator expression does nothing

Alexander Belopolsky belopol...@users.sourceforge.net added the comment: Isn't this the same issue as #3267? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10544 ___

[issue10544] yield expression inside generator expression does nothing

Guido van Rossum gu...@python.org added the comment: Yes it is, but I was never asked about it back then. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10544 ___

[issue10544] yield expression inside generator expression does nothing

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: Hmm, what an interesting and unexpected side-effect of the efforts to hide the loop induction variable. -- nosy: +rhettinger ___ Python tracker rep...@bugs.python.org

[issue10544] yield expression inside generator expression does nothing

New submission from Inyeol Lee inyeol@gmail.com: Simple coroutine with for loop works: def pack_a(): while True: L = [] for i in range(2): L.append((yield)) print(L) pa = pack_a() next(pa) pa.send(1) pa.send(2) [1, 2] If