Re: [Python-Dev] Generator objects and list comprehensions?
> On Feb 2, 2017, at 2:17 AM, Anders Munch wrote: > > Give Python 2 a little more credit. We are, it told you what your issue was: yield outside a function. Consider: >>> def f(): ... l = [(yield 1) for x in range(10)] ... print(l) >>> gen = f() >>> for i in range(11): ... gen.send(i or None) ... 1 1 1 1 1 1 1 1 1 1 [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] But this is a very convoluted "feature" and likely people don't expect *this* to be what's happening. - Ł ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generator objects and list comprehensions?
Craig Rodrigues : > Make this return a list on Python 3, like in Python 2: [(yield 1) for x in > range(10)] Give Python 2 a little more credit. Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> [(yield 1) for x in range(10)] File "", line 1 SyntaxError: 'yield' outside function regards, Anders ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generator objects and list comprehensions?
On 30 January 2017 at 19:05, Brett Cannon wrote: > On Sun, 29 Jan 2017 at 16:39 Craig Rodrigues wrote: >> I'm OK with either approach. Leaving things the way they are in Python 3 >> is no good, IMHO. > > My vote is it be a SyntaxError since you're not getting what you expect from > the syntax. I'd agree that's a sensible place for us to end up, as any code relying on the current behaviour is really too clever to be maintainable. In terms of getting there, we'll likely want: - SyntaxWarning or DeprecationWarning in 3.7 - Py3k warning in 2.7.x - SyntaxError in 3.8 Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generator objects and list comprehensions?
On Sun, 29 Jan 2017 at 16:39 Craig Rodrigues wrote: > On Thu, Jan 26, 2017 at 4:09 AM, Ivan Levkivskyi > wrote: > > > > Concerning list/set/dict comprehensions, I am much more in favor of making > comprehensions simply equivalent to for-loops (more or less like you > proposed using yield from). The only reason to introduce auxiliary function > scope was to prevent the loop variables from leaking outside comprehensions. > Formally, this is indeed backward incompatible, but I doubt many people > depend on the current counter-intuitive behavior. > > Concerning generator expressions, probably it is indeed better to simply > prohibit yield inside them. > > > Thank you to everyone who responded to my post and provided excellent > analysis. > > For Python, I don't know what the best way to proceed is: > > OPTION 1 > > > Make a SyntaxError: [(yield 1) for x in range(10)] > and update the documentation to explain that this is an invalid construct. > > This would have certainly helped me identify the source of the problem as > I tried porting buildbot 0.9 to Python 3. > > However, while not very common, there is Python 2.x code that uses that. > I found these cases in the buildbot code which I changed so as to work on > Python 2 and 3: > > https://github.com/buildbot/buildbot/pull/2661 > https://github.com/buildbot/buildbot/pull/2673 > > > OPTION 2 > = > Make this return a list on Python 3, like in Python 2: [ > (yield 1) for x in range(10)] > > As pointed out by others on the this mailing list, there are some > problems associated with that. I don't know if there are many Python 2 > codebases out there > with this construct, but it would be nice to have one less Python 2 -> 3 > porting gotcha. > > > I'm OK with either approach. Leaving things the way they are in Python 3 > is no good, IMHO. > My vote is it be a SyntaxError since you're not getting what you expect from the syntax. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generator objects and list comprehensions?
On Thu, Jan 26, 2017 at 4:09 AM, Ivan Levkivskyi wrote: > > > Concerning list/set/dict comprehensions, I am much more in favor of making > comprehensions simply equivalent to for-loops (more or less like you > proposed using yield from). The only reason to introduce auxiliary function > scope was to prevent the loop variables from leaking outside comprehensions. > Formally, this is indeed backward incompatible, but I doubt many people > depend on the current counter-intuitive behavior. > > Concerning generator expressions, probably it is indeed better to simply > prohibit yield inside them. > > Thank you to everyone who responded to my post and provided excellent analysis. For Python, I don't know what the best way to proceed is: OPTION 1 Make a SyntaxError: [(yield 1) for x in range(10)] and update the documentation to explain that this is an invalid construct. This would have certainly helped me identify the source of the problem as I tried porting buildbot 0.9 to Python 3. However, while not very common, there is Python 2.x code that uses that. I found these cases in the buildbot code which I changed so as to work on Python 2 and 3: https://github.com/buildbot/buildbot/pull/2661 https://github.com/buildbot/buildbot/pull/2673 OPTION 2 = Make this return a list on Python 3, like in Python 2: [ (yield 1) for x in range(10)] As pointed out by others on the this mailing list, there are some problems associated with that. I don't know if there are many Python 2 codebases out there with this construct, but it would be nice to have one less Python 2 -> 3 porting gotcha. I'm OK with either approach. Leaving things the way they are in Python 3 is no good, IMHO. -- Craig ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generator objects and list comprehensions?
On 26 January 2017 at 00:53, Nathaniel Smith wrote: > > I'm not sure this is actually a good idea, given the potential for > ambiguity and backwards compatibility considerations -- I'm kind of leaning > towards the deprecate/error option on balance :-). But it's an option that > would probably be less confusing than the status quo, and would make it > easier to write the kind of straddling code the original poster was trying > to write. > > Concerning list/set/dict comprehensions, I am much more in favor of making comprehensions simply equivalent to for-loops (more or less like you proposed using yield from). The only reason to introduce auxiliary function scope was to prevent the loop variables from leaking outside comprehensions. Formally, this is indeed backward incompatible, but I doubt many people depend on the current counter-intuitive behavior. Concerning generator expressions, probably it is indeed better to simply prohibit yield inside them. -- Ivan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generator objects and list comprehensions?
On Jan 25, 2017 8:16 AM, "Joe Jevnik" wrote:
List, set, and dict comprehensions compile like:
# input
result = [expression for lhs in iterator_expression]
# output
def comprehension(iterator):
out = []
for lhs in iterator:
out.append(expression)
return out
result = comprehension(iter(iterator_expression))
When you make `expression` a `yield` the compiler thinks that
`comprehension` is a generator function instead of a normal function.
We can manually translate the following comprehension:
result = [(yield n) for n in (0, 1)]
def comprehension(iterator):
out = []
for n in iterator:
# (yield n) as an expression is the value sent into this generator
out.append((yield n))
return out
result = comprehension(iter((0, 1)))
We can see this in the behavior of `send` on the resulting generator:
In [1]: g = [(yield n) for n in (0, 1)]
In [2]: next(g)
Out[2]: 0
In [3]: g.send('hello')
Out[3]: 1
In [4]: g.send('world')
---
StopIteration Traceback (most recent call last)
in ()
> 1 g.send('world')
StopIteration: ['hello', 'world']
The `return out` gets translated into `raise StopIteration(out)` because
the code is a generator.
The bytecode for this looks like:
In [5]: %%dis
...: [(yield n) for n in (0, 1)]
...:
1 0 LOAD_CONST 0 ( at
0x7f4bae68eed0, file "", line 1>)
3 LOAD_CONST 1 ('')
6 MAKE_FUNCTION0
9 LOAD_CONST 5 ((0, 1))
12 GET_ITER
13 CALL_FUNCTION1 (1 positional, 0 keyword pair)
16 POP_TOP
17 LOAD_CONST 4 (None)
20 RETURN_VALUE
.
---
1 0 BUILD_LIST 0
3 LOAD_FAST0 (.0)
>>6 FOR_ITER13 (to 22)
9 STORE_FAST 1 (n)
12 LOAD_FAST1 (n)
15 YIELD_VALUE
16 LIST_APPEND 2
19 JUMP_ABSOLUTE6
>> 22 RETURN_VALUE
In `` you can see us create the function, call `iter` on
`(0, 1)`, and then call `(iter(0, 1))`. In `` you can
see the loop like we had above, but unlike a normal comprehension we have a
`YIELD_VALUE` (from the `(yield n)`) in the comprehension.
The reason that this is different in Python 2 is that list comprehension,
and only list comprehensions, compile inline. Instead of creating a new
function for the loop, it is done in the scope of the comprehension. For
example:
# input
result = [expression for lhs in iterator_expression]
# output
result = []
for lhs in iterator_expression:
result.append(lhs)
This is why `lhs` bleeds out of the comprehension. In Python 2, adding
making `expression` a `yield` causes the _calling_ function to become a
generator:
def f():
[(yield n) for n in range(3)]
# no return
# py2
>>> type(f())
generator
# py3
>> type(f())
NoneType
In Python 2 the following will even raise a syntax error:
In [5]: def f():
...: return [(yield n) for n in range(3)]
...:
...:
File "", line 2
return [(yield n) for n in range(3)]
SyntaxError: 'return' with argument inside generator
Generator expressions are a little different because the compilation
already includes a `yield`.
# input
(expression for lhs in iterator_expression)
# output
def genexpr(iterator):
for lhs in iterator:
yield expression
You can actually abuse this to write a cute `flatten` function:
`flatten = lambda seq: (None for sub in seq if (yield from sub) and False)`
because it translates to:
def genexpr(seq):
for sub in seq:
# (yield from sub) as an expression returns the last sent in value
if (yield from sub) and False:
# we never enter this branch
yield None
That was a long way to explain what the problem was. I think that that
solution is to stop using `yield` in comprehensions because it is
confusing, or to make `yield` in a comprehension a syntax error.
Another option would be to restore the py2 behavior by inserting an
implicit 'yield from' whenever the embedded expression contains a yield. So
in your example where
result = [(yield n) for n in (0, 1)]
currently gets expanded to
result = comprehension(iter((0, 1)))
it could notice that the expanded function 'comprehension' is a generator,
and emit code like this instead:
result = yield from comprehension(iter((0, 1)))
At least, this would work for list/dict/set comprehensions; not so much for
generator expressions. I assume this is basically how 'await' in
comprehensions works in 3.6.
I'm not sure this is actually a good idea, given the potential for
ambiguity and backwards compatibility considerations -- I'm kind of leaning
towards the deprecate/error option on balance :-).
Re: [Python-Dev] Generator objects and list comprehensions?
On Wed, Jan 25, 2017 at 6:28 AM, Joe Jevnik wrote: > That was a long way to explain what the problem was. I think that that > solution is to stop using `yield` in comprehensions because it is > confusing, or to make `yield` in a comprehension a syntax error. > > Thanks for the very clear explanation. Would an addition to the documentation for 3.6 giving an abbreviated version help while the devs consider whether any changes are appropriate for 3.7? Steve Holden ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generator objects and list comprehensions?
On 25.01.2017 07:28, Joe Jevnik wrote: That was a long way to explain what the problem was. I think that that solution is to stop using `yield` in comprehensions because it is confusing, or to make `yield` in a comprehension a syntax error. Same here; mixing comprehensions and yield (from) can't be explained easily. A SyntaxError would be most appropriate. Regards, Sven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generator objects and list comprehensions?
List, set, and dict comprehensions compile like:
# input
result = [expression for lhs in iterator_expression]
# output
def comprehension(iterator):
out = []
for lhs in iterator:
out.append(expression)
return out
result = comprehension(iter(iterator_expression))
When you make `expression` a `yield` the compiler thinks that
`comprehension` is a generator function instead of a normal function.
We can manually translate the following comprehension:
result = [(yield n) for n in (0, 1)]
def comprehension(iterator):
out = []
for n in iterator:
# (yield n) as an expression is the value sent into this generator
out.append((yield n))
return out
result = comprehension(iter((0, 1)))
We can see this in the behavior of `send` on the resulting generator:
In [1]: g = [(yield n) for n in (0, 1)]
In [2]: next(g)
Out[2]: 0
In [3]: g.send('hello')
Out[3]: 1
In [4]: g.send('world')
---
StopIteration Traceback (most recent call last)
in ()
> 1 g.send('world')
StopIteration: ['hello', 'world']
The `return out` gets translated into `raise StopIteration(out)` because
the code is a generator.
The bytecode for this looks like:
In [5]: %%dis
...: [(yield n) for n in (0, 1)]
...:
1 0 LOAD_CONST 0 ( at
0x7f4bae68eed0, file "", line 1>)
3 LOAD_CONST 1 ('')
6 MAKE_FUNCTION0
9 LOAD_CONST 5 ((0, 1))
12 GET_ITER
13 CALL_FUNCTION1 (1 positional, 0 keyword pair)
16 POP_TOP
17 LOAD_CONST 4 (None)
20 RETURN_VALUE
.
---
1 0 BUILD_LIST 0
3 LOAD_FAST0 (.0)
>>6 FOR_ITER13 (to 22)
9 STORE_FAST 1 (n)
12 LOAD_FAST1 (n)
15 YIELD_VALUE
16 LIST_APPEND 2
19 JUMP_ABSOLUTE6
>> 22 RETURN_VALUE
In `` you can see us create the function, call `iter` on
`(0, 1)`, and then call `(iter(0, 1))`. In `` you can
see the loop like we had above, but unlike a normal comprehension we have a
`YIELD_VALUE` (from the `(yield n)`) in the comprehension.
The reason that this is different in Python 2 is that list comprehension,
and only list comprehensions, compile inline. Instead of creating a new
function for the loop, it is done in the scope of the comprehension. For
example:
# input
result = [expression for lhs in iterator_expression]
# output
result = []
for lhs in iterator_expression:
result.append(lhs)
This is why `lhs` bleeds out of the comprehension. In Python 2, adding
making `expression` a `yield` causes the _calling_ function to become a
generator:
def f():
[(yield n) for n in range(3)]
# no return
# py2
>>> type(f())
generator
# py3
>> type(f())
NoneType
In Python 2 the following will even raise a syntax error:
In [5]: def f():
...: return [(yield n) for n in range(3)]
...:
...:
File "", line 2
return [(yield n) for n in range(3)]
SyntaxError: 'return' with argument inside generator
Generator expressions are a little different because the compilation
already includes a `yield`.
# input
(expression for lhs in iterator_expression)
# output
def genexpr(iterator):
for lhs in iterator:
yield expression
You can actually abuse this to write a cute `flatten` function:
`flatten = lambda seq: (None for sub in seq if (yield from sub) and False)`
because it translates to:
def genexpr(seq):
for sub in seq:
# (yield from sub) as an expression returns the last sent in value
if (yield from sub) and False:
# we never enter this branch
yield None
That was a long way to explain what the problem was. I think that that
solution is to stop using `yield` in comprehensions because it is
confusing, or to make `yield` in a comprehension a syntax error.
On Wed, Jan 25, 2017 at 12:38 AM, Craig Rodrigues
wrote:
> Hi,
>
> Glyph pointed this out to me here: http://twistedmatrix.
> com/pipermail/twisted-python/2017-January/031106.html
>
> If I do this on Python 3.6:
>
> >> [(yield 1) for x in range(10)]
> at 0x10cd210f8>
>
> If I understand this: https://docs.python.org/
> 3/reference/expressions.html#list-displays
> then this is a list display and should give a list, not a generator object.
> Is there a bug in Python, or does the documentation need to be updated?
>
> --
> Craig
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> joe%40quantopian.com
>
>
___
Python-D
Re: [Python-Dev] Generator objects and list comprehensions?
On 25 January 2017 at 07:01, Chris Angelico wrote: > >>> [(yield 1) for x in range(10)] > > at 0x10cd210f8> > This is an old bug, see e.g. http://bugs.python.org/issue10544 The ``yield`` inside comprehensions is bound to the auxiliary function. Instead it should be bound to an enclosing function, like it is done for ``await``. The behaviour of ``await`` in comprehensions is intuitive (since it is simply equivalent to a for-loop): >>> async def f(i): ... return i >>> async def g_for(): ... lst = [] ... for i in range(5): ... lst.append(await f(i)) ... print(lst) >>> g_for().send(None) [0, 1, 2, 3, 4] Traceback (most recent call last): File "", line 1, in StopIteration >>> async def g_comp(): ... print([await f(i) for i in range(5)]) >>> g_comp().send(None)# exactly the same as g_for [0, 1, 2, 3, 4] Traceback (most recent call last): File "", line 1, in StopIteration While current behaviour of ``yield`` in comprehensions is confusing: >>> def c_for(): ... lst = [] ... for i in range(5): ... lst.append((yield i)) ... print(lst) >>> c_for().send(None) 0 >>> c_for().send(None) 1 # etc. >>> def c_comp(): ... print([(yield i) for i in range(5)]) >>> c_comp().send(None) # Naively this should be equivalent to the above, but... . at 0x7f1fd1faa630> Traceback (most recent call last): File "", line 1, in AttributeError: 'NoneType' object has no attribute 'send' I promised myself to write a patch, but didn't have time for this yet. I hope I will do this at some point soon. -- Ivan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Generator objects and list comprehensions?
On Wed, Jan 25, 2017 at 4:38 PM, Craig Rodrigues wrote: > > Glyph pointed this out to me here: > http://twistedmatrix.com/pipermail/twisted-python/2017-January/031106.html > > If I do this on Python 3.6: > >>> [(yield 1) for x in range(10)] > at 0x10cd210f8> > > If I understand this: > https://docs.python.org/3/reference/expressions.html#list-displays > then this is a list display and should give a list, not a generator object. > Is there a bug in Python, or does the documentation need to be updated? That looks like an odd interaction between yield expressions and list comps. Since a list comprehension is actually implemented as a nested function, your code actually looks more-or-less like this: >>> def (iter): result = [] for x in iter: result.append((yield 1)) return result >>> (iter(range(10)) at 0x10cd210f8> This function is a generator, and calling it returns what you see above. If you step that iterator, it'll yield 1 ten times, and then raise StopIteration with the resulting list. Based on a cursory examination of the issue at hand, I think what you're working with might be functioning as a coroutine? If so, you may find that using "await" instead of "yield" dodges the problem, as it won't turn the list comp into a generator. But I can't be 100% certain of that. (Also, that would definitely stop you from having single-codebase 2.7/3.x code.) ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
