Re: [Python-Dev] Generator objects and list comprehensions?

[Lukasz Langa]

> On Feb 2, 2017, at 2:17 AM, Anders Munch  wrote:
> 
> Give Python 2 a little more credit.

We are, it told you what your issue was: yield outside a function. Consider:

  >>> def f():
  ...   l = [(yield 1) for x in range(10)]
  ...   print(l)
  >>> gen = f()
  >>> for i in range(11):
  ...   gen.send(i or None)
  ...
  1
  1
  1
  1
  1
  1
  1
  1
  1
  1
  [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

But this is a very convoluted "feature" and likely people don't expect *this* 
to be what's happening.

- Ł
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generator objects and list comprehensions?

[Anders Munch]

Craig Rodrigues :
> Make this return a list on Python 3, like in Python 2:  [(yield 1) for x in 
> range(10)] 

Give Python 2 a little more credit.

Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on 
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> [(yield 1) for x in range(10)]
  File "", line 1
SyntaxError: 'yield' outside function

regards, Anders

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generator objects and list comprehensions?

[Nick Coghlan]

On 30 January 2017 at 19:05, Brett Cannon  wrote:
> On Sun, 29 Jan 2017 at 16:39 Craig Rodrigues  wrote:
>> I'm OK with either approach.  Leaving things the way they are in Python 3
>> is no good, IMHO.
>
> My vote is it be a SyntaxError since you're not getting what you expect from
> the syntax.

I'd agree that's a sensible place for us to end up, as any code
relying on the current behaviour is really too clever to be
maintainable.

In terms of getting there, we'll likely want:

- SyntaxWarning or DeprecationWarning in 3.7
- Py3k warning in 2.7.x
- SyntaxError in 3.8

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generator objects and list comprehensions?

[Brett Cannon]

On Sun, 29 Jan 2017 at 16:39 Craig Rodrigues  wrote:

> On Thu, Jan 26, 2017 at 4:09 AM, Ivan Levkivskyi 
> wrote:
>
>
>
> Concerning list/set/dict comprehensions, I am much more in favor of making
> comprehensions simply equivalent to for-loops (more or less like you
> proposed using yield from). The only reason to introduce auxiliary function
> scope was to prevent the loop variables from leaking outside comprehensions.
> Formally, this is indeed backward incompatible, but I doubt many people
> depend on the current counter-intuitive behavior.
>
> Concerning generator expressions, probably it is indeed better to simply
> prohibit yield inside them.
>
>
> Thank you to everyone who responded to my post and provided excellent
> analysis.
>
> For Python, I don't know what the best way to proceed is:
>
> OPTION 1
> 
>
> Make a SyntaxError:  [(yield 1) for x in range(10)]
> and update the documentation to explain that this is an invalid construct.
>
> This would have certainly helped me identify the source of the problem as
> I tried porting buildbot 0.9 to Python 3.
>
> However, while not very common, there is Python 2.x code that uses that.
> I found these cases in the buildbot code which I changed so as to work on
> Python 2 and 3:
>
> https://github.com/buildbot/buildbot/pull/2661
> https://github.com/buildbot/buildbot/pull/2673
>
>
>  OPTION 2
> =
> Make this return a list on Python 3, like in Python 2:  [
> (yield 1) for x in range(10)]
>
> As pointed out by others on the this mailing list, there are some
> problems associated with that.  I don't know if there are many Python 2
> codebases out there
> with this construct, but it would be nice to have one less Python 2 -> 3
> porting gotcha.
>
>
> I'm OK with either approach.  Leaving things the way they are in Python 3
> is no good, IMHO.
>

My vote is it be a SyntaxError since you're not getting what you expect
from the syntax.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generator objects and list comprehensions?

[Craig Rodrigues]

On Thu, Jan 26, 2017 at 4:09 AM, Ivan Levkivskyi 
wrote:

>
>
> Concerning list/set/dict comprehensions, I am much more in favor of making
> comprehensions simply equivalent to for-loops (more or less like you
> proposed using yield from). The only reason to introduce auxiliary function
> scope was to prevent the loop variables from leaking outside comprehensions.
> Formally, this is indeed backward incompatible, but I doubt many people
> depend on the current counter-intuitive behavior.
>
> Concerning generator expressions, probably it is indeed better to simply
> prohibit yield inside them.
>
>
Thank you to everyone who responded to my post and provided excellent
analysis.

For Python, I don't know what the best way to proceed is:

OPTION 1


Make a SyntaxError:  [(yield 1) for x in range(10)]
and update the documentation to explain that this is an invalid construct.

This would have certainly helped me identify the source of the problem as I
tried porting buildbot 0.9 to Python 3.

However, while not very common, there is Python 2.x code that uses that.
I found these cases in the buildbot code which I changed so as to work on
Python 2 and 3:

https://github.com/buildbot/buildbot/pull/2661
https://github.com/buildbot/buildbot/pull/2673


 OPTION 2
=
Make this return a list on Python 3, like in Python 2:  [
(yield 1) for x in range(10)]

As pointed out by others on the this mailing list, there are some
problems associated with that.  I don't know if there are many Python 2
codebases out there
with this construct, but it would be nice to have one less Python 2 -> 3
porting gotcha.


I'm OK with either approach.  Leaving things the way they are in Python 3
is no good, IMHO.
--
Craig
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generator objects and list comprehensions?

[Ivan Levkivskyi]

On 26 January 2017 at 00:53, Nathaniel Smith  wrote:

>
> I'm not sure this is actually a good idea, given the potential for
> ambiguity and backwards compatibility considerations -- I'm kind of leaning
> towards the deprecate/error option on balance :-). But it's an option that
> would probably be less confusing than the status quo, and would make it
> easier to write the kind of straddling code the original poster was trying
> to write.
>
>
Concerning list/set/dict comprehensions, I am much more in favor of making
comprehensions simply equivalent to for-loops (more or less like you
proposed using yield from). The only reason to introduce auxiliary function
scope was to prevent the loop variables from leaking outside comprehensions.
Formally, this is indeed backward incompatible, but I doubt many people
depend on the current counter-intuitive behavior.

Concerning generator expressions, probably it is indeed better to simply
prohibit yield inside them.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generator objects and list comprehensions?

[Nathaniel Smith]

On Jan 25, 2017 8:16 AM, "Joe Jevnik"  wrote:

List, set, and dict comprehensions compile like:

# input
result = [expression for lhs in iterator_expression]

# output
def comprehension(iterator):
out = []
for lhs in iterator:
out.append(expression)
return out

result = comprehension(iter(iterator_expression))

When you make `expression` a `yield` the compiler thinks that
`comprehension` is a generator function instead of a normal function.

We can manually translate the following comprehension:

result = [(yield n) for n in (0, 1)]

def comprehension(iterator):
out = []
for n in iterator:
# (yield n) as an expression is the value sent into this generator
out.append((yield n))
return out

result = comprehension(iter((0, 1)))


We can see this in the behavior of `send` on the resulting generator:

In [1]: g = [(yield n) for n in (0, 1)]

In [2]: next(g)
Out[2]: 0

In [3]: g.send('hello')
Out[3]: 1

In [4]: g.send('world')
---
StopIteration Traceback (most recent call last)
 in ()
> 1 g.send('world')

StopIteration: ['hello', 'world']

The `return out` gets translated into `raise StopIteration(out)` because
the code is a generator.

The bytecode for this looks like:

In [5]: %%dis
   ...: [(yield n) for n in (0, 1)]
   ...:


  1   0 LOAD_CONST   0 ( at
0x7f4bae68eed0, file "", line 1>)
  3 LOAD_CONST   1 ('')
  6 MAKE_FUNCTION0
  9 LOAD_CONST   5 ((0, 1))
 12 GET_ITER
 13 CALL_FUNCTION1 (1 positional, 0 keyword pair)
 16 POP_TOP
 17 LOAD_CONST   4 (None)
 20 RETURN_VALUE

.
---
  1   0 BUILD_LIST   0
  3 LOAD_FAST0 (.0)
>>6 FOR_ITER13 (to 22)
  9 STORE_FAST   1 (n)
 12 LOAD_FAST1 (n)
 15 YIELD_VALUE
 16 LIST_APPEND  2
 19 JUMP_ABSOLUTE6
>>   22 RETURN_VALUE


In `` you can see us create the  function, call `iter` on
`(0, 1)`, and then call `(iter(0, 1))`. In `` you can
see the loop like we had above, but unlike a normal comprehension we have a
`YIELD_VALUE` (from the `(yield n)`) in the comprehension.

The reason that this is different in Python 2 is that list comprehension,
and only list comprehensions, compile inline. Instead of creating a new
function for the loop, it is done in the scope of the comprehension. For
example:

# input
result = [expression for lhs in iterator_expression]

# output
result = []
for lhs in iterator_expression:
result.append(lhs)

This is why `lhs` bleeds out of the comprehension. In Python 2, adding
making `expression` a `yield` causes the _calling_ function to become a
generator:

def f():
[(yield n) for n in range(3)]
# no return

# py2
>>> type(f())
generator

# py3
>> type(f())
NoneType

In Python 2 the following will even raise a syntax error:

In [5]: def f():
   ...: return [(yield n) for n in range(3)]
   ...:
   ...:
  File "", line 2
return [(yield n) for n in range(3)]
SyntaxError: 'return' with argument inside generator


Generator expressions are a little different because the compilation
already includes a `yield`.

# input
(expression for lhs in iterator_expression)

# output
def genexpr(iterator):
for lhs in iterator:
yield expression

You can actually abuse this to write a cute `flatten` function:

`flatten = lambda seq: (None for sub in seq if (yield from sub) and False)`

because it translates to:

def genexpr(seq):
for sub in seq:
# (yield from sub) as an expression returns the last sent in value
if (yield from sub) and False:
# we never enter this branch
yield None


That was a long way to explain what the problem was. I think that that
solution is to stop using `yield` in comprehensions because it is
confusing, or to make `yield` in a comprehension a syntax error.


Another option would be to restore the py2 behavior by inserting an
implicit 'yield from' whenever the embedded expression contains a yield. So
in your example where

result = [(yield n) for n in (0, 1)]

currently gets expanded to

result = comprehension(iter((0, 1)))

it could notice that the expanded function 'comprehension' is a generator,
and emit code like this instead:

result = yield from comprehension(iter((0, 1)))

At least, this would work for list/dict/set comprehensions; not so much for
generator expressions. I assume this is basically how 'await' in
comprehensions works in 3.6.

I'm not sure this is actually a good idea, given the potential for
ambiguity and backwards compatibility considerations -- I'm kind of leaning
towards the 

Re: [Python-Dev] Generator objects and list comprehensions?

[Steve Holden]

On Wed, Jan 25, 2017 at 6:28 AM, Joe Jevnik  wrote:

> That was a long way to explain what the problem was. I think that that
> solution is to stop using `yield` in comprehensions because it is
> confusing, or to make `yield` in a comprehension a syntax error.
>
> Thanks for the very clear explanation.

Would an addition to the documentation for 3.6 giving an abbreviated
version help while the devs consider whether any changes are appropriate
for 3.7?


Steve Holden
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generator objects and list comprehensions?

[Sven R. Kunze]

On 25.01.2017 07:28, Joe Jevnik wrote:


That was a long way to explain what the problem was. I think that that 
solution is to stop using `yield` in comprehensions because it is 
confusing, or to make `yield` in a comprehension a syntax error.




Same here; mixing comprehensions and yield (from) can't be explained 
easily. A SyntaxError would be most appropriate.


Regards,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generator objects and list comprehensions?

[Joe Jevnik]

List, set, and dict comprehensions compile like:

# input
result = [expression for lhs in iterator_expression]

# output
def comprehension(iterator):
out = []
for lhs in iterator:
out.append(expression)
return out

result = comprehension(iter(iterator_expression))

When you make `expression` a `yield` the compiler thinks that
`comprehension` is a generator function instead of a normal function.

We can manually translate the following comprehension:

result = [(yield n) for n in (0, 1)]

def comprehension(iterator):
out = []
for n in iterator:
# (yield n) as an expression is the value sent into this generator
out.append((yield n))
return out

result = comprehension(iter((0, 1)))


We can see this in the behavior of `send` on the resulting generator:

In [1]: g = [(yield n) for n in (0, 1)]

In [2]: next(g)
Out[2]: 0

In [3]: g.send('hello')
Out[3]: 1

In [4]: g.send('world')
---
StopIteration Traceback (most recent call last)
 in ()
> 1 g.send('world')

StopIteration: ['hello', 'world']

The `return out` gets translated into `raise StopIteration(out)` because
the code is a generator.

The bytecode for this looks like:

In [5]: %%dis
   ...: [(yield n) for n in (0, 1)]
   ...:


  1   0 LOAD_CONST   0 ( at
0x7f4bae68eed0, file "", line 1>)
  3 LOAD_CONST   1 ('')
  6 MAKE_FUNCTION0
  9 LOAD_CONST   5 ((0, 1))
 12 GET_ITER
 13 CALL_FUNCTION1 (1 positional, 0 keyword pair)
 16 POP_TOP
 17 LOAD_CONST   4 (None)
 20 RETURN_VALUE

.
---
  1   0 BUILD_LIST   0
  3 LOAD_FAST0 (.0)
>>6 FOR_ITER13 (to 22)
  9 STORE_FAST   1 (n)
 12 LOAD_FAST1 (n)
 15 YIELD_VALUE
 16 LIST_APPEND  2
 19 JUMP_ABSOLUTE6
>>   22 RETURN_VALUE


In `` you can see us create the  function, call `iter` on
`(0, 1)`, and then call `(iter(0, 1))`. In `` you can
see the loop like we had above, but unlike a normal comprehension we have a
`YIELD_VALUE` (from the `(yield n)`) in the comprehension.

The reason that this is different in Python 2 is that list comprehension,
and only list comprehensions, compile inline. Instead of creating a new
function for the loop, it is done in the scope of the comprehension. For
example:

# input
result = [expression for lhs in iterator_expression]

# output
result = []
for lhs in iterator_expression:
result.append(lhs)

This is why `lhs` bleeds out of the comprehension. In Python 2, adding
making `expression` a `yield` causes the _calling_ function to become a
generator:

def f():
[(yield n) for n in range(3)]
# no return

# py2
>>> type(f())
generator

# py3
>> type(f())
NoneType

In Python 2 the following will even raise a syntax error:

In [5]: def f():
   ...: return [(yield n) for n in range(3)]
   ...:
   ...:
  File "", line 2
return [(yield n) for n in range(3)]
SyntaxError: 'return' with argument inside generator


Generator expressions are a little different because the compilation
already includes a `yield`.

# input
(expression for lhs in iterator_expression)

# output
def genexpr(iterator):
for lhs in iterator:
yield expression

You can actually abuse this to write a cute `flatten` function:

`flatten = lambda seq: (None for sub in seq if (yield from sub) and False)`

because it translates to:

def genexpr(seq):
for sub in seq:
# (yield from sub) as an expression returns the last sent in value
if (yield from sub) and False:
# we never enter this branch
yield None


That was a long way to explain what the problem was. I think that that
solution is to stop using `yield` in comprehensions because it is
confusing, or to make `yield` in a comprehension a syntax error.

On Wed, Jan 25, 2017 at 12:38 AM, Craig Rodrigues 
wrote:

> Hi,
>
> Glyph pointed this out to me here: http://twistedmatrix.
> com/pipermail/twisted-python/2017-January/031106.html
>
> If I do this on Python 3.6:
>
> >>  [(yield 1) for x in range(10)]
>  at 0x10cd210f8>
>
> If I understand this: https://docs.python.org/
> 3/reference/expressions.html#list-displays
> then this is a list display and should give a list, not a generator object.
> Is there a bug in Python, or does the documentation need to be updated?
>
> --
> Craig
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> joe%40quantopian.com
>
>

Re: [Python-Dev] Generator objects and list comprehensions?

[Ivan Levkivskyi]

On 25 January 2017 at 07:01, Chris Angelico  wrote:

> >>>  [(yield 1) for x in range(10)]
> >  at 0x10cd210f8>
>

This is an old bug, see e.g. http://bugs.python.org/issue10544
The ``yield`` inside comprehensions is bound to the auxiliary function.
Instead it should be bound to an enclosing function, like it is done for
``await``.
The behaviour of ``await`` in comprehensions is intuitive
(since it is simply equivalent to a for-loop):

>>> async def f(i):
... return i

>>> async def g_for():
... lst = []
... for i in range(5):
... lst.append(await f(i))
... print(lst)

>>> g_for().send(None)
[0, 1, 2, 3, 4]
Traceback (most recent call last):
  File "", line 1, in 
StopIteration

>>> async def g_comp():
... print([await f(i) for i in range(5)])

>>> g_comp().send(None)# exactly the same as g_for
[0, 1, 2, 3, 4]
Traceback (most recent call last):
  File "", line 1, in 
StopIteration

While current behaviour of ``yield`` in comprehensions is confusing:

>>> def c_for():
... lst = []
... for i in range(5):
... lst.append((yield i))
... print(lst)

>>> c_for().send(None)
0
>>> c_for().send(None)
1
# etc.

>>> def c_comp():
... print([(yield i) for i in range(5)])

>>> c_comp().send(None)  # Naively this should be equivalent to the above,
but...
. at 0x7f1fd1faa630>
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'NoneType' object has no attribute 'send'

I promised myself to write a patch, but didn't have time for this yet.
I hope I will do this at some point soon.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generator objects and list comprehensions?

[Chris Angelico]

On Wed, Jan 25, 2017 at 4:38 PM, Craig Rodrigues  wrote:
>
> Glyph pointed this out to me here:
> http://twistedmatrix.com/pipermail/twisted-python/2017-January/031106.html
>
> If I do this on Python 3.6:
>
>>>  [(yield 1) for x in range(10)]
>  at 0x10cd210f8>
>
> If I understand this:
> https://docs.python.org/3/reference/expressions.html#list-displays
> then this is a list display and should give a list, not a generator object.
> Is there a bug in Python, or does the documentation need to be updated?

That looks like an odd interaction between yield expressions and list
comps. Since a list comprehension is actually implemented as a nested
function, your code actually looks more-or-less like this:

>>> def (iter):
result = []
for x in iter:
result.append((yield 1))
return result
>>> (iter(range(10))
 at 0x10cd210f8>

This function is a generator, and calling it returns what you see
above. If you step that iterator, it'll yield 1 ten times, and then
raise StopIteration with the resulting list.

Based on a cursory examination of the issue at hand, I think what
you're working with might be functioning as a coroutine? If so, you
may find that using "await" instead of "yield" dodges the problem, as
it won't turn the list comp into a generator. But I can't be 100%
certain of that. (Also, that would definitely stop you from having
single-codebase 2.7/3.x code.)

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Generator objects and list comprehensions?

[Craig Rodrigues]

Hi,

Glyph pointed this out to me here:
http://twistedmatrix.com/pipermail/twisted-python/2017-January/031106.html

If I do this on Python 3.6:

>>  [(yield 1) for x in range(10)]
 at 0x10cd210f8>

If I understand this:
https://docs.python.org/3/reference/expressions.html#list-displays
then this is a list display and should give a list, not a generator object.
Is there a bug in Python, or does the documentation need to be updated?

--
Craig
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com