Re: [Python-Dev] Python initialization and embedded Python

[Victor Stinner]

2017-11-21 16:57 GMT+01:00 Eric Snow :
>> I understand that moving global variables to _PyRuntime helps to
>> clarify how these variables are initialized and then finalized, but
>> memory allocators are a complex corner case.
>
> Agreed.  I spent a large portion of my time getting the allocators
> right when working on the original _PyRuntime patch.  It's tricky
> code.

Oh, I forgot to notify you: when I worked on Py_Main(), I got crashes
because PyMem_RawMalloc() wasn't usable before calling
Py_Initialize(). This is what I call a regresion, and that's why I
started this thread :-)

I fixed the issue by calling _PyRuntime_Initialize() as the very first
function in main().

I also had to add _PyMem_GetDefaultRawAllocator() to get a
deterministic memory allocator, rather than depending on the allocator
set an application embedding Python, we must be sure that the same
allocator is used to initialize and finalize Python.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Antoine Pitrou]

On Wed, 22 Nov 2017 10:38:32 +0100
Victor Stinner  wrote:
> 
> I fixed the issue by calling _PyRuntime_Initialize() as the very first
> function in main().
> 
> I also had to add _PyMem_GetDefaultRawAllocator() to get a
> deterministic memory allocator, rather than depending on the allocator
> set an application embedding Python, we must be sure that the same
> allocator is used to initialize and finalize Python.

This is a bit worrying.  Do Python embedders have to go through the
same dance?

IMHO this really needs a simple solution documented somewhere.  Also,
hopefully when you do the wrong thing, you get a clear error message to
know how to fix your code?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Victor Stinner]

2017-11-22 12:04 GMT+01:00 Antoine Pitrou :
> IMHO this really needs a simple solution documented somewhere.  Also,
> hopefully when you do the wrong thing, you get a clear error message to
> know how to fix your code?

Right now, calling PyMem_RawMalloc() before calling
_PyRuntime_Initialize() calls the function at address NULL, so you get
a segmentation fault.

Documenting the new requirements is part of the discussion, it's one
option how to fix this issue.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Antoine Pitrou]

On Wed, 22 Nov 2017 12:12:32 +0100
Victor Stinner  wrote:
> 2017-11-22 12:04 GMT+01:00 Antoine Pitrou :
> > IMHO this really needs a simple solution documented somewhere.  Also,
> > hopefully when you do the wrong thing, you get a clear error message to
> > know how to fix your code?  
> 
> Right now, calling PyMem_RawMalloc() before calling
> _PyRuntime_Initialize() calls the function at address NULL, so you get
> a segmentation fault.

Can we get something more readable? For example:

FATAL ERROR: PyMem_RawMalloc(): malloc function is NULL, did you call
_PyRuntime_Initialize?

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Serhiy Storchaka]

From 
https://stackoverflow.com/questions/45190729/differences-between-generator-comprehension-expressions.


g = [(yield i) for i in range(3)]

Syntactically this looks like a list comprehension, and g should be a 
list, right? But actually it is a generator. This code is equivalent to 
the following code:


def _make_list(it):
result = []
for i in it:
result.append(yield i)
return result
g = _make_list(iter(range(3)))

Due to "yield" in the expression _make_list() is not a function 
returning a list, but a generator function returning a generator.


This change in semantic looks unintentional to me. It looks like leaking 
an implementation detail. If a list comprehension would be implemented 
not via creating and calling an intermediate function, but via an 
inlined loop (like in Python 2) this would be a syntax error if used 
outside of a function or would make an outer function a generator function.


__result = []
__i = None
try:
for __i in range(3):
__result.append(yield __i)
g = __result
finally:
del __result, __i

I don't see how the current behavior can be useful.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

Serhiy,

I think this is indeed a problem. For me the biggest surprise was that
`yield` inside a comprehension does not turn a surrounding function into
comprehension, see also
https://stackoverflow.com/questions/29334054/why-am-i-getting-different-results-when-using-a-list-comprehension-with-coroutin

In fact there is a b.p.o. issue for this https://bugs.python.org/issue10544,
it is assigned to me since July, but I was focused on other things recently.
My plan was to restore the Python 2 semantics while still avoiding the leak
of comprehension variable to the enclosing scope (the initial reason of
introducing auxiliary "_make_list" function IIUC).
So that:

1) g = [(yield i) for i in range(3)] outside a function will be a
SyntaxError (yield outside a function)
2) g = [(yield i) for i in range(3)] inside a function will turn that
enclosing function into generator.
3) accessing i after g = [(yield i) for i in range(3)] will give a
NameError: name 'i' is not defined

If you have time to work on this, then I will be glad if you take care of
this issue, you can re-assign it.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

On Wed, 22 Nov 2017 15:03:09 +0200
Serhiy Storchaka  wrote:
>  From 
> https://stackoverflow.com/questions/45190729/differences-between-generator-comprehension-expressions.
> 
>  g = [(yield i) for i in range(3)]
> 
> Syntactically this looks like a list comprehension, and g should be a 
> list, right? But actually it is a generator. This code is equivalent to 
> the following code:
> 
>  def _make_list(it):
>  result = []
>  for i in it:
>  result.append(yield i)
>  return result
>  g = _make_list(iter(range(3)))
> 
> Due to "yield" in the expression _make_list() is not a function 
> returning a list, but a generator function returning a generator.
> 
> This change in semantic looks unintentional to me. It looks like leaking 
> an implementation detail.

Perhaps we can deprecate the use of "yield" in comprehensions and make
it a syntax error in a couple versions?

I don't see a reason for writing such code rather than the more
explicit variants.  It looks really obscure, regardless of the actual
semantics.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Serhiy Storchaka]

22.11.17 15:25, Ivan Levkivskyi пише:
I think this is indeed a problem.. For me the biggest surprise was that 
`yield` inside a comprehension does not turn a surrounding function into 
comprehension, see also

https://stackoverflow.com/questions/29334054/why-am-i-getting-different-results-when-using-a-list-comprehension-with-coroutin

In fact there is a b.p.o. issue for this 
https://bugs.python.org/issue10544, it is assigned to me since July, but 
I was focused on other things recently.
My plan was to restore the Python 2 semantics while still avoiding the 
leak of comprehension variable to the enclosing scope (the initial 
reason of introducing auxiliary "_make_list" function IIUC).

So that:

1) g = [(yield i) for i in range(3)] outside a function will be a 
SyntaxError (yield outside a function)
2) g = [(yield i) for i in range(3)] inside a function will turn that 
enclosing function into generator.
3) accessing i after g = [(yield i) for i in range(3)] will give a 
NameError: name 'i' is not defined


If you have time to work on this, then I will be glad if you take care 
of this issue, you can re-assign it.


I have the same plan. I know how implement this for comprehensions, but 
the tricky question is what to do with generator expressions? Ideally


result = [expr for i in iterable]

and

result = list(expr for i in iterable)

should have the same semantic. I.e. if expr is "(yield i)", this should 
turn the enclosing function into a generator function, and fill the list 
with values passed to the generator's .send(). I have no idea how 
implement this.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 14:38, Antoine Pitrou  wrote:

> On Wed, 22 Nov 2017 15:03:09 +0200
> Serhiy Storchaka  wrote:
> >  From
> > https://stackoverflow.com/questions/45190729/
> differences-between-generator-comprehension-expressions.
> >
> >  g = [(yield i) for i in range(3)]
> >
> > Syntactically this looks like a list comprehension, and g should be a
> > list, right? But actually it is a generator. This code is equivalent to
> > the following code:
> >
> >  def _make_list(it):
> >  result = []
> >  for i in it:
> >  result.append(yield i)
> >  return result
> >  g = _make_list(iter(range(3)))
> >
> > Due to "yield" in the expression _make_list() is not a function
> > returning a list, but a generator function returning a generator.
> >
> > This change in semantic looks unintentional to me. It looks like leaking
> > an implementation detail.
>
> Perhaps we can deprecate the use of "yield" in comprehensions and make
> it a syntax error in a couple versions?
>
> I don't see a reason for writing such code rather than the more
> explicit variants.  It looks really obscure, regardless of the actual
> semantics.
>

People actually try this (probably simply because they like comprehensions)
see two mentioned Stackoverflow questions, plus there are two b.p.o. issues.
So this will be a breaking change. Second, recent PEP 530 allowed writing a
similar comprehensions with `await`:

async def process(funcs):
result = [await fun() for fun in funcs]  # OK
...

Moreover, it has the semantics very similar to the proposed by Serhiy for
`yield` (i.e. equivalent to for-loop without name leaking into outer scope).
Taking into account that the actual fix is not so hard, I don't think it
makes sense to have all the hassles of deprecation period.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 22 November 2017 at 13:53, Ivan Levkivskyi  wrote:
> On 22 November 2017 at 14:38, Antoine Pitrou  wrote:
>>
>> On Wed, 22 Nov 2017 15:03:09 +0200
>> Serhiy Storchaka  wrote:
>> >  From
>> >
>> > https://stackoverflow.com/questions/45190729/differences-between-generator-comprehension-expressions.
>> >
>> >  g = [(yield i) for i in range(3)]
>> >
>> > Syntactically this looks like a list comprehension, and g should be a
>> > list, right? But actually it is a generator. This code is equivalent to
>> > the following code:
>> >
>> >  def _make_list(it):
>> >  result = []
>> >  for i in it:
>> >  result.append(yield i)
>> >  return result
>> >  g = _make_list(iter(range(3)))
>> >
>> > Due to "yield" in the expression _make_list() is not a function
>> > returning a list, but a generator function returning a generator.
>> >
>> > This change in semantic looks unintentional to me. It looks like leaking
>> > an implementation detail.
>>
>> Perhaps we can deprecate the use of "yield" in comprehensions and make
>> it a syntax error in a couple versions?
>>
>> I don't see a reason for writing such code rather than the more
>> explicit variants.  It looks really obscure, regardless of the actual
>> semantics.
>
>
> People actually try this (probably simply because they like comprehensions)
> see two mentioned Stackoverflow questions, plus there are two b.p.o. issues.
> So this will be a breaking change. Second, recent PEP 530 allowed writing a
> similar comprehensions with `await`:
>
> async def process(funcs):
> result = [await fun() for fun in funcs]  # OK
> ...
>
> Moreover, it has the semantics very similar to the proposed by Serhiy for
> `yield` (i.e. equivalent to for-loop without name leaking into outer scope).
> Taking into account that the actual fix is not so hard, I don't think it
> makes sense to have all the hassles of deprecation period.

I agree with Antoine. This (yield in a comprehension) seems far too
tricky, and I'd rather it just be rejected. If I saw it in a code
review, I'd certainly insist it be rewritten as an explicit loop.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 14:53, Serhiy Storchaka  wrote:

> 22.11.17 15:25, Ivan Levkivskyi пише:
>
>> I think this is indeed a problem.. For me the biggest surprise was that
>> `yield` inside a comprehension does not turn a surrounding function into
>> comprehension, see also
>> https://stackoverflow.com/questions/29334054/why-am-i-gettin
>> g-different-results-when-using-a-list-comprehension-with-coroutin
>>
>> In fact there is a b.p.o. issue for this https://bugs.python.org/issue1
>> 0544, it is assigned to me since July, but I was focused on other things
>> recently.
>> My plan was to restore the Python 2 semantics while still avoiding the
>> leak of comprehension variable to the enclosing scope (the initial reason
>> of introducing auxiliary "_make_list" function IIUC).
>> So that:
>>
>> 1) g = [(yield i) for i in range(3)] outside a function will be a
>> SyntaxError (yield outside a function)
>> 2) g = [(yield i) for i in range(3)] inside a function will turn that
>> enclosing function into generator.
>> 3) accessing i after g = [(yield i) for i in range(3)] will give a
>> NameError: name 'i' is not defined
>>
>> If you have time to work on this, then I will be glad if you take care of
>> this issue, you can re-assign it.
>>
>
> I have the same plan. I know how implement this for comprehensions, but
> the tricky question is what to do with generator expressions? Ideally
>
> result = [expr for i in iterable]
>
> and
>
> result = list(expr for i in iterable)
>
> should have the same semantic. I.e. if expr is "(yield i)", this should
> turn the enclosing function into a generator function, and fill the list
> with values passed to the generator's .send(). I have no idea how implement
> this.
>
>
Yes, generator expressions are also currently problematic with `await`:

>>> async def g(i):
... print(i)
...
>>> async def f():
... result = list(await g(i) for i in range(3))
... print(result)
...
>>> f().send(None)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in f
TypeError: 'async_generator' object is not iterable

Maybe Yury can say something about this?

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 15:09, Paul Moore  wrote:

> On 22 November 2017 at 13:53, Ivan Levkivskyi 
> wrote:
> > On 22 November 2017 at 14:38, Antoine Pitrou 
> wrote:
> >>
> >> On Wed, 22 Nov 2017 15:03:09 +0200
> >> Serhiy Storchaka  wrote:
> >> >  From
> >> >
> >> > https://stackoverflow.com/questions/45190729/
> differences-between-generator-comprehension-expressions.
> >> >
> >> >  g = [(yield i) for i in range(3)]
> >> >
> >> > Syntactically this looks like a list comprehension, and g should be a
> >> > list, right? But actually it is a generator. This code is equivalent
> to
> >> > the following code:
> >> >
> >> >  def _make_list(it):
> >> >  result = []
> >> >  for i in it:
> >> >  result.append(yield i)
> >> >  return result
> >> >  g = _make_list(iter(range(3)))
> >> >
> >> > Due to "yield" in the expression _make_list() is not a function
> >> > returning a list, but a generator function returning a generator.
> >> >
> >> > This change in semantic looks unintentional to me. It looks like
> leaking
> >> > an implementation detail.
> >>
> >> Perhaps we can deprecate the use of "yield" in comprehensions and make
> >> it a syntax error in a couple versions?
> >>
> >> I don't see a reason for writing such code rather than the more
> >> explicit variants.  It looks really obscure, regardless of the actual
> >> semantics.
> >
> >
> > People actually try this (probably simply because they like
> comprehensions)
> > see two mentioned Stackoverflow questions, plus there are two b.p.o.
> issues.
> > So this will be a breaking change. Second, recent PEP 530 allowed
> writing a
> > similar comprehensions with `await`:
> >
> > async def process(funcs):
> > result = [await fun() for fun in funcs]  # OK
> > ...
> >
> > Moreover, it has the semantics very similar to the proposed by Serhiy for
> > `yield` (i.e. equivalent to for-loop without name leaking into outer
> scope).
> > Taking into account that the actual fix is not so hard, I don't think it
> > makes sense to have all the hassles of deprecation period.
>
> I agree with Antoine. This (yield in a comprehension) seems far too
> tricky, and I'd rather it just be rejected. If I saw it in a code
> review, I'd certainly insist it be rewritten as an explicit loop.
>
>
There are many things that I would reject in code review, but they are
still allowed in Python,
this is one of the reasons why code reviews exist. Also I am not sure how
`yield` in a comprehension
is more tricky than `await` in a comprehension. Anyway, this looks more
like a taste question.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

On Wed, 22 Nov 2017 15:15:49 +0100
Ivan Levkivskyi  wrote:
> There are many things that I would reject in code review, but they are
> still allowed in Python,
> this is one of the reasons why code reviews exist. Also I am not sure how
> `yield` in a comprehension
> is more tricky than `await` in a comprehension.

I am not sure either, but do note that "yield" and "await" are two
different things with different semantics, so allowing "await" while
disallowing "yield" wouldn't strike me as inconsistent.

The exact semantics of "yield" inside a comprehension is a common
source of surprise or bewilderment, and the only actual use I've seen
of it is to show it off as a "clever trick".  So instead of fixing (and
perhaps complicating) the implementation to try and make it do the
supposedly right thing, I am proposing to simply disallow it so that
we are done with the controversy :-)

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 22 November 2017 at 14:15, Ivan Levkivskyi  wrote:
> On 22 November 2017 at 15:09, Paul Moore  wrote:
>>
>> On 22 November 2017 at 13:53, Ivan Levkivskyi 
>> wrote:
>> > On 22 November 2017 at 14:38, Antoine Pitrou 
>> > wrote:
>> >>
>> >> On Wed, 22 Nov 2017 15:03:09 +0200
>> >> Serhiy Storchaka  wrote:
>> >> >  From
>> >> >
>> >> >
>> >> > https://stackoverflow.com/questions/45190729/differences-between-generator-comprehension-expressions.
>> >> >
>> >> >  g = [(yield i) for i in range(3)]
>> >> >
>> >> > Syntactically this looks like a list comprehension, and g should be a
>> >> > list, right? But actually it is a generator. This code is equivalent
>> >> > to
>> >> > the following code:
>> >> >
>> >> >  def _make_list(it):
>> >> >  result = []
>> >> >  for i in it:
>> >> >  result.append(yield i)
>> >> >  return result
>> >> >  g = _make_list(iter(range(3)))
>> >> >
>> >> > Due to "yield" in the expression _make_list() is not a function
>> >> > returning a list, but a generator function returning a generator.
>> >> >
>> >> > This change in semantic looks unintentional to me. It looks like
>> >> > leaking
>> >> > an implementation detail.
>> >>
>> >> Perhaps we can deprecate the use of "yield" in comprehensions and make
>> >> it a syntax error in a couple versions?
>> >>
>> >> I don't see a reason for writing such code rather than the more
>> >> explicit variants.  It looks really obscure, regardless of the actual
>> >> semantics.
>> >
>> >
>> > People actually try this (probably simply because they like
>> > comprehensions)
>> > see two mentioned Stackoverflow questions, plus there are two b.p.o.
>> > issues.
>> > So this will be a breaking change. Second, recent PEP 530 allowed
>> > writing a
>> > similar comprehensions with `await`:
>> >
>> > async def process(funcs):
>> > result = [await fun() for fun in funcs]  # OK
>> > ...
>> >
>> > Moreover, it has the semantics very similar to the proposed by Serhiy
>> > for
>> > `yield` (i.e. equivalent to for-loop without name leaking into outer
>> > scope).
>> > Taking into account that the actual fix is not so hard, I don't think it
>> > makes sense to have all the hassles of deprecation period.
>>
>> I agree with Antoine. This (yield in a comprehension) seems far too
>> tricky, and I'd rather it just be rejected. If I saw it in a code
>> review, I'd certainly insist it be rewritten as an explicit loop.
>>
>
> There are many things that I would reject in code review, but they are still
> allowed in Python,
> this is one of the reasons why code reviews exist. Also I am not sure how
> `yield` in a comprehension
> is more tricky than `await` in a comprehension. Anyway, this looks more like
> a taste question.

I generally don't understand "await" in any context, so I deferred
judgement on that :-) Based on your comment that they are equally
tricky, I'd suggest we prohibit them both ;-)

Less facetiously, comprehensions are defined in the language reference
in terms of a source translation to nested loops. That description
isn't 100% precise, but nevertheless, if yield/async in a
comprehension doesn't behave like that, I'd consider it a bug. So
current behaviour (for both yield and await) is a bug, and your
proposed semantics for yield is correct. Await in a comprehension
should work similarly - the docs for await expressions just say "Can
only be used inside a coroutine function", so based on that they are
legal within a comprehension inside a coroutine function (if the
semantics in that case isn't obvious, it should be clarified - the
await expression docs are terse to the point that I can't tell
myself).

So I'm +1 on your suggestion.

I remain concerned that yield/await expressions are adding a lot of
complexity to the language, that isn't explained in the manuals in a
way that's accessible to non-specialists. Maybe "ban (or reject at
code review) all the complex stuff" isn't a reasonable approach, but
nor is expecting everyone encountering async code to have read and
understood all the async PEPs and docs. We need to consider
maintenance programmers as well (who are often looking at code with
only a relatively high-level understanding).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 15:46, Antoine Pitrou  wrote:

> On Wed, 22 Nov 2017 15:15:49 +0100
> Ivan Levkivskyi  wrote:
> > There are many things that I would reject in code review, but they are
> > still allowed in Python,
> > this is one of the reasons why code reviews exist. Also I am not sure how
> > `yield` in a comprehension
> > is more tricky than `await` in a comprehension.
>
> I am not sure either, but do note that "yield" and "await" are two
> different things with different semantics, so allowing "await" while
> disallowing "yield" wouldn't strike me as inconsistent.
>
> The exact semantics of "yield" inside a comprehension is a common
> source of surprise or bewilderment, and the only actual use I've seen
> of it is to show it off as a "clever trick".  So instead of fixing (and
> perhaps complicating) the implementation to try and make it do the
> supposedly right thing, I am proposing to simply disallow it so that
> we are done with the controversy :-)
>
>
Actually, I am not sure there is really a controversy, I have not yet met a
person who _expects_
`yield` in a comprehension to work as it works now, instead everyone thinks
it is just equivalent to a for-loop.

Anyway, I have some compromise idea, will send it in a separate e-mail.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 15:47, Paul Moore  wrote:

> I generally don't understand "await" in any context, so I deferred
> judgement on that :-) Based on your comment that they are equally
> tricky, I'd suggest we prohibit them both ;-)
>
> Less facetiously, comprehensions are defined in the language reference
> in terms of a source translation to nested loops. That description
> isn't 100% precise, but nevertheless, if yield/async in a
> comprehension doesn't behave like that, I'd consider it a bug. So
> current behaviour (for both yield and await) is a bug, and your
> proposed semantics for yield is correct.
>

I think there may be a small misunderstanding here. The situation is
different for comprehensions and generator expressions,
let me summarize the current state:

- yield in comprehensions works "wrong" (a shorthand for not according to
the docs/naive expectations, i.e. not equivalent to for loop)
- await in comprehensions works "right"
- yield in generator expressions works "wrong"
- await in generator expressions works "wrong"

After some thinking, both `yield` and `await` look quite mind bending in
_generator expressions_, so maybe the right compromise strategy is:

- fix yield in comprehensions
- await in comprehensions already works
- make both `yield` and `await` a SyntaxError in generator expressions.

What do you think?

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 22 November 2017 at 15:10, Ivan Levkivskyi  wrote:
> I think there may be a small misunderstanding here. The situation is
> different for comprehensions and generator expressions,
> let me summarize the current state:
[...]
> What do you think?

Can you point me to where in the docs it explains the semantic
difference between a generaor expression and a list comprehension?
Ignoring the trivial point that they return different values - think
of [a for a in expr] vs list(a for a in expr) if you like - these
should be semantically the same, I believe?

TBH, the docs aren't super-precise in this area. Before we start
debating the "correct" behaviour of edge cases, I think we need to
agree (and document) the semantics more precisely.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Yury Selivanov]

On Wed, Nov 22, 2017 at 10:10 AM, Ivan Levkivskyi  wrote:
> On 22 November 2017 at 15:47, Paul Moore  wrote:
>>
>> I generally don't understand "await" in any context, so I deferred
>> judgement on that :-) Based on your comment that they are equally
>> tricky, I'd suggest we prohibit them both ;-)
>>
>> Less facetiously, comprehensions are defined in the language reference
>> in terms of a source translation to nested loops. That description
>> isn't 100% precise, but nevertheless, if yield/async in a
>> comprehension doesn't behave like that, I'd consider it a bug. So
>> current behaviour (for both yield and await) is a bug, and your
>> proposed semantics for yield is correct.
>
>
> I think there may be a small misunderstanding here. The situation is
> different for comprehensions and generator expressions,
> let me summarize the current state:
>
> - yield in comprehensions works "wrong" (a shorthand for not according to
> the docs/naive expectations, i.e. not equivalent to for loop)
> - await in comprehensions works "right"
> - yield in generator expressions works "wrong"
> - await in generator expressions works "wrong"

"await" in generator expressions works *correct*. I explained it here:
https://bugs.python.org/issue32113

To clarify a bit more:

   r = (V for I in X)

is equivalent to:

   def _():
  for I in X:
yield V
   r = _()

For synchronous generator expression:

   r = (f(i) for i in range(3))

is really:

   def _():
  for i in range(3):
yield f(i)
   r = _()

For an asynchronous generator expression:

   r = (await f(i) for i in range(3))

is equivalent to:

   def _():
  for i in range(3):
yield (await f(i))
   r = _()

I'll update PEP 530 to clarify these transformations better.

>
> After some thinking, both `yield` and `await` look quite mind bending in
> _generator expressions_, so maybe the right compromise strategy is:
>
> - fix yield in comprehensions
> - await in comprehensions already works
> - make both `yield` and `await` a SyntaxError in generator expressions.

I'm all for prohibiting using 'yield' expression in generator
expressions or comprehensions.  The semantics is way to hard to
understand and hence be of any value.

Making 'await' a SyntaxError is absolutely not an option.  Async
generator expressions are a shorthand syntax for defining asynchronous
generators (PEP 525), and it's already being used in the wild.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 16:53, Paul Moore  wrote:

> On 22 November 2017 at 15:10, Ivan Levkivskyi 
> wrote:
> > I think there may be a small misunderstanding here. The situation is
> > different for comprehensions and generator expressions,
> > let me summarize the current state:
> [...]
> > What do you think?
>
> Can you point me to where in the docs it explains the semantic
> difference between a generaor expression and a list comprehension?
> Ignoring the trivial point that they return different values - think
> of [a for a in expr] vs list(a for a in expr) if you like - these
> should be semantically the same, I believe?
>
> TBH, the docs aren't super-precise in this area. Before we start
> debating the "correct" behaviour of edge cases, I think we need to
> agree (and document) the semantics more precisely.
>
> Paul
>

Looking at
https://docs.python.org/3/reference/expressions.html?highlight=generator%20expression#generator-expressions
it is indeed not super clear what it _should_ mean.
I agree this requires documentation updates.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 16:56, Yury Selivanov 
wrote:

> On Wed, Nov 22, 2017 at 10:10 AM, Ivan Levkivskyi 
> wrote:
> > On 22 November 2017 at 15:47, Paul Moore  wrote:
> [...]
> I'm all for prohibiting using 'yield' expression in generator
> expressions or comprehensions.  The semantics is way to hard to
> understand and hence be of any value.
>
> Making 'await' a SyntaxError is absolutely not an option.  Async
> generator expressions are a shorthand syntax for defining asynchronous
> generators (PEP 525), and it's already being used in the wild.
>

OK, makes sense, so it looks like we may have the following plan:

- fix `yield` in comprehensions
- update PEP 530 and docs re generator expressions vs comprehensions
- make `yield` in generator expressions a SyntaxError

If everyone agrees, then I propose to open a separate issue on b.p.o. to
coordinate the efforts.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 22 November 2017 at 15:56, Yury Selivanov  wrote:
> "await" in generator expressions works *correct*. I explained it here:
> https://bugs.python.org/issue32113
>
> To clarify a bit more:
>
>r = (V for I in X)
>
> is equivalent to:
>
>def _():
>   for I in X:
> yield V
>r = _()

The docs don't actually say that this equivalence is definitive.
There's a lot of vagueness - possibly because the equivalence wasn't
precise in 2.X (due to name leakage) and it wasn't updated to say that
comprehensions are now defined *precisely* in terms of this
equivalence.

But surely this means that:

1. await isn't allowed in comprehensions/generator expressions,
because the dummy function (_ in your expansion above) is not a
coroutine function.
2. yield expressions in a comprehension.generator will yield extra
values into the generated list (or the stream returned from the
generator epression). That seems wrong to me, at least in terms of how
I'd expect such an expression to behave.

So I think there's a problem with treating the equivalence as the
definition - it's informative, but not normative.

> Making 'await' a SyntaxError is absolutely not an option.  Async
> generator expressions are a shorthand syntax for defining asynchronous
> generators (PEP 525), and it's already being used in the wild.

But by the logic you just described, await isn't (or shouldn't be)
allowed, surely?

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 22 November 2017 at 16:08, Ivan Levkivskyi  wrote:
> On 22 November 2017 at 16:56, Yury Selivanov 
> wrote:
>>
>> On Wed, Nov 22, 2017 at 10:10 AM, Ivan Levkivskyi 
>> wrote:
>> > On 22 November 2017 at 15:47, Paul Moore  wrote:
>> [...]
>> I'm all for prohibiting using 'yield' expression in generator
>> expressions or comprehensions.  The semantics is way to hard to
>> understand and hence be of any value.
>>
>> Making 'await' a SyntaxError is absolutely not an option.  Async
>> generator expressions are a shorthand syntax for defining asynchronous
>> generators (PEP 525), and it's already being used in the wild.
>
>
> OK, makes sense, so it looks like we may have the following plan:
>
> - fix `yield` in comprehensions

I'm still not clear what "fix" would actually mean, but you propose
clarifying the docs below, so I assume it means "according to whatever
the updated docs say"...

> - update PEP 530 and docs re generator expressions vs comprehensions

Docs more importantly than PEP IMO. And are you implying that there's
a difference between generator expressions and comprehensions? I
thought both were intended to behave as if expanded to a function
containing nested for loops? Nothing said in this thread so far (about
semantics, as opposed to about current behaviour) implies there's a
deliberate difference.

> - make `yield` in generator expressions a SyntaxError

That contradicts the suggestion that generator expressions are
equivalent to the expanded function definition we saw before.

> If everyone agrees, then I propose to open a separate issue on b.p.o. to
> coordinate the efforts.

I agree it needs clarifying. Not sure I agree with your proposed
semantics, but I guess that can wait for when we have a concrete doc
change proposal.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 22 November 2017 at 15:56, Yury Selivanov  wrote:
> For synchronous generator expression:
>
>r = (f(i) for i in range(3))
>
> is really:
>
>def _():
>   for i in range(3):
> yield f(i)
>r = _()
>
> For an asynchronous generator expression:
>
>r = (await f(i) for i in range(3))
>
> is equivalent to:
>
>def _():
>   for i in range(3):
> yield (await f(i))
>r = _()

Wait, I missed this on first reading. The note in the docs for
generator expressions defining asynchronous generator expressions is
*incredibly* easy to miss, and doesn't say anything about the
semantics (the expansion you quote above) being different for the two
cases. This definitely needs clarifying in the docs.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Antoine Pitrou]

On Wed, 22 Nov 2017 17:08:14 +0100
Ivan Levkivskyi  wrote:
> On 22 November 2017 at 16:56, Yury Selivanov 
> wrote:
> 
> > On Wed, Nov 22, 2017 at 10:10 AM, Ivan Levkivskyi 
> > wrote:  
> > > On 22 November 2017 at 15:47, Paul Moore  wrote:  
> > [...]
> > I'm all for prohibiting using 'yield' expression in generator
> > expressions or comprehensions.  The semantics is way to hard to
> > understand and hence be of any value.
> >
> > Making 'await' a SyntaxError is absolutely not an option.  Async
> > generator expressions are a shorthand syntax for defining asynchronous
> > generators (PEP 525), and it's already being used in the wild.
> >  
> 
> OK, makes sense, so it looks like we may have the following plan:
> 
> - fix `yield` in comprehensions
> - update PEP 530 and docs re generator expressions vs comprehensions
> - make `yield` in generator expressions a SyntaxError

Given a comprehension (e.g. list comprehension) is expected to work
nominally as `constructor(generator expression)`
(e.g. `list(generator expression)`), I think both generator expressions
*and* comprehensions should raise a SyntaxError.

(more exactly, they should first emit a SyntaxWarning and then raise a
SyntaxError in a couple of versions)

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 17:24, Antoine Pitrou  wrote:

> On Wed, 22 Nov 2017 17:08:14 +0100
> Ivan Levkivskyi  wrote:
> > On 22 November 2017 at 16:56, Yury Selivanov 
> > wrote:
> >
> > > On Wed, Nov 22, 2017 at 10:10 AM, Ivan Levkivskyi <
> levkivs...@gmail.com>
> > > wrote:
> > > > On 22 November 2017 at 15:47, Paul Moore 
> wrote:
> > > [...]
> > > I'm all for prohibiting using 'yield' expression in generator
> > > expressions or comprehensions.  The semantics is way to hard to
> > > understand and hence be of any value.
> > >
> > > Making 'await' a SyntaxError is absolutely not an option.  Async
> > > generator expressions are a shorthand syntax for defining asynchronous
> > > generators (PEP 525), and it's already being used in the wild.
> > >
> >
> > OK, makes sense, so it looks like we may have the following plan:
> >
> > - fix `yield` in comprehensions
> > - update PEP 530 and docs re generator expressions vs comprehensions
> > - make `yield` in generator expressions a SyntaxError
>
> Given a comprehension (e.g. list comprehension) is expected to work
> nominally as `constructor(generator expression)`
>

As Yury just explained, these two are not equivalent if there is an `await`
in the comprehension/generator expression.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 17:16, Paul Moore  wrote:

> On 22 November 2017 at 16:08, Ivan Levkivskyi 
> wrote:
> > On 22 November 2017 at 16:56, Yury Selivanov 
> > wrote:
> >>
> >> On Wed, Nov 22, 2017 at 10:10 AM, Ivan Levkivskyi  >
> >> wrote:
> >> > On 22 November 2017 at 15:47, Paul Moore  wrote:
> >> [...]
> >> I'm all for prohibiting using 'yield' expression in generator
> >> expressions or comprehensions.  The semantics is way to hard to
> >> understand and hence be of any value.
> >>
> >> Making 'await' a SyntaxError is absolutely not an option.  Async
> >> generator expressions are a shorthand syntax for defining asynchronous
> >> generators (PEP 525), and it's already being used in the wild.
> >
> >
> > OK, makes sense, so it looks like we may have the following plan:
> >
> > - fix `yield` in comprehensions
>
> I'm still not clear what "fix" would actually mean, but you propose
> clarifying the docs below, so I assume it means "according to whatever
> the updated docs say"...
>
>
I mean the initial proposal: make comprehensions equivalent to a for-loop


> > - update PEP 530 and docs re generator expressions vs comprehensions
>
> Docs more importantly than PEP IMO. And are you implying that there's
> a difference between generator expressions and comprehensions? I
> thought both were intended to behave as if expanded to a function
> containing nested for loops? Nothing said in this thread so far (about
> semantics, as opposed to about current behaviour) implies there's a
> deliberate difference.
>

I think there may be a difference:

comprehension `g = [(yield i) for i in range(3)]` is defined as this code:

__result = []
__i = None
try:
for __i in range(3):
__result.append(yield __i)
g = __result
finally:
del __result, __i

while `g = list((yield i) for i in range(3))` is defined as this code:

def __gen():
   for i in range(3):
 yield (yield i)
g = list(__gen())

 Although these two definitions are equivalent in simple cases (like having
`f(i)` instead of `yield i`)

But this is debatable, I think before we move to other points we need to
agree on the clear definitions of semantics of generator expressions and
comprehensions.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 22 November 2017 at 16:30, Ivan Levkivskyi  wrote:
> On 22 November 2017 at 17:24, Antoine Pitrou  wrote:
>> Given a comprehension (e.g. list comprehension) is expected to work
>> nominally as `constructor(generator expression)`
>
> As Yury just explained, these two are not equivalent if there is an `await`
> in the comprehension/generator expression.

As Antoine said, people *expect* them to work the same. If they don't,
then that's a subtle change that came in as a result of the new async
functionality. It's a shame that this wasn't made clearer at the time
- one of the major issues I see with async is that it works "nearly,
but not quite" the way people expect, and we need to do more to help
people integrate async into their intuition. Just reiterating "that's
not right" isn't much help - we need to educate people in better
intuitions if the traditional ones are no longer accurate.

At the moment, I know I tend to treat Python semantics as I always
did, but with an implicit proviso, "unless async is involved, when I
can't assume any of my intuitions apply". That's *not* a good
situation to be in.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

Sorry, forgot some details in the second "definition":

 try:
 def __gen():
 for i in range(3):
 yield (yield i)
 g = list(__gen())
 finally:
 del __gen

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 17:43, Paul Moore  wrote:

> On 22 November 2017 at 16:30, Ivan Levkivskyi 
> wrote:
> > On 22 November 2017 at 17:24, Antoine Pitrou 
> wrote:
> >> Given a comprehension (e.g. list comprehension) is expected to work
> >> nominally as `constructor(generator expression)`
> >
> > As Yury just explained, these two are not equivalent if there is an
> `await`
> > in the comprehension/generator expression.
>
> As Antoine said, people *expect* them to work the same.
>

The difference is that a generator expression can be used independently,
one can assign it to a variable etc. not necessary to wrap it into a list()
Anyway, can you propose an equivalent "defining" code for both? Otherwise
it is not clear what you are defending.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 22 November 2017 at 16:38, Ivan Levkivskyi  wrote:
> On 22 November 2017 at 17:16, Paul Moore  wrote:
>>
>> Docs more importantly than PEP IMO. And are you implying that there's
>> a difference between generator expressions and comprehensions? I
>> thought both were intended to behave as if expanded to a function
>> containing nested for loops? Nothing said in this thread so far (about
>> semantics, as opposed to about current behaviour) implies there's a
>> deliberate difference.
>
>
> I think there may be a difference:
>
> comprehension `g = [(yield i) for i in range(3)]` is defined as this code:
>
> __result = []
> __i = None
> try:
> for __i in range(3):
> __result.append(yield __i)
> g = __result
> finally:
> del __result, __i

Not in the docs, it isn't... The docs explicitly state that a new
scope is involved.

People may *intuitively understand it* like this, but it's not the
definition. If it were the definition, I'd be fine, as people can
reason about it and know that the conclusions they come to (no matter
how unintuitive) are accurate.

But if it's how to *understand* what a list comprehension does, then
different rules apply - corner cases may work differently, but
conversely the cases that work differently have to clearly be corner
cases, otherwise it's no use as a mental model.

> while `g = list((yield i) for i in range(3))` is defined as this code:
>
> def __gen():
>for i in range(3):
>  yield (yield i)
> g = list(__gen())

Again, not in the docs.

>  Although these two definitions are equivalent in simple cases (like having
> `f(i)` instead of `yield i`)

And if they don't imply list(...) is the same as [...] then that will
come as a surprise to many people.

> But this is debatable, I think before we move to other points we need to
> agree on the clear definitions of semantics of generator expressions and
> comprehensions.

Absolutely. And I'd like those semantics to be expressed in a way that
doesn't need "except when await/async is involved" exceptions.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Yury Selivanov]

On Wed, Nov 22, 2017 at 11:11 AM, Paul Moore  wrote:
[..]
> But by the logic you just described, await isn't (or shouldn't be)
> allowed, surely?

No, that's a stretch. People can understand and actually use 'yield
(await foo())', but they usually can't figure what  'yield (yield
foo())' actually does:

   def foo():
   yield (yield 1)

Who on this mailing list can meaningfully use the 'foo()' generator?

   async def bar():
   yield (await foo())

'bar()' is perfectly usable in an 'async for' statement and it's easy
to understand what it does if you spend an hour writing async/await
code.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 17:50, Paul Moore  wrote:

> On 22 November 2017 at 16:38, Ivan Levkivskyi 
> wrote:
> > On 22 November 2017 at 17:16, Paul Moore  wrote:
> >>
> >> Docs more importantly than PEP IMO. And are you implying that there's
> >> a difference between generator expressions and comprehensions? I
> >> thought both were intended to behave as if expanded to a function
> >> containing nested for loops? Nothing said in this thread so far (about
> >> semantics, as opposed to about current behaviour) implies there's a
> >> deliberate difference.
> >
> >
> > I think there may be a difference:
> >
> > comprehension `g = [(yield i) for i in range(3)]` is defined as this
> code:
> >
> > __result = []
> > __i = None
> > try:
> > for __i in range(3):
> > __result.append(yield __i)
> > g = __result
> > finally:
> > del __result, __i
>
> Not in the docs, it isn't...


Yes, since there is almost nothing there, this is what I _propose_ (or
actually Serhiy proposed it first)


> The docs explicitly state that a new
> scope is involved.
>

But docs don't say it is a _function_ scope. The meaning of that statement
(as I understand it) is just that the loop variable doen't leak from
comprehension.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 22 November 2017 at 16:47, Ivan Levkivskyi  wrote:
> On 22 November 2017 at 17:43, Paul Moore  wrote:
>>
>> On 22 November 2017 at 16:30, Ivan Levkivskyi 
>> wrote:
>> > On 22 November 2017 at 17:24, Antoine Pitrou 
>> > wrote:
>> >> Given a comprehension (e.g. list comprehension) is expected to work
>> >> nominally as `constructor(generator expression)`
>> >
>> > As Yury just explained, these two are not equivalent if there is an
>> > `await`
>> > in the comprehension/generator expression.
>>
>> As Antoine said, people *expect* them to work the same.
>
>
> The difference is that a generator expression can be used independently, one
> can assign it to a variable etc. not necessary to wrap it into a list()
> Anyway, can you propose an equivalent "defining" code for both? Otherwise it
> is not clear what you are defending.

I can't propose a precise description of the semantics in the form "(x
for x in expr) is precisely equivalent to <...>" (for generator
expressions or list/dict/set comprehensions). I'm not even sure that
there *is* a definition in those terms - I don't see it as a given
that generator expressions are simply syntax shorthand. It's not
necessary that such an expansion exists - there's no such expansion
for most expressions in Python. The equivalences I would suggest,
you've stated previously in this thread are inaccurate, so it's not
clear to me what options you've left open for me.

What I can say is that the semantics of generator expressions and
comprehensions should match the understanding that has been documented
and taught for many years now. Or at least, if the semantics change,
then documentation, "what's new in Python x.y", and tutorials should
be updated to match. In my view, the following understandings have
been true since comprehensions and generators were introduced, and
should not be changed without careful management of the change. (I
accept that things *have* changed, so we have a retroactive exercise
that we need to do to mitigate any damage to people's understanding
from the changes, but I'm trying to start from what I believe is the
baseline understanding that people have had since pre-async, and don't
yet understand how to modify).

1. List comprehensions expand into nested for/if statements in the
"obvious" way - with an empty list created to start and append used to
add items to it.
   1a. Variables introduced in the comprehension don't "leak" (see below).
2. Generator expressions expand into generators with the same "nested
loop" behaviour, and a yield of the generated value.
3. List comprehensions are the same as list(the equivalent generator
expression).

With regard to 1a, note that Python 2 *did* leak names. The change to
stop those leaks is arguably a minor, low-level detail (much like the
sort of things coming up in regard to yield expressions and async).
But it was considered significant enough to go through a long and
careful debate, and a non-trivial amount of publicity ensuring users
were aware of, and understood, the change. I see that as a good
example of how the semantics *can* change, but we should ensure we
manage users' understanding.

I'm not sure we're going to reach agreement on this. Let it stand that
I believe the above 3 points are key to how people understand
comprehensions/generator expressions, and I find the subtle
discrepancies that have been introduced by async to be difficult to
understand, and counter to my intuition. I don't believe I'm alone in
this.

I can't tell you how to explain this clearly to me - by definition, I
don't know a good explanation. I take it as self-evident that having a
big chunk of Python semantics that leaves people thinking "I'll avoid
this because it's confusing" is bad - maybe you (or others) see it as
simply "advanced concepts" that I (and people like me) can ignore, but
while I usually do just that, threads like this keep cropping up for
which that approach doesn't work...

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ethan Furman]

On 11/22/2017 09:15 AM, Paul Moore wrote:


1. List comprehensions expand into nested for/if statements in the
"obvious" way - with an empty list created to start and append used to
add items to it.
1a. Variables introduced in the comprehension don't "leak" (see below).
2. Generator expressions expand into generators with the same "nested
loop" behaviour, and a yield of the generated value.
3. List comprehensions are the same as list(the equivalent generator
expression).


For what it's worth, that is how I understand it.

--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 18:15, Paul Moore  wrote:

> On 22 November 2017 at 16:47, Ivan Levkivskyi 
> wrote:
> > On 22 November 2017 at 17:43, Paul Moore  wrote:
> >>
> >> On 22 November 2017 at 16:30, Ivan Levkivskyi 
> >> wrote:
> >> > On 22 November 2017 at 17:24, Antoine Pitrou 
> >> > wrote:
> >> >> Given a comprehension (e.g. list comprehension) is expected to work
> >> >> nominally as `constructor(generator expression)`
> >> >
> >> > As Yury just explained, these two are not equivalent if there is an
> >> > `await`
> >> > in the comprehension/generator expression.
> >>
> >> As Antoine said, people *expect* them to work the same.
> >
> >
> > The difference is that a generator expression can be used independently,
> one
> > can assign it to a variable etc. not necessary to wrap it into a list()
> > Anyway, can you propose an equivalent "defining" code for both?
> Otherwise it
> > is not clear what you are defending.
>
> [...snip...]
>
> 1. List comprehensions expand into nested for/if statements in the
> "obvious" way - with an empty list created to start and append used to
> add items to it.
>1a. Variables introduced in the comprehension don't "leak" (see below).
> 2. Generator expressions expand into generators with the same "nested
> loop" behaviour, and a yield of the generated value.
> 3. List comprehensions are the same as list(the equivalent generator
> expression).
>

Great, I agree with all three rules.
But there is a problem, it is hard to make these three rules consistent in
some corner cases even _without async_.
For example, with the original problematic example, it is not clear to me
how to apply the rule 2 so that it is consistent with 3:

def fun_comp():
return [(yield i) for i in range(3)]

def fun_gen():
return list((yield i) for i in range(3))

I think the solution may be to formulate the rules in terms of the iterator
protocol (__iter__ and __next__).
I will try to think more about this.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ethan Furman]

On 11/22/2017 05:03 AM, Serhiy Storchaka wrote:


 From 
https://stackoverflow.com/questions/45190729/differences-between-generator-comprehension-expressions.

 g = [(yield i) for i in range(3)]

Syntactically this looks like a list comprehension, and g should be a list, 
right? But actually it is a generator. This
code is equivalent to the following code:

 def _make_list(it):
 result = []
 for i in it:
 result.append(yield i)
 return result
 g = _make_list(iter(range(3)))

Due to "yield" in the expression _make_list() is not a function returning a 
list, but a generator function returning a
generator.


The [] syntax says g should be list.  Seems to me we could do either of:

1)  raise if the returned object is not a list;
2)  wrap a returned object in a list if it isn't one already;

In other words, (2) would make

   g = [(yield i) for i in range(3)]

and

   g = [((yield i) for i in range(3))]

be the same.

I have no idea how either of those solutions would interact with async/await.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

Wow, 44 messages in 4 hours. That must be some kind of record.

If/when there's an action item, can someone summarize for me?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Serhiy Storchaka]

22.11.17 17:10, Ivan Levkivskyi пише:

- fix yield in comprehensions


I'm working on this. I was going to inline the generating function a 
long time ago for small performance benefit, but the benefit looked too 
small for complicating the compiler code. Now I have other reason for 
doing this.



- await in comprehensions already works


I'm surprised by this. But I don't understand await. If this works, 
maybe it is possible to make working `yield` in generator expressions.



- make both `yield` and `await` a SyntaxError in generator expressions.


This looks as a temporary solution to me. Ideally we should make this 
working too. But this will be harder.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Jelle Zijlstra]

2017-11-22 9:58 GMT-08:00 Guido van Rossum :

> Wow, 44 messages in 4 hours. That must be some kind of record.
>
> If/when there's an action item, can someone summarize for me?
>
> The main disagreement seems to be about what this code should do:

g = [(yield i) for i in range(3)]

Currently, this makes `g` into a generator, not a list. Everybody seems to
agree this is nonintuitive and should be changed.

One proposal is to make it so `g` gets assigned a list, and the `yield`
happens in the enclosing scope (so the enclosing function would have to be
a generator). This was the way things worked in Python 2, I believe.

Another proposal is to make this code a syntax error, because it's
confusing either way. (For what it's worth, that would be my preference.)

There is related discussion about the semantics of list comprehensions
versus calling list() on a generator expression, and of async semantics,
but I don't think there's any clear point of action there.


> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> jelle.zijlstra%40gmail.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 18:15, Paul Moore  wrote:

> On 22 November 2017 at 16:47, Ivan Levkivskyi 
> wrote:
> > On 22 November 2017 at 17:43, Paul Moore  wrote:
> >>
> >> On 22 November 2017 at 16:30, Ivan Levkivskyi 
> >> wrote:
> >> > On 22 November 2017 at 17:24, Antoine Pitrou 
> >> > wrote:
> >> >> Given a comprehension (e.g. list comprehension) is expected to work
> >> >> nominally as `constructor(generator expression)`
> >> >
> >> > As Yury just explained, these two are not equivalent if there is an
> >> > `await`
> >> > in the comprehension/generator expression.
> >>
> >> As Antoine said, people *expect* them to work the same.
> >
> >
> > The difference is that a generator expression can be used independently,
> one
> > can assign it to a variable etc. not necessary to wrap it into a list()
> > Anyway, can you propose an equivalent "defining" code for both?
> Otherwise it
> > is not clear what you are defending.
>
> [...]
>
> 1. List comprehensions expand into nested for/if statements in the
> "obvious" way - with an empty list created to start and append used to
> add items to it.
>1a. Variables introduced in the comprehension don't "leak" (see below).
> 2. Generator expressions expand into generators with the same "nested
> loop" behaviour, and a yield of the generated value.
> 3. List comprehensions are the same as list(the equivalent generator
> expression).
>
>
Paul, OK, I think how to formulate these rules more "precisely" so that
they will be all consistent even if there is a `yield` inside.
The key idea is that neither comprehensions nor generator expressions
should create a function scope surrounding the `expr` in (expr for ind in
iterable) and [expr for ind in iterable].
(this still can be some hidden implementation detail)

So as Serhiy proposed in one of his first posts any comprehensions and
generator expressions with `yield` are not valid outside functions.
If such comprehension or generator expression is inside a function, then it
makes it a generator, and all the `yiled`s are yielded from that generator,
for example:

def fun_gen():
return list((yield i) for i in range(3))

should work as following:

g = func_gen()

g.send(42)
0
g.send(43)
1
g.send(44)
2
try:
g.send(45)
except StopIteration as e:
assert e.value  == [42, 43, 44]

And exactly the same with

def fun_comp():
[(yield i) for i in range(3)]

I hope with this we can close the no-async part of the problem.
Currently this is not how it works, and should be fixed. Do you agree?

The async part can then be considered separately.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

On Wed, Nov 22, 2017 at 10:54 AM, Jelle Zijlstra 
wrote:

>
>
> 2017-11-22 9:58 GMT-08:00 Guido van Rossum :
>
>> Wow, 44 messages in 4 hours. That must be some kind of record.
>>
>> If/when there's an action item, can someone summarize for me?
>>
>> The main disagreement seems to be about what this code should do:
>
> g = [(yield i) for i in range(3)]
>
> Currently, this makes `g` into a generator, not a list. Everybody seems to
> agree this is nonintuitive and should be changed.
>
> One proposal is to make it so `g` gets assigned a list, and the `yield`
> happens in the enclosing scope (so the enclosing function would have to be
> a generator). This was the way things worked in Python 2, I believe.
>
> Another proposal is to make this code a syntax error, because it's
> confusing either way. (For what it's worth, that would be my preference.)
>

Hm, yes, I don't think we should try to preserve the accidental meaning it
had in either Py2 or Py3. Let's make it illegal. (OTOH, await in the same
position must keep working since it's not broken and not unintuitive
either.) Possibly we should make it a (hard) warning in 3.7 and break in
3.8.


> There is related discussion about the semantics of list comprehensions
> versus calling list() on a generator expression, and of async semantics,
> but I don't think there's any clear point of action there.
>

OK.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 19:54, Jelle Zijlstra 
wrote:

>
> One proposal is to make it so `g` gets assigned a list, and the `yield`
> happens in the enclosing scope (so the enclosing function would have to be
> a generator). This was the way things worked in Python 2, I believe.
>
> Another proposal is to make this code a syntax error, because it's
> confusing either way. (For what it's worth, that would be my preference.)
>
>
Concerning this two options it looks like me and Serhiy like the first one,
Paul is undecided (), and Antoine is in favor of option 2.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

On Wed, Nov 22, 2017 at 11:08 AM, Ivan Levkivskyi 
wrote:

> On 22 November 2017 at 19:54, Jelle Zijlstra 
> wrote:
>
>>
>> One proposal is to make it so `g` gets assigned a list, and the `yield`
>> happens in the enclosing scope (so the enclosing function would have to be
>> a generator). This was the way things worked in Python 2, I believe.
>>
>> Another proposal is to make this code a syntax error, because it's
>> confusing either way. (For what it's worth, that would be my preference.)
>>
>>
> Concerning this two options it looks like me and Serhiy like the first
> one, Paul is undecided (), and Antoine is in favor of option 2.
>

While that may be the right thing to do, it's a silent change in semantics,
which I find pretty disturbing -- how would people debug such a failure?
That's why I think we should deprecate or hard break it for at least one
release.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Yury Selivanov]

On Wed, Nov 22, 2017 at 2:08 PM, Ivan Levkivskyi  wrote:
> On 22 November 2017 at 19:54, Jelle Zijlstra 
> wrote:
>>
>>
>> One proposal is to make it so `g` gets assigned a list, and the `yield`
>> happens in the enclosing scope (so the enclosing function would have to be a
>> generator). This was the way things worked in Python 2, I believe.
>>
>> Another proposal is to make this code a syntax error, because it's
>> confusing either way. (For what it's worth, that would be my preference.)
>>
>
> Concerning this two options it looks like me and Serhiy like the first one,
> Paul is undecided (), and Antoine is in favor of option 2.

FWIW I'm in favour of deprecating 'yield' expression in comprehensions
and generator expressions in 3.7 and removing them from 3.8.
async/await support should stay as is.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 20:15, Guido van Rossum  wrote:

> On Wed, Nov 22, 2017 at 11:08 AM, Ivan Levkivskyi 
> wrote:
>
>> On 22 November 2017 at 19:54, Jelle Zijlstra 
>> wrote:
>>
>>>
>>> One proposal is to make it so `g` gets assigned a list, and the `yield`
>>> happens in the enclosing scope (so the enclosing function would have to be
>>> a generator). This was the way things worked in Python 2, I believe.
>>>
>>> Another proposal is to make this code a syntax error, because it's
>>> confusing either way. (For what it's worth, that would be my preference.)
>>>
>>>
>> Concerning this two options it looks like me and Serhiy like the first
>> one, Paul is undecided (), and Antoine is in favor of option 2.
>>
>
> While that may be the right thing to do, it's a silent change in
> semantics, which I find pretty disturbing -- how would people debug such a
> failure?
>

Some may call this just fixing a bug (At least in two mentioned
Stackoverflow questions and in two b.p.o. issues the current behaviour is
considered a bug).
But anyway, it is not me who decides :-)

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 20:05, Guido van Rossum  wrote:

> On Wed, Nov 22, 2017 at 10:54 AM, Jelle Zijlstra  > wrote
>
>> 2017-11-22 9:58 GMT-08:00 Guido van Rossum :
>>
> (OTOH, await in the same position must keep working since it's not broken
>> and not unintuitive either.)
>>
>
>

This is very questionable IMO.
So do you think that [await x for y in z] and list(await x for y in z)
being not equivalent is intuitive?

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Guido van Rossum]

On Wed, Nov 22, 2017 at 11:12 AM, Ivan Levkivskyi 
wrote:

> On 22 November 2017 at 20:05, Guido van Rossum  wrote:
>
>> On Wed, Nov 22, 2017 at 10:54 AM, Jelle Zijlstra <
>> jelle.zijls...@gmail.com> wrote
>>
>>> 2017-11-22 9:58 GMT-08:00 Guido van Rossum :
>>>
>> (OTOH, await in the same position must keep working since it's not broken
>>> and not unintuitive either.)
>>>
>>
>>
>
> This is very questionable IMO.
> So do you think that [await x for y in z] and list(await x for y in z)
> being not equivalent is intuitive?
>

I see, that's why this is such a long thread. :-(

But are they different? I can't find an example where they don't give the
same outcome.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 20:33, Guido van Rossum  wrote:

> On Wed, Nov 22, 2017 at 11:12 AM, Ivan Levkivskyi 
> wrote:
>
>> On 22 November 2017 at 20:05, Guido van Rossum  wrote:
>>
>>> On Wed, Nov 22, 2017 at 10:54 AM, Jelle Zijlstra <
>>> jelle.zijls...@gmail.com> wrote
>>>
 2017-11-22 9:58 GMT-08:00 Guido van Rossum :

>>> (OTOH, await in the same position must keep working since it's not
 broken and not unintuitive either.)

>>>
>>>
>>
>> This is very questionable IMO.
>> So do you think that [await x for y in z] and list(await x for y in z)
>> being not equivalent is intuitive?
>>
>
> I see, that's why this is such a long thread. :-(
>
> But are they different? I can't find an example where they don't give the
> same outcome.
>
>
I think this is a minimal example https://bugs.python.org/issue32113
Also Yury explains there why [await x for y in z ] is different from
list(await x for y in z).
Although I understand why it works this way, TBH it is not very intuitive.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Yury Selivanov]

On Wed, Nov 22, 2017 at 2:37 PM, Ivan Levkivskyi  wrote:
> On 22 November 2017 at 20:33, Guido van Rossum  wrote:
>>
>> On Wed, Nov 22, 2017 at 11:12 AM, Ivan Levkivskyi 
>> wrote:
>>>
>>> On 22 November 2017 at 20:05, Guido van Rossum  wrote:

 On Wed, Nov 22, 2017 at 10:54 AM, Jelle Zijlstra
  wrote
>
> 2017-11-22 9:58 GMT-08:00 Guido van Rossum :
>
> (OTOH, await in the same position must keep working since it's not
> broken and not unintuitive either.)


>>>
>>>
>>> This is very questionable IMO.
>>> So do you think that [await x for y in z] and list(await x for y in z)

Comprehensions are declarative, and that's why [], and {} work with
async/await.  When you're using parens () you *explicitly* tell Python
compiler that you want a generator expression.

And the distinction between comprehensions and generator expressions
also exists for synchronous code:

   x = [a for a in range(10)]
   x[0]

and

   x = (a for a in range(10))
   x[0]  # TypeError

Is the above "intuitive" for all Python users?  Probably not.  Write
it once, get your TypeError, read the error message and you understand
what's going on here.

Is the difference between "[await x for y in z ]" and "list(await x
for y in z)" intuitive for all Python users?  Again, probably not.
But for those who write async code it is.

I also don't recall seeing a lot of `list(x for x in ...)` pattern.
Usually people just use list or dict comprehensions directly (and they
are faster, btw).

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[David Mertz]

Inasmuch as I get to opine, I'm +1 on SyntaxError. There is no behavior for
that spelling that I would find intuitive or easy to explain to students.
And as far as I can tell, the ONLY time anything has ever been spelled that
way is in comments saying "look at this weird edge case behavior in Python."

On Nov 22, 2017 10:57 AM, "Jelle Zijlstra"  wrote:



2017-11-22 9:58 GMT-08:00 Guido van Rossum :

> Wow, 44 messages in 4 hours. That must be some kind of record.
>
> If/when there's an action item, can someone summarize for me?
>
> The main disagreement seems to be about what this code should do:

g = [(yield i) for i in range(3)]

Currently, this makes `g` into a generator, not a list. Everybody seems to
agree this is nonintuitive and should be changed.

One proposal is to make it so `g` gets assigned a list, and the `yield`
happens in the enclosing scope (so the enclosing function would have to be
a generator). This was the way things worked in Python 2, I believe.

Another proposal is to make this code a syntax error, because it's
confusing either way. (For what it's worth, that would be my preference.)

There is related discussion about the semantics of list comprehensions
versus calling list() on a generator expression, and of async semantics,
but I don't think there's any clear point of action there.


> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jelle.
> zijlstra%40gmail.com
>
>

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/
mertz%40gnosis.cx
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Sven R. Kunze]

Isn't yield like a return?

A return in a list/dict/set comprehension makes no sense to me.

So, +1 on SyntaxError from me too.

Cheers.


On 22.11.2017 21:29, David Mertz wrote:
Inasmuch as I get to opine, I'm +1 on SyntaxError. There is no 
behavior for that spelling that I would find intuitive or easy to 
explain to students. And as far as I can tell, the ONLY time anything 
has ever been spelled that way is in comments saying "look at this 
weird edge case behavior in Python."


On Nov 22, 2017 10:57 AM, "Jelle Zijlstra" > wrote:




2017-11-22 9:58 GMT-08:00 Guido van Rossum mailto:gu...@python.org>>:

Wow, 44 messages in 4 hours. That must be some kind of record.

If/when there's an action item, can someone summarize for me?

The main disagreement seems to be about what this code should do:

    g = [(yield i) for i in range(3)]

Currently, this makes `g` into a generator, not a list. Everybody
seems to agree this is nonintuitive and should be changed.

One proposal is to make it so `g` gets assigned a list, and the
`yield` happens in the enclosing scope (so the enclosing function
would have to be a generator). This was the way things worked in
Python 2, I believe.

Another proposal is to make this code a syntax error, because it's
confusing either way. (For what it's worth, that would be my
preference.)

There is related discussion about the semantics of list
comprehensions versus calling list() on a generator expression,
and of async semantics, but I don't think there's any clear point
of action there.

-- 
--Guido van Rossum (python.org/~guido

)

___
Python-Dev mailing list
Python-Dev@python.org 
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe:

https://mail.python.org/mailman/options/python-dev/jelle.zijlstra%40gmail.com





___
Python-Dev mailing list
Python-Dev@python.org 
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe:
https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx





___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ethan Furman]

On 11/22/2017 11:01 AM, Ivan Levkivskyi wrote:


I think how to formulate these rules more "precisely" so that they will be all 
consistent even if there is a
`yield` inside.
The key idea is that neither comprehensions nor generator expressions should 
create a function scope surrounding the
`expr` in (expr for ind in iterable) and [expr for ind in iterable].
(this still can be some hidden implementation detail)

So as Serhiy proposed in one of his first posts any comprehensions and 
generator expressions with `yield` are not valid
outside functions.
If such comprehension or generator expression is inside a function, then it 
makes it a generator, and all the `yield`s
are yielded from that generator, for example:


Whether it's inside or outside a function should be irrelevant -- a comprehension / generator expression should have no 
influence on the type of the resulting function (and at least synchronous comprehensions / generator expressions should 
be able to live outside of a function).



def fun_gen():
 return list((yield i) for i in range(3))


The return says it's returning a list, so that's what it should be returning.


should work as following:

g = func_gen()

g.send(42)
0
g.send(43)
1
g.send(44)
2
try:
 g.send(45)
except StopIteration as e:
 assert e.value  == [42, 43, 44]

And exactly the same with

def fun_comp():
 [(yield i) for i in range(3)]

I hope with this we can close the no-async part of the problem.
Currently this is not how it works, and should be fixed. Do you agree?


No.

NB: If we go with making yield inside a comprehension / generator expression a 
SyntaxError then this subthread can die.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

22 Лис 2017 22:19 "Ethan Furman"  пише:

On 11/22/2017 11:01 AM, Ivan Levkivskyi wrote:

I think how to formulate these rules more "precisely" so that they will be
> all consistent even if there is a
> `yield` inside.
> The key idea is that neither comprehensions nor generator expressions
> should create a function scope surrounding the
> `expr` in (expr for ind in iterable) and [expr for ind in iterable].
> (this still can be some hidden implementation detail)
>
> So as Serhiy proposed in one of his first posts any comprehensions and
> generator expressions with `yield` are not valid
> outside functions.
> If such comprehension or generator expression is inside a function, then
> it makes it a generator, and all the `yield`s
>
> are yielded from that generator, for example:
>

Whether it's inside or outside a function should be irrelevant -- a
comprehension / generator expression should have no influence on the type
of the resulting function (and at least synchronous comprehensions /
generator expressions should be able to live outside of a function).


def fun_gen():
>  return list((yield i) for i in range(3))
>

The return says it's returning a list, so that's what it should be returning


def f():
yield 1
return list()

Here return also says it should return a list, so this is not an argument.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ethan Furman]

On 11/22/2017 01:25 PM, Ivan Levkivskyi wrote:

22 Лис 2017 22:19 "Ethan Furman" пише:



Whether it's inside or outside a function should be irrelevant -- a 
comprehension / generator expression should have
no influence on the type of the resulting function (and at least synchronous 
comprehensions / generator expressions
should be able to live outside of a function).

def fun_gen():
  return list((yield i) for i in range(3))

The return says it's returning a list, so that's what it should be returning


def f():
 yield 1
 return list()

Here return also says it should return a list, so this is not an argument.


Right, the argument is that calling the `list` constructor should return a list -- not a database proxy, not a web page, 
and not a generator.


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

22 Лис 2017 22:33 "Ethan Furman"  пише:

On 11/22/2017 01:25 PM, Ivan Levkivskyi wrote:

> 22 Лис 2017 22:19 "Ethan Furman" пише:
>

Whether it's inside or outside a function should be irrelevant -- a
>> comprehension / generator expression should have
>> no influence on the type of the resulting function (and at least
>> synchronous comprehensions / generator expressions
>> should be able to live outside of a function).
>>
>> def fun_gen():
>>   return list((yield i) for i in range(3))
>>
>> The return says it's returning a list, so that's what it should be
>> returning
>>
>
> def f():
>  yield 1
>  return list()
>
> Here return also says it should return a list, so this is not an argument.
>

Right, the argument is that calling the `list` constructor should return a
list -- not a database proxy, not a web page, and not a generator.


Then you didn't read my example carefully, since the whole point is that
the list constructor there returns a list.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Ivan Levkivskyi wrote:

while `g = list((yield i) for i in range(3))` is defined as this code:

def __gen():
   for i in range(3):
 yield (yield i)
g = list(__gen())


Since this is almost certainly not what was intended, I think that
'yield' inside a generator expression should simply be disallowed.

The problem with making it work intuitively is that there's no way
to distinguish between the implicit yields producing values for the
generator comprehension and the explicit ones that the programmer
is expecting to be passed through and yielded from the enclosing
generator function.

Fixing that would require adding another whole layer of complexity
similar to what Yuri did to support async generators, and I can't
see that being worth the effort.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Script bootstrapping executables on Windows

[Thomas Mansencal]

Hi,

I hope that what follows will be useful for other people: after stepping
through code for a few hours this morning, I ended up finding the location
of the boostrapping executable. It (they) actually ship with setuptools,
e.g. C:\Python27\Lib\site-packages\setuptools\cli-64.exe and get copied and
renamed with the script name. The source code is here:
https://github.com/pypa/setuptools/blob/master/launcher.c
, and I was
able to find the error mentioned in OP:
https://github.com/pypa/setuptools/blob/master/launcher.c#L209

Cheers,

Thomas

On Wed, Nov 22, 2017 at 5:08 PM Thomas Mansencal 
wrote:

> Hi,
>
> This is a Windows specific question, to give a bit of context I'm working
> in a studio where depending on the project we use different Python
> interpreters installed in different locations, e.g. Python 2.7.13, Python
> 2.7.14, Python 3.6. We set PATH, PYTHONHOME and PYTHONPATH accordingly
> depending the interpreter in use.
>
> Our Python packages are living atomically on the network and are added to
> the environment on a per project basis by extending PYTHONPATH. This is in
> contrast to using a monolithic virtual environment built with virtualenv or
> conda. Assuming it is compatible, a Python package might be used with any
> of the 3 aforementioned interpreters, e.g. yapf (a code formatter).
>
> Now, on Windows, if you for example *pip install yapf*, a yapf.exe
> boostrapping executable and a yapf-script.py file are being generated. The
> boostrapping executable seems to look for the yapf-script.py file and
> launch it using the absolute hardcoded interpreter path of the
> yapf-script.py shebang.
>
> Given the above we run into issues if for example yapf was deployed using
> Python 2.7.13 but the Python 2.7.14 interpreter is being used in the
> environment instead. We get a "failed to create process." error in that
> case.
>
> What we would like to do is not being tied to the absolute interpreter
> path but have it defined with a variable or just use #!python. I have tried
> to search for the above error in cpython source code and the installation
> directory without luck. I would like to know what module/package is
> responsible for generating the boostrapping executables to understand how
> it works and see if we can doctor it for our usage.
>
> Bests,
>
> Thomas
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Paul Moore wrote:

At the moment, I know I tend to treat Python semantics as I always
did, but with an implicit proviso, "unless async is involved, when I
can't assume any of my intuitions apply". That's *not* a good
situation to be in.


It's disappointing that PEP 3152 was rejected, because I think
it would have helped people like you by providing a mental
model for async stuff that's much easier to reason about.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 22 November 2017 at 21:07, Yury Selivanov 
wrote:

> On Wed, Nov 22, 2017 at 2:37 PM, Ivan Levkivskyi 
> wrote:
> > On 22 November 2017 at 20:33, Guido van Rossum  wrote:
> >>
> >> On Wed, Nov 22, 2017 at 11:12 AM, Ivan Levkivskyi  >
> >> wrote:
> >>>
> >>> On 22 November 2017 at 20:05, Guido van Rossum 
> wrote:
> 
>  On Wed, Nov 22, 2017 at 10:54 AM, Jelle Zijlstra
>   wrote
> >
> > 2017-11-22 9:58 GMT-08:00 Guido van Rossum :
> >
> > (OTOH, await in the same position must keep working since it's not
> > broken and not unintuitive either.)
> 
> 
> >>>
> >>>
> >>> This is very questionable IMO.
> >>> So do you think that [await x for y in z] and list(await x for y in z)
>
> Comprehensions are declarative, and that's why [], and {} work with
> async/await.  When you're using parens () you *explicitly* tell Python
> compiler that you want a generator expression.
>
> And the distinction between comprehensions and generator expressions
> also exists for synchronous code:
>
>x = [a for a in range(10)]
>x[0]
>
> and
>
>x = (a for a in range(10))
>x[0]  # TypeError
>
> Is the above "intuitive" for all Python users?  Probably not.  Write
> it once, get your TypeError, read the error message and you understand
> what's going on here.
>
> Is the difference between "[await x for y in z ]" and "list(await x
> for y in z)" intuitive for all Python users?  Again, probably not.
> But for those who write async code it is.
>

Just found another example of intuitive behaviour:

>>> async def f():
... for i in range(3):
... yield i
...
>>> async def g():
... return [(yield i) async for i in f()]
...
>>> g().send(None)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 2, in g
TypeError: object async_generator can't be used in 'await' expression

of course it is obvious for anyone who writes async code, but anyway an
interesting example.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Yury Selivanov]

On Wed, Nov 22, 2017 at 6:46 PM, Ivan Levkivskyi  wrote:
[..]
> Just found another example of intuitive behaviour:
>
 async def f():
> ... for i in range(3):
> ... yield i
> ...
 async def g():
> ... return [(yield i) async for i in f()]
> ...
 g().send(None)
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "", line 2, in g
> TypeError: object async_generator can't be used in 'await' expression
>
> of course it is obvious for anyone who writes async code, but anyway an
> interesting example.

I wouldn't say that it's obvious to anyone...

I think this thread has started to discuss the use of 'yield'
expression in comprehensions, and the outcome of the discussion is
that everyone thinks that we should deprecate that syntax in 3.7,
remove in 3.8.  Let's start with that? :)

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ethan Furman]

On 11/22/2017 01:40 PM, Ivan Levkivskyi wrote:

22 Лис 2017 22:33 "Ethan Furman" mailto:et...@stoneleaf.us>> пише:



Right, the argument is that calling the `list` constructor should return a list 
-- not a database proxy, not a web
page, and not a generator.


Then you didn't read my example carefully, since the whole point is that the 
list constructor there returns a list.


This example?

try:
g.send(45)
except StopIteration as e:
assert e.value  == [42, 43, 44]

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 559 - built-in noop()

[Victor Stinner]

Aha, contextlib.nullcontext() was just added, cool!

https://github.com/python/cpython/commit/0784a2e5b174d2dbf7b144d480559e650c5cf64c
https://bugs.python.org/issue10049

Victor

2017-09-09 21:54 GMT+02:00 Victor Stinner :
> I always wanted this feature (no kidding).
>
> Would it be possible to add support for the context manager?
>
> with noop(): ...
>
> Maybe noop can be an instance of:
>
> class Noop:
>   def __enter__(self, *args, **kw): return self
>   def __exit__(self, *args): pass
>   def __call__(self, *args, **kw): return self
>
> Victor
>
> Le 9 sept. 2017 11:48 AM, "Barry Warsaw"  a écrit :
>>
>> I couldn’t resist one more PEP from the Core sprint.  I won’t reveal where
>> or how this one came to me.
>>
>> -Barry
>>
>> PEP: 559
>> Title: Built-in noop()
>> Author: Barry Warsaw 
>> Status: Draft
>> Type: Standards Track
>> Content-Type: text/x-rst
>> Created: 2017-09-08
>> Python-Version: 3.7
>> Post-History: 2017-09-09
>>
>>
>> Abstract
>> 
>>
>> This PEP proposes adding a new built-in function called ``noop()`` which
>> does
>> nothing but return ``None``.
>>
>>
>> Rationale
>> =
>>
>> It is trivial to implement a no-op function in Python.  It's so easy in
>> fact
>> that many people do it many times over and over again.  It would be useful
>> in
>> many cases to have a common built-in function that does nothing.
>>
>> One use case would be for PEP 553, where you could set the breakpoint
>> environment variable to the following in order to effectively disable it::
>>
>> $ setenv PYTHONBREAKPOINT=noop
>>
>>
>> Implementation
>> ==
>>
>> The Python equivalent of the ``noop()`` function is exactly::
>>
>> def noop(*args, **kws):
>> return None
>>
>> The C built-in implementation is available as a pull request.
>>
>>
>> Rejected alternatives
>> =
>>
>> ``noop()`` returns something
>> 
>>
>> YAGNI.
>>
>> This is rejected because it complicates the semantics.  For example, if
>> you
>> always return both ``*args`` and ``**kws``, what do you return when none
>> of
>> those are given?  Returning a tuple of ``((), {})`` is kind of ugly, but
>> provides consistency.  But you might also want to just return ``None``
>> since
>> that's also conceptually what the function was passed.
>>
>> Or, what if you pass in exactly one positional argument, e.g. ``noop(7)``.
>> Do
>> you return ``7`` or ``((7,), {})``?  And so on.
>>
>> The author claims that you won't ever need the return value of ``noop()``
>> so
>> it will always return ``None``.
>>
>> Coghlin's Dialogs (edited for formatting):
>>
>> My counterargument to this would be ``map(noop, iterable)``,
>> ``sorted(iterable, key=noop)``, etc. (``filter``, ``max``, and
>> ``min`` all accept callables that accept a single argument, as do
>> many of the itertools operations).
>>
>> Making ``noop()`` a useful default function in those cases just
>> needs the definition to be::
>>
>>def noop(*args, **kwds):
>>return args[0] if args else None
>>
>> The counterargument to the counterargument is that using ``None``
>> as the default in all these cases is going to be faster, since it
>> lets the algorithm skip the callback entirely, rather than calling
>> it and having it do nothing useful.
>>
>>
>> Copyright
>> =
>>
>> This document has been placed in the public domain.
>>
>>
>> ..
>>Local Variables:
>>mode: indented-text
>>indent-tabs-mode: nil
>>sentence-end-double-space: t
>>fill-column: 70
>>coding: utf-8
>>End:
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Nick Coghlan]

On 22 November 2017 at 21:12, Victor Stinner 
wrote:

> 2017-11-22 12:04 GMT+01:00 Antoine Pitrou :
> > IMHO this really needs a simple solution documented somewhere.  Also,
> > hopefully when you do the wrong thing, you get a clear error message to
> > know how to fix your code?
>
> Right now, calling PyMem_RawMalloc() before calling
> _PyRuntime_Initialize() calls the function at address NULL, so you get
> a segmentation fault.
>
> Documenting the new requirements is part of the discussion, it's one
> option how to fix this issue.
>

My own recommendation is that we add Eric's new test case to the embedding
test suite and just make sure it works:

wchar_t *program = Py_DecodeLocale("spam", NULL);
Py_SetProgramName(program);
Py_Initialize();
Py_Finalize();
PyMem_RawFree(program);

It does place some additional constraints on us in terms of handling static
initialization of the allocator state, and ensuring we revert back to that
state in Py_Finalize, but I think it's the only way we're going to be able
to reliably replace all calls to malloc & free with PyMem_RawMalloc and
PyMem_RawFree without causing weird problems.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Paul Moore wrote:

3. List comprehensions are the same as list(the equivalent generator
expression).


I don't think that's ever been quite true -- there have
always been odd cases such as what happens if you
raise StopIteration in list(generator_expression).

To my mind, these equivalences have never been intended
as exact descriptions of the semantics, but just a way
of quickly getting across the general idea. Further
words are needed to pin down all the fine details.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Greg Ewing]

Ivan Levkivskyi wrote:
The key idea is that neither comprehensions nor generator expressions 
should create a function scope surrounding the `expr`


I don't see how you can avoid an implicit function scope in
the case of a generator expression, though. And I can't see
how to make yield in a generator expression do anything
sensible.

Consider this:

   def g():
  return ((yield i) for i in range(10))

Presumably the yield should turn g into a generator, but...
then what? My brain is hurting trying to figure out what
it should do.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 23 November 2017 at 14:36, Greg Ewing 
wrote:

> Paul Moore wrote:
>
>> 3. List comprehensions are the same as list(the equivalent generator
>> expression).
>>
>
> I don't think that's ever been quite true -- there have
> always been odd cases such as what happens if you
> raise StopIteration in list(generator_expression).
>
> To my mind, these equivalences have never been intended
> as exact descriptions of the semantics, but just a way
> of quickly getting across the general idea. Further
> words are needed to pin down all the fine details.
>

Getting the name resolution to be identical was definitely one of my goals
when working on the Python 3 comprehension scoping changes.

The fact that implicit scopes and yield expressions interact strangely was
just a pre-existing oddity from when PEP 342 was first implemented (and one
we were able to avoid for async/await by retaining the same "await is only
permitted in async comprehensions" constraint that exists for explicit
scopes).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Chris Angelico]

On Thu, Nov 23, 2017 at 3:36 PM, Greg Ewing  wrote:
> Paul Moore wrote:
>>
>> 3. List comprehensions are the same as list(the equivalent generator
>> expression).
>
>
> I don't think that's ever been quite true -- there have
> always been odd cases such as what happens if you
> raise StopIteration in list(generator_expression).

You mean if the genexp leaks one? That's basically an error either way
- the genexp will raise RuntimeError, but it's still an exception.

>>> from __future__ import generator_stop
>>> def boom(): raise StopIteration
...
>>> [x if x < 3 else boom() for x in range(5)]
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1, in 
  File "", line 1, in boom
StopIteration
>>> list(x if x < 3 else boom() for x in range(5))
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1, in boom
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "", line 1, in 
RuntimeError: generator raised StopIteration
>>>

So that's _one_ difference removed (mostly).

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Stephen J. Turnbull]

Paul Moore writes:

 > 1. List comprehensions expand into nested for/if statements in the
 > "obvious" way - with an empty list created to start and append used to
 > add items to it.
 >1a. Variables introduced in the comprehension don't "leak" (see below).
 > 2. Generator expressions expand into generators with the same "nested
 > loop" behaviour, and a yield of the generated value.
 > 3. List comprehensions are the same as list(the equivalent generator
 > expression).

I'm a little late to this discussion, but I don't see how 3 can be
true if 1 and 2 are.

Because I believe my model of generator expressions is pretty simple,
I'll present it here:

- the generator expression in 2 and 3 implicitly creates a generator
  function containing the usual loop
  - the point being that it will "capture" all of the yields (implicit
*and* explicit) in the generator expression
- then implicitly invokes it, 
- passing the (iterable) generator object returned to the containing
  expression.

"Look Ma, no yields (or generator functions) left!"

However, my model of comprehensions is exactly a for loop that appends
to an empty list repeatedly, but doesn't leak iteration variables.  So
a yield in a comprehension turns the function *containing* the
comprehension (which may or may not exist) into a generator function.
In other words, I don't agree with Ethan that a yield inside a list
comprehension should not affect the generator-ness of the containing
function.  What would this mean under that condition:

[f(x) for x in (yield iterable)]

then?  IOW,

x = ((yield i) for i in iterable)

IS valid syntax at top level, while

x = [(yield i) for i in iterable]

IS NOT valid syntax at top level given those semantics.  The latter
"works" in Python 3.6;

>>> for i in [(yield i) for i in (1, 2, 3)]:
...  i
... 
1
2
3

though I think it should be a syntax error, and "bare" yield does not
"work":

>>> i = 1
>>> yield i
  File "", line 1
SyntaxError: 'yield' outside function
>>> (yield i)
  File "", line 1
SyntaxError: 'yield' outside function

FWIW, that's the way I'd want it, and the way I've always understood
comprehensions and generator expressions.  I think this is consistent
with Yuri's, Serhiy's, and Ivan's claims, but I admit I'm not sure.
Of course the compiler need not create a generator function and invoke
it, but the resulting bytecode should be the same as if it did.

This means the semantics of [FOR-EXPRESSION] and [(FOR-EXPRESSION)]
should differ in the same way.

Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 23 November 2017 at 15:33, Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> However, my model of comprehensions is exactly a for loop that appends
> to an empty list repeatedly, but doesn't leak iteration variables.


Not since Python 3.0. Instead, they create a nested function, the same way
generator expressions do (which is why the name resolution semantics are
now identical between the two cases).

The differences in structure between the four cases (genexp, list/set/dict
comprehensions) then relate mainly to what the innermost loop does:

result.append(expr) # list comp
result.add(expr) # set comp
result[k] = v # dict comp
yield expr # genexp

Thus, when the expression itself is a yield expression, you get:

result.append(yield expr) # list comp
result.add(yield expr) # set comp
result[k] = (yield v) # dict comp
yield (yield expr) # genexp

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Stephen J. Turnbull]

Greg Ewing writes:

 > Consider this:
 > 
 > def g():
 >return ((yield i) for i in range(10))
 > 
 > Presumably the yield should turn g into a generator, but...
 > then what? My brain is hurting trying to figure out what
 > it should do.

I don't understand why you presume that.  The generator expression
doesn't do that anywhere else.  My model is that implicitly the
generator expression is creating a function that becomes a generator
factory, which is implicitly called to return the iterable generator
object, which contains the yield.  Because the call takes place
implicitly = at compile time, all the containing function "sees" is an
iterable (which happens to be a generator object).  "Look Ma, no
yields left!"  And then g returns the generator object.

What am I missing?

In other words, g above is equivalent to

def g():
def _g():
for i in range(10):
# the outer yield is the usual implicit yield from the
# expansion of the generator expression, and the inner
# yield is explicit in your code.
yield (yield i)
return _g()

(modulo some issues of leaking identifiers).  I have not figured out
why either your g or my g does what it does, but they do the same
thing.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Tres Seaver]

On 11/22/2017 07:00 PM, Yury Selivanov wrote:

> I wouldn't say that it's obvious to anyone...
> 
> I think this thread has started to discuss the use of 'yield'
> expression in comprehensions, and the outcome of the discussion is
> that everyone thinks that we should deprecate that syntax in 3.7,
> remove in 3.8.  Let's start with that? :)

You guys are no fun:  such a change would remove at least one evil
"bwah-ha-ha" cackle from David Beazley's next PyCon talk. :)


Tres.
-- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 01:00, Yury Selivanov 
wrote:

> On Wed, Nov 22, 2017 at 6:46 PM, Ivan Levkivskyi 
> wrote:
> [..]
> > Just found another example of intuitive behaviour:
> >
>  async def f():
> > ... for i in range(3):
> > ... yield i
> > ...
>  async def g():
> > ... return [(yield i) async for i in f()]
> > ...
>  g().send(None)
> > Traceback (most recent call last):
> >   File "", line 1, in 
> >   File "", line 2, in g
> > TypeError: object async_generator can't be used in 'await' expression
> >
> > of course it is obvious for anyone who writes async code, but anyway an
> > interesting example.
>
> I wouldn't say that it's obvious to anyone...
>
> I think this thread has started to discuss the use of 'yield'
> expression in comprehensions, and the outcome of the discussion is
> that everyone thinks that we should deprecate that syntax in 3.7,
> remove in 3.8.  Let's start with that? :)


I am not sure everyone agrees with this. My main obstacle is following,
consider motivation for the `await` part of PEP 530
which is in my understanding is roughly like this:

"People sometimes want to refactor for-loops containing `await` into a
comprehension but that doesn't work (particularly because of the hidden
function scope) - lets fix this"

I don't see how this compare to:

"People sometimes want to refactor for-loops containing `yield` into a
comprehension but that doesn't work (particularly because of the hidden
function scope) - lets make it a SyntaxError"

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Ivan Levkivskyi]

On 23 November 2017 at 05:44, Greg Ewing 
wrote:

> Ivan Levkivskyi wrote:
>
>> The key idea is that neither comprehensions nor generator expressions
>> should create a function scope surrounding the `expr`
>>
>
> I don't see how you can avoid an implicit function scope in
> the case of a generator expression, though. And I can't see
> how to make yield in a generator expression do anything
> sensible.
>
> Consider this:
>
>def g():
>   return ((yield i) for i in range(10))
>
> Presumably the yield should turn g into a generator, but...
> then what? My brain is hurting trying to figure out what
> it should do.
>
>
I think this code should be just equivalent to this code

def g():
temp = [(yield i) for i in range(10)]
return (v for v in temp)

Semantics of the comprehension should be clear here (just an equivalent
for-loop without name leaking)

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Sven R. Kunze]

On 23.11.2017 08:38, Ivan Levkivskyi wrote:

I think this code should be just equivalent to this code

    def g():
    temp = [(yield i) for i in range(10)]
    return (v for v in temp)

Semantics of the comprehension should be clear here (just an 
equivalent for-loop without name leaking)


Excuse me if I disagree here. If I were to understand this in real-world 
code, I cannot imagine what will happen here.


A "yield" within a comprehension is like a "return" in a comprehension. 
It makes no sense at all.

Also a "yield" and a "return with value" is also rarely seen.

Comprehensions build new objects, they are not for control flow, IMO.

Cheers,
Sven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com