Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
24.11.17 02:50, Nick Coghlan пише: If we went down that path, then a list comprehension like the following: results = [(yield future) for future in list_of_futures] might be compiled as being equivalent to: def __listcomp_generator(iterable): result = [] for future in iterable: result.append((yield future)) return result results = yield from _listcomp_generator(list_of_futures) The only difference between the current comprehension code and this idea is "an explicit yield expression in a comprehension implies the use of 'yield from' when calling the nested function". Oh, nice! This is a much simpler solution! And it solves all related problems. I like it. This has an overhead in comparison with inlining the code, but the latter can be considered just as an optimization. We can apply it when prove the need in optimizing this construction. For now it is enough if it just works. The fact that two independent mental models lead to the same result is an argument for their correctness. I'm not so sure about "yield" in generators, this will need further thoughts and experiments. "yield" can be used not only in the item expression, but in conditions and inner iterables. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
24.11.17 00:20, Greg Ewing пише: Serhiy Storchaka wrote: Ivan explained that this function should be rough equivalent to def f(): t = [(yield i) for i in range(3)] return (x for x in t) This seems useless to me. It turns a lazy iterator into an eager one, which is a gross violation of the author's intent in using a generator expression. This is a *rough* equivalent. There are differences in details. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On 25 November 2017 at 04:30, Guido van Rossum wrote: > On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum > wrote: > >> The more I hear about this topic, the more I think that `await`, `yield` >> and `yield from` should all be banned from occurring in all comprehensions >> and generator expressions. That's not much different from disallowing >> `return` or `break`. >> > > From the responses it seems that I tried to simplify things too far. Let's > say that `await` in comprehensions is fine, as long as that comprehension > is contained in an `async def`. While we *could* save `yield [from]` in > comprehensions, I still see it as mostly a source of confusion, and the > fact that the presence of `yield [from]` *implicitly* makes the surrounding > `def` a generator makes things worse. It just requires too many mental > contortions to figure out what it does. > There were some arguments that `await` is like a function call, while `yield` is like `return`. TBH, I don't really like these arguments since to me they are to vague. Continuing this logic one can say that `return` is just a fancy function call (calling continuation with the result). To me there is one clear distinction: `return` and `break` are statements, while `yield`, `yield from`, and `await` are expressions. Continuing the topic of the ban, what exactly should be banned? For example will this still be valid? def pack_two(): return [(yield), (yield)] # Just a list display I don't see how this is controversial. It is clear that `pack_two` is a generator. If this is going to be prohibited, then one may be surprised by lack of referential transparency, since this will be valid: def pack_two(): first = (yield) second = (yield) return [first, second] If the first example will be allowed, then one will be surprised why it can't be rewritten as def pack_two(): return [(yield) for _ in range(2)] I have found several other examples where it is not clear whether they should be prohibited with `yield` or not. I still propose to rule out all of the above from generator expressions, > because those can escape from the surrounding scope. > Here I agree. Also note that the above problem does not apply to generator expressions since (x, x) and (x for _ in range(2)) are two very different expressions. -- Ivan ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Allow tuple unpacking in return and yield statements
First time contributing back -- if I should be filing a PEP or something like that for this, please let me know. Coming from https://bugs.python.org/issue32117, unparenthesized tuple unpacking is allowed in assignments: rest = (4, 5, 6) a = 1, 2, 3, *rest but not in yield or return statements (these result in SyntaxErrors): return 1, 2, 3, *rest yield 1, 2, 3, *rest The unpacking in assignments was enabled by a pre-3.2 commit that I haven't yet been able to track back to a discussion, but I suspect this asymmetry is unintentional. Here's the original commit: https://github.com/python/cpython/commit/4905e80c3d2f6abb613d212f0313d1dfe09475dc I've submitted a patch (CLA is signed and submitted, not yet processed), and Serihy said that since it changes the grammar I should have it reviewed here and have signoff by the BDFL. While I haven't had a need for this myself, it was brought up by a user on StackOverflow (https://stackoverflow.com/questions/47272460/python-tuple-unpacking-in-return-statement/47326859). Thanks! Dave ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On 25 November 2017 at 14:55, Ivan Levkivskyi wrote: > Continuing the topic of the ban, what exactly should be banned? For example > will this still be valid? > > def pack_two(): > return [(yield), (yield)] # Just a list display > > I don't see how this is controversial. It is clear that `pack_two` is a > generator. It's not clear to me... Seriously, I wouldn't know what this would do. Presumably it needs someone calling send() to generate results (because that's now yield expressions work) but beyond that, I don't really understand what it does. Maybe an example that wasn't artificial would be more obvious, I'm not sure. > If this is going to be prohibited, then one may be surprised by lack of > referential transparency, since this will be valid: > > def pack_two(): > first = (yield) > second = (yield) > return [first, second] The fact that you can't inline first and second doesn't bother me. I can't fully explain why, but it doesn't. (Technically, I *can* explain why, but you'd disagree with my explanation :-)) > If the first example will be allowed, then one will be surprised why it > can't be rewritten as > > def pack_two(): > return [(yield) for _ in range(2)] > > I have found several other examples where it is not clear whether they > should be prohibited with `yield` or not. So far, none of your examples have demonstrated anything that Guido's suggestion to ban yield would make confusing *to me*. Maybe this demonstrates nothing more than how inconsistent and shallow my understanding of yield expressions is. That's fine, I can live with that. I can't give you any assurances that my level of understanding is common among non-experts, but I will say that in my view, Guido's proposal feels sensible and intuitive. And after all, if the use of yield expressions becomes significantly more common, and the general level of familiarity with the concept increases, it's easy enough to relax the restriction later, in line with the average user's level of comfort. Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
At this point, the fact that several Python core developers fail to understand the pieces of code presented as examples should be a hint that the syntax here is far from desirable... Regards Antoine. On Sat, 25 Nov 2017 15:47:14 + Paul Moore wrote: > On 25 November 2017 at 14:55, Ivan Levkivskyi wrote: > > Continuing the topic of the ban, what exactly should be banned? For example > > will this still be valid? > > > > def pack_two(): > > return [(yield), (yield)] # Just a list display > > > > I don't see how this is controversial. It is clear that `pack_two` is a > > generator. > > It's not clear to me... > > Seriously, I wouldn't know what this would do. Presumably it needs > someone calling send() to generate results (because that's now yield > expressions work) but beyond that, I don't really understand what it > does. Maybe an example that wasn't artificial would be more obvious, > I'm not sure. > > > If this is going to be prohibited, then one may be surprised by lack of > > referential transparency, since this will be valid: > > > > def pack_two(): > > first = (yield) > > second = (yield) > > return [first, second] > > The fact that you can't inline first and second doesn't bother me. I > can't fully explain why, but it doesn't. (Technically, I *can* explain > why, but you'd disagree with my explanation :-)) > > > If the first example will be allowed, then one will be surprised why it > > can't be rewritten as > > > > def pack_two(): > > return [(yield) for _ in range(2)] > > > > I have found several other examples where it is not clear whether they > > should be prohibited with `yield` or not. > > So far, none of your examples have demonstrated anything that Guido's > suggestion to ban yield would make confusing *to me*. > > Maybe this demonstrates nothing more than how inconsistent and shallow > my understanding of yield expressions is. That's fine, I can live with > that. I can't give you any assurances that my level of understanding > is common among non-experts, but I will say that in my view, Guido's > proposal feels sensible and intuitive. And after all, if the use of > yield expressions becomes significantly more common, and the general > level of familiarity with the concept increases, it's easy enough to > relax the restriction later, in line with the average user's level of > comfort. > > Paul ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Sat, Nov 25, 2017 at 6:55 AM, Ivan Levkivskyi
wrote:
> On 25 November 2017 at 04:30, Guido van Rossum wrote:
>
>> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum
>> wrote:
>>
>>> The more I hear about this topic, the more I think that `await`, `yield`
>>> and `yield from` should all be banned from occurring in all comprehensions
>>> and generator expressions. That's not much different from disallowing
>>> `return` or `break`.
>>>
>>
>> From the responses it seems that I tried to simplify things too far.
>> Let's say that `await` in comprehensions is fine, as long as that
>> comprehension is contained in an `async def`. While we *could* save `yield
>> [from]` in comprehensions, I still see it as mostly a source of confusion,
>> and the fact that the presence of `yield [from]` *implicitly* makes the
>> surrounding `def` a generator makes things worse. It just requires too many
>> mental contortions to figure out what it does.
>>
>
> There were some arguments that `await` is like a function call, while
> `yield` is like `return`.
> TBH, I don't really like these arguments since to me they are to vague.
> Continuing this logic one can say that
> `return` is just a fancy function call (calling continuation with the
> result). To me there is one clear distinction:
> `return` and `break` are statements, while `yield`, `yield from`, and
> `await` are expressions.
>
Indeed. However, `yield from` as an expression is mostly a deprecated way
to write `await`, and `yield` as an expression is mostly an alternative way
of writing coroutines (it's the only way that exists in Python 2). Another
big difference is that the use of `yield [from]` affects the surrounding
function, making it a generator.
Continuing the topic of the ban, what exactly should be banned? For example
> will this still be valid?
>
> def pack_two():
> return [(yield), (yield)] # Just a list display
>
It's not a comprehension so it's still valid.
> I don't see how this is controversial. It is clear that `pack_two` is a
> generator.
> If this is going to be prohibited, then one may be surprised by lack of
> referential transparency, since this will be valid:
>
> def pack_two():
> first = (yield)
> second = (yield)
> return [first, second]
>
> If the first example will be allowed, then one will be surprised why it
> can't be rewritten as
>
> def pack_two():
> return [(yield) for _ in range(2)]
>
And yet Nick's example shows that that is not equivalent!
def example():
comp1 = yield from [(yield x) for x in ('1st', '2nd')]
comp2 = yield from [(yield x) for x in ('3rd', '4th')]
return comp1, comp2
In this example each thing that looks syntactically like a list
comprehension becomes actually a generator expression at at runtime! And so
does your example, so instead of a list of two items, it returns a
generator that will produce two values when iterated over.
That's not referential transparency to me, it feels more like a bug in the
code generator.
I want to ban this because apparently nobody besides Nick knows about this
behavior (I certainly didn't, and from the above it seems you don't either).
> I have found several other examples where it is not clear whether they
> should be prohibited with `yield` or not.
>
Such as?
> I still propose to rule out all of the above from generator expressions,
>> because those can escape from the surrounding scope.
>>
>
> Here I agree. Also note that the above problem does not apply to generator
> expressions since (x, x) and (x for _ in range(2)) are
> two very different expressions.
>
PS. A more radical proposal (not for 3.7) would be to deprecate yield as an
expression. It once was only a statement, but PEP 342 introduced yield as
an expression in order to do coroutines. We now have `async def` and
`await` as a superior coroutine mechanism. But we must continue to support
yield expressions because there is a lot of Python 2/3 compatible code that
depends on it. (E.g. Tornado.)
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On 25 November 2017 at 16:57, Guido van Rossum wrote: > On Sat, Nov 25, 2017 at 6:55 AM, Ivan Levkivskyi > wrote: > >> On 25 November 2017 at 04:30, Guido van Rossum wrote: >> >>> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum >>> wrote: >>> The more I hear about this topic, the more I think that `await`, `yield` and `yield from` should all be banned from occurring in all comprehensions and generator expressions. That's not much different from disallowing `return` or `break`. >>> >>> From the responses it seems that I tried to simplify things too far. >>> Let's say that `await` in comprehensions is fine, as long as that >>> comprehension is contained in an `async def`. While we *could* save `yield >>> [from]` in comprehensions, I still see it as mostly a source of confusion, >>> and the fact that the presence of `yield [from]` *implicitly* makes the >>> surrounding `def` a generator makes things worse. It just requires too many >>> mental contortions to figure out what it does. >>> >> >> [...] >> If the first example will be allowed, then one will be surprised why it >> can't be rewritten as >> >> def pack_two(): >> return [(yield) for _ in range(2)] >> > > And yet Nick's example shows that that is not equivalent! > > [...] > > In this example each thing that looks syntactically like a list > comprehension becomes actually a generator expression at at runtime! And so > does your example, so instead of a list of two items, it returns a > generator that will produce two values when iterated over. > > That's not referential transparency to me, it feels more like a bug in the > code generator. > > I want to ban this because apparently nobody besides Nick knows about this > behavior (I certainly didn't, and from the above it seems you don't either). > This whole thread started as a proposal to fix this bug and to make the two forms equivalent, so I don't know what you are talking about. Also as there appeared arguments of authority (thanks Antoine) its time to stop this discussion for me. -- Ivan ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Sat, Nov 25, 2017 at 8:07 AM, Ivan Levkivskyi wrote: > On 25 November 2017 at 16:57, Guido van Rossum wrote: > >> On Sat, Nov 25, 2017 at 6:55 AM, Ivan Levkivskyi >> wrote: >> >>> On 25 November 2017 at 04:30, Guido van Rossum wrote: >>> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum wrote: > The more I hear about this topic, the more I think that `await`, > `yield` and `yield from` should all be banned from occurring in all > comprehensions and generator expressions. That's not much different from > disallowing `return` or `break`. > From the responses it seems that I tried to simplify things too far. Let's say that `await` in comprehensions is fine, as long as that comprehension is contained in an `async def`. While we *could* save `yield [from]` in comprehensions, I still see it as mostly a source of confusion, and the fact that the presence of `yield [from]` *implicitly* makes the surrounding `def` a generator makes things worse. It just requires too many mental contortions to figure out what it does. >>> >>> [...] >>> If the first example will be allowed, then one will be surprised why it >>> can't be rewritten as >>> >>> def pack_two(): >>> return [(yield) for _ in range(2)] >>> >> >> And yet Nick's example shows that that is not equivalent! >> >> [...] >> >> In this example each thing that looks syntactically like a list >> comprehension becomes actually a generator expression at at runtime! And so >> does your example, so instead of a list of two items, it returns a >> generator that will produce two values when iterated over. >> >> That's not referential transparency to me, it feels more like a bug in >> the code generator. >> >> I want to ban this because apparently nobody besides Nick knows about >> this behavior (I certainly didn't, and from the above it seems you don't >> either). >> > > This whole thread started as a proposal to fix this bug and to make the > two forms equivalent, so I don't know what you are talking about. > I see. I misread your equivalence example as "this how it works" -- you meant it as "this is how I propose we fix it". The fix is not unreasonable but I still would like to retreat to the territory where `yield [from]` in comprehensions (and generator expressions) is deemed invalid. > Also as there appeared arguments of authority (thanks Antoine) its time to > stop this discussion for me. > I'd be happy to stop with the conclusion that we're going to rip out some confusing syntax rather than trying to generate code for it -- IMO we've proved to ourselves that this stuff is too complicated to be useful. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Fri, Nov 24, 2017, 19:32 Guido van Rossum, wrote: > On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum > wrote: > >> The more I hear about this topic, the more I think that `await`, `yield` >> and `yield from` should all be banned from occurring in all comprehensions >> and generator expressions. That's not much different from disallowing >> `return` or `break`. >> > > From the responses it seems that I tried to simplify things too far. Let's > say that `await` in comprehensions is fine, as long as that comprehension > is contained in an `async def`. While we *could* save `yield [from]` in > comprehensions, I still see it as mostly a source of confusion, and the > fact that the presence of `yield [from]` *implicitly* makes the surrounding > `def` a generator makes things worse. It just requires too many mental > contortions to figure out what it does. > > I still propose to rule out all of the above from generator expressions, > because those can escape from the surrounding scope. > +1 from me. -Brett > -- > --Guido van Rossum (python.org/~guido) > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
So we are keeping asynchronous generator expressions as long as they are defined in an 'async def' coroutine? Yury On Sat, Nov 25, 2017 at 12:17 PM Brett Cannon wrote: > > > On Fri, Nov 24, 2017, 19:32 Guido van Rossum, wrote: > >> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum >> wrote: >> >>> The more I hear about this topic, the more I think that `await`, `yield` >>> and `yield from` should all be banned from occurring in all comprehensions >>> and generator expressions. That's not much different from disallowing >>> `return` or `break`. >>> >> >> From the responses it seems that I tried to simplify things too far. >> Let's say that `await` in comprehensions is fine, as long as that >> comprehension is contained in an `async def`. While we *could* save `yield >> [from]` in comprehensions, I still see it as mostly a source of confusion, >> and the fact that the presence of `yield [from]` *implicitly* makes the >> surrounding `def` a generator makes things worse. It just requires too many >> mental contortions to figure out what it does. >> >> I still propose to rule out all of the above from generator expressions, >> because those can escape from the surrounding scope. >> > > +1 from me. > > -Brett > > >> -- >> --Guido van Rossum (python.org/~guido) >> ___ >> Python-Dev mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org >> > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
> > On Sat, Nov 25, 2017 at 12:17 PM Brett Cannon wrote: > On Fri, Nov 24, 2017, 19:32 Guido van Rossum, wrote: >> >>> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum >>> wrote: >>> The more I hear about this topic, the more I think that `await`, `yield` and `yield from` should all be banned from occurring in all comprehensions and generator expressions. That's not much different from disallowing `return` or `break`. >>> >>> From the responses it seems that I tried to simplify things too far. >>> Let's say that `await` in comprehensions is fine, as long as that >>> comprehension is contained in an `async def`. While we *could* save `yield >>> [from]` in comprehensions, I still see it as mostly a source of confusion, >>> and the fact that the presence of `yield [from]` *implicitly* makes the >>> surrounding `def` a generator makes things worse. It just requires too many >>> mental contortions to figure out what it does. >>> >>> I still propose to rule out all of the above from generator expressions, >>> because those can escape from the surrounding scope. >>> >> >> +1 from me >> > On Sat, Nov 25, 2017 at 9:21 AM, Yury Selivanov wrote: > So we are keeping asynchronous generator expressions as long as they are > defined in an 'async def' coroutine? > I would be happy to declare that `await` is out of scope for this thread. It seems that it is always well-defined and sensible what it does in comprehensions and in genexprs. (Although I can't help noticing that PEP 530 does not appear to propose `await` in generator expressions -- it proposes `async for` in comprehensions and in genexprs, and `await` in comprehensions only -- but they appear to be accepted nevertheless.) So we're back to the original issue, which is that `yield` inside a comprehension accidentally makes it become a generator rather than a list, set or dict. I believe that this can be fixed. But I don't believe we should fix it. I believe we should ban `yield` from comprehensions and from genexprs. We don't need it, and it's confused most everyone. And the ban should extend to `yield from` in those same contexts. I think we have a hope for consensus on this. (I also think that if we had invented `await` earlier we wouldn't have gone down the path of `yield` expressions -- but historically it appears we wouldn't have invented `await` at all if we hadn't first tried `yield` and then `yield from` to build coroutines, so I don't think this so bad after all. :-) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
FWIW, on a side point. I use 'yield' and 'yield from' ALL THE TIME in real code. Probably 80% of those would be fine with yield statements, but a significant fraction use `gen.send()`. On the other hand, I have yet once to use 'await', or 'async' outside of pedagogical contexts. There are a whole lot of generators, including ones utilizing state injection, that are useful without the scaffolding of an event loop, in synchronous code. Of course, I never use them in comprehensions or generator expressions. And even after reading every post in this thread, the behavior (either existing or desired by some) such constructs have is murky and difficult for me to reason about. I strongly support deprecation or even just immediate SyntaxError in 3.7. On Nov 25, 2017 12:38 PM, "Guido van Rossum" wrote: > On Sat, Nov 25, 2017 at 12:17 PM Brett Cannon wrote: >> > On Fri, Nov 24, 2017, 19:32 Guido van Rossum, wrote: >>> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum wrote: > The more I hear about this topic, the more I think that `await`, > `yield` and `yield from` should all be banned from occurring in all > comprehensions and generator expressions. That's not much different from > disallowing `return` or `break`. > From the responses it seems that I tried to simplify things too far. Let's say that `await` in comprehensions is fine, as long as that comprehension is contained in an `async def`. While we *could* save `yield [from]` in comprehensions, I still see it as mostly a source of confusion, and the fact that the presence of `yield [from]` *implicitly* makes the surrounding `def` a generator makes things worse. It just requires too many mental contortions to figure out what it does. I still propose to rule out all of the above from generator expressions, because those can escape from the surrounding scope. >>> >>> +1 from me >>> >> > On Sat, Nov 25, 2017 at 9:21 AM, Yury Selivanov > wrote: > >> So we are keeping asynchronous generator expressions as long as they are >> defined in an 'async def' coroutine? >> > > I would be happy to declare that `await` is out of scope for this thread. > It seems that it is always well-defined and sensible what it does in > comprehensions and in genexprs. (Although I can't help noticing that PEP > 530 does not appear to propose `await` in generator expressions -- it > proposes `async for` in comprehensions and in genexprs, and `await` in > comprehensions only -- but they appear to be accepted nevertheless.) > > So we're back to the original issue, which is that `yield` inside a > comprehension accidentally makes it become a generator rather than a list, > set or dict. I believe that this can be fixed. But I don't believe we > should fix it. I believe we should ban `yield` from comprehensions and from > genexprs. We don't need it, and it's confused most everyone. And the ban > should extend to `yield from` in those same contexts. I think we have a > hope for consensus on this. > > (I also think that if we had invented `await` earlier we wouldn't have > gone down the path of `yield` expressions -- but historically it appears we > wouldn't have invented `await` at all if we hadn't first tried `yield` and > then `yield from` to build coroutines, so I don't think this so bad after > all. :-) > > -- > --Guido van Rossum (python.org/~guido) > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > mertz%40gnosis.cx > > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Second post: PEP 557, Data Classes
The updated version should show up at https://www.python.org/dev/peps/pep-0557/ shortly. The major changes from the previous version are: - Add InitVar to specify initialize-only fields. - Renamed __dataclass_post_init__() to __post_init(). - Rename cmp to compare. - Added eq, separate from compare, so you can test unorderable items for equality. - Flushed out asdict() and astuple(). - Changed replace() to just call __init__(), and dropped the complex post-create logic. The only open issues I know of are: - Should object comparison require an exact match on the type? https://github.com/ericvsmith/dataclasses/issues/51 - Should the replace() function be renamed to something else? https://github.com/ericvsmith/dataclasses/issues/77 Most of the items that were previously discussed on python-dev were discussed in detail at https://github.com/ericvsmith/dataclasses. Before rehashing an old discussion, please check there first. Also at https://github.com/ericvsmith/dataclasses is an implementation, with tests, that should work with 3.6 and 3.7. The only action item for the code is to clean up the implementation of InitVar, but that's waiting for PEP 560. Oh, and if PEP 563 is accepted I'll also need to do some work. Feedback is welcomed! Eric. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Sat, Nov 25, 2017 at 3:27 PM, Guido van Rossum wrote: > On Sat, Nov 25, 2017 at 9:21 AM, Yury Selivanov > wrote: >> >> So we are keeping asynchronous generator expressions as long as they are >> defined in an 'async def' coroutine? > > > I would be happy to declare that `await` is out of scope for this thread. It > seems that it is always well-defined and sensible what it does in > comprehensions and in genexprs. (Although I can't help noticing that PEP 530 > does not appear to propose `await` in generator expressions -- it proposes > `async for` in comprehensions and in genexprs, and `await` in comprehensions > only -- but they appear to be accepted nevertheless.) Great! As for PEP 530, after reading this discussion I realized how many things in it are underspecified. I'll be working on PEP 550 successor next week and will also try to update PEP 530 to make it clearer. > So we're back to the original issue, which is that `yield` inside a > comprehension accidentally makes it become a generator rather than a list, > set or dict. I believe that this can be fixed. But I don't believe we should > fix it. I believe we should ban `yield` from comprehensions and from > genexprs. +1 from me. Yury ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Allow tuple unpacking in return and yield statements
On 11/25/2017 1:55 AM, David Cuthbert wrote: First time contributing back -- if I should be filing a PEP or something like that for this, please let me know. I don't think a PEP is needed. Coming from https://bugs.python.org/issue32117, unparenthesized tuple unpacking is allowed in assignments: rest = (4, 5, 6) a = 1, 2, 3, *rest Because except for (), it is ',', not '()' that makes a tuple a tuple. but not in yield or return statements (these result in SyntaxErrors): return 1, 2, 3, *rest yield 1, 2, 3, *rest To be crystal clear, a parenthesized tuple with unpacking *is* valid. return (1, 2, 3, *rest) yield (1, 2, 3, *rest) So is an un-parenthesized tuple without unpacking. Since return and yield are often the first half of a cross-namespace assignment, requiring the () is a bit surprising. Perhaps someone else has a good reason for the difference. Otherwise, +1 on the change. The unpacking in assignments was enabled by a pre-3.2 commit that I haven't yet been able to track back to a discussion, but I suspect this asymmetry is unintentional. Here's the original commit: https://github.com/python/cpython/commit/4905e80c3d2f6abb613d212f0313d1dfe09475dc I've submitted a patch (CLA is signed and submitted, not yet processed), and Serihy said that since it changes the grammar I should have it reviewed here and have signoff by the BDFL. While I haven't had a need for this myself, it was brought up by a user on StackOverflow (https://stackoverflow.com/questions/47272460/python-tuple-unpacking-in-return-statement/47326859). Thanks! Dave -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
Nick Coghlan wrote:
def example():
comp1 = yield from [(yield x) for x in ('1st', '2nd')]
comp2 = yield from [(yield x) for x in ('3rd', '4th')]
return comp1, comp2
If the implicit "yield from" idea seems too magical, then the other
direction we could go is to make the immediate "yield from" mandatory,
Whether it's "too magical" or not depends on your stance with
regard to the implicit function scope of a comprehension. There
seem to be two irreconcilable schools of thought on that:
(1) It's an implementation detail that happens to be used to
stop the loop variable from leaking. Most of the time you can
ignore it and use the nested-loop-expansion mental model.
Most people seem to think this way in practice.
(2) It's an important part of the semantics of comprehensions
that needs to be taken into account at all times. This seems
to be Guido's position.
If you're in school (1), then the current behaviour of yield
in a comprehension is the thing that's weird and magical.
Moreover, it's bad magic, because it does something you
almost certainly don't want. Adding an implicit yield-from
would make things *less* magical and more understandable.
However, it will break things for people in school (2), who
are aware of the arcane details and know enough to put the
yield-from in themselves when needed.
Just because it breaks their code doesn't necessarily mean
they will find it more magical, though. Generators are
already sufficiently magical that they're probably smart
enough to figure out what's going on.
Personally I'm somewhat uncomfortable with having a rule
that "yield" in a comprehension creates a subgenerator,
because the syntactic clues that a given "yield" is in a
comprehension are fairly subtle.
For the same reason, I'm uncomfortable with the nested
function scope of a comprehension being an official part
of the semantics. All other function scopes are introduced
by a very clear piece of syntax, but with comprehensions
you have to notice a combination of things that don't
have anything to do with function scopes by themselves.
So I think I've just talked myself into the opinion that
anything that would allow you to tell whether comprehensions
have an implicit function scope or not should be banned.
--
Greg
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Allow tuple unpacking in return and yield statements
I think the proposal is reasonable and won't require a PEP. On Nov 25, 2017 3:25 PM, "Terry Reedy" wrote: > On 11/25/2017 1:55 AM, David Cuthbert wrote: > >> First time contributing back -- if I should be filing a PEP or something >> like that for this, please let me know. >> > > I don't think a PEP is needed. > > Coming from https://bugs.python.org/issue32117, unparenthesized tuple >> unpacking is allowed in assignments: >> >> rest = (4, 5, 6) >> a = 1, 2, 3, *rest >> > > Because except for (), it is ',', not '()' that makes a tuple a tuple. > > but not in yield or return statements (these result in SyntaxErrors): >> >> return 1, 2, 3, *rest >> yield 1, 2, 3, *rest >> > > To be crystal clear, a parenthesized tuple with unpacking *is* valid. > > return (1, 2, 3, *rest) > yield (1, 2, 3, *rest) > > So is an un-parenthesized tuple without unpacking. > > Since return and yield are often the first half of a cross-namespace > assignment, requiring the () is a bit surprising. Perhaps someone else has > a good reason for the difference. Otherwise, +1 on the change. > > The unpacking in assignments was enabled by a pre-3.2 commit that I >> haven't yet been able to track back to a discussion, but I suspect this >> asymmetry is unintentional. Here's the original commit: >> https://github.com/python/cpython/commit/4905e80c3d2f6abb613 >> d212f0313d1dfe09475dc >> >> I've submitted a patch (CLA is signed and submitted, not yet processed), >> and Serihy said that since it changes the grammar I should have it reviewed >> here and have signoff by the BDFL. >> > > > > > While I haven't had a need for this myself, it was brought up by a user on >> StackOverflow (https://stackoverflow.com/questions/47272460/python-tuple- >> unpacking-in-return-statement/47326859). >> >> Thanks! >> Dave >> >> >> > > -- > Terry Jan Reedy > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido% > 40python.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Sat, Nov 25, 2017 at 1:05 PM, David Mertz wrote: > FWIW, on a side point. I use 'yield' and 'yield from' ALL THE TIME in real > code. Probably 80% of those would be fine with yield statements, but a > significant fraction use `gen.send()`. > > On the other hand, I have yet once to use 'await', or 'async' outside of > pedagogical contexts. There are a whole lot of generators, including ones > utilizing state injection, that are useful without the scaffolding of an > event loop, in synchronous code. > Maybe you didn't realize async/await don't need an event loop? Driving an async/await-based coroutine is just as simple as driving a yield-from-based one (`await` does exactly the same thing as `yield from`). But I won't argue with the usefulness of `yield [from]` in expressions. That is not the topic of this thread. > Of course, I never use them in comprehensions or generator expressions. > And even after reading every post in this thread, the behavior (either > existing or desired by some) such constructs have is murky and difficult > for me to reason about. I strongly support deprecation or even just > immediate SyntaxError in 3.7. > Maybe the rest of the discussion should be about deprecation vs. SyntaxError in Python 3.7. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
Serhiy Storchaka wrote: Ivan explained that this function should be rough equivalent to def f(): t = [(yield i) for i in range(3)] return (x for x in t) This is a *rough* equivalent. There are differences in details. The details would seem to be overwhelmingly important, though. I take it you're saying the semantics should be "like the above except that the returned iterator is lazy". But that seems impossible, because f() can't return anything until it finishes having all its values sent to it. -- Greg ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Sat, Nov 25, 2017 at 3:24 PM, Greg Ewing
wrote:
> Nick Coghlan wrote:
>
> def example():
>> comp1 = yield from [(yield x) for x in ('1st', '2nd')]
>> comp2 = yield from [(yield x) for x in ('3rd', '4th')]
>> return comp1, comp2
>>
>
> If the implicit "yield from" idea seems too magical, then the other
>> direction we could go is to make the immediate "yield from" mandatory,
>>
>
> Whether it's "too magical" or not depends on your stance with
> regard to the implicit function scope of a comprehension. There
> seem to be two irreconcilable schools of thought on that:
>
> (1) It's an implementation detail that happens to be used to
> stop the loop variable from leaking. Most of the time you can
> ignore it and use the nested-loop-expansion mental model.
> Most people seem to think this way in practice.
>
> (2) It's an important part of the semantics of comprehensions
> that needs to be taken into account at all times. This seems
> to be Guido's position.
>
> If you're in school (1), then the current behaviour of yield
> in a comprehension is the thing that's weird and magical.
> Moreover, it's bad magic, because it does something you
> almost certainly don't want. Adding an implicit yield-from
> would make things *less* magical and more understandable.
>
> However, it will break things for people in school (2), who
> are aware of the arcane details and know enough to put the
> yield-from in themselves when needed.
>
> Just because it breaks their code doesn't necessarily mean
> they will find it more magical, though. Generators are
> already sufficiently magical that they're probably smart
> enough to figure out what's going on.
>
> Personally I'm somewhat uncomfortable with having a rule
> that "yield" in a comprehension creates a subgenerator,
> because the syntactic clues that a given "yield" is in a
> comprehension are fairly subtle.
>
> For the same reason, I'm uncomfortable with the nested
> function scope of a comprehension being an official part
> of the semantics. All other function scopes are introduced
> by a very clear piece of syntax, but with comprehensions
> you have to notice a combination of things that don't
> have anything to do with function scopes by themselves.
>
> So I think I've just talked myself into the opinion that
> anything that would allow you to tell whether comprehensions
> have an implicit function scope or not should be banned.
>
That's not an unreasonable position, and one that would leave a door open
to a general design for inner scopes that are not implemented using
functions or code objects (something to ponder for 3.8 perhaps?). In terms
of the current thread the only thing it would disallow seems to be the
current behavior of yield in comprehensions (and perhaps genexprs?).
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On Sat, Nov 25, 2017 at 3:37 PM, Guido van Rossum wrote: > Maybe you didn't realize async/await don't need an event loop? Driving an > async/await-based coroutine is just as simple as driving a yield-from-based > one (`await` does exactly the same thing as `yield from`). > I realize I *can*, but it seems far from straightforward. I guess this is really a python-list question or something, but what is the async/await spelling of something toy like: In [1]: def fib(): ...: a, b = 1, 1 ...: while True: ...: yield a ...: a, b = b, a+b ...: In [2]: from itertools import takewhile In [3]: list(takewhile(lambda x: x<200, fib())) Out[3]: [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144] > Maybe the rest of the discussion should be about deprecation vs. > SyntaxError in Python 3.7. > I vote SyntaxError, of course. :-) -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second post: PEP 557, Data Classes
One more change: - Per-field metadata, for use by third parties. Also, thanks to Guido and Ivan for all of their feedback on the various issues that got the PEP to this point. Eric. On 11/25/2017 4:06 PM, Eric V. Smith wrote: The updated version should show up at https://www.python.org/dev/peps/pep-0557/ shortly. The major changes from the previous version are: - Add InitVar to specify initialize-only fields. - Renamed __dataclass_post_init__() to __post_init(). - Rename cmp to compare. - Added eq, separate from compare, so you can test unorderable items for equality. - Flushed out asdict() and astuple(). - Changed replace() to just call __init__(), and dropped the complex post-create logic. The only open issues I know of are: - Should object comparison require an exact match on the type? https://github.com/ericvsmith/dataclasses/issues/51 - Should the replace() function be renamed to something else? https://github.com/ericvsmith/dataclasses/issues/77 Most of the items that were previously discussed on python-dev were discussed in detail at https://github.com/ericvsmith/dataclasses. Before rehashing an old discussion, please check there first. Also at https://github.com/ericvsmith/dataclasses is an implementation, with tests, that should work with 3.6 and 3.7. The only action item for the code is to clean up the implementation of InitVar, but that's waiting for PEP 560. Oh, and if PEP 563 is accepted I'll also need to do some work. Feedback is welcomed! Eric. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On 26 November 2017 at 02:59, Guido van Rossum wrote: > > I'd be happy to stop with the conclusion that we're going to rip out some > confusing syntax rather than trying to generate code for it -- IMO we've > proved to ourselves that this stuff is too complicated to be useful. I'll also note that even if we go through a period of deprecating and then prohibiting the syntax entirely, we'll still have the option of bringing support for "yield" in comprehensions back later with deliberately designed semantics (as happened for "await" in https://www.python.org/dev/peps/pep-0530/#await-in-comprehensions), as opposed to the accident-of-implementation semantics they have now. It may also turn out that as more asynchronous code is able to switch to being 3.6+ only, allowing "await" and prohibiting "yield" will prove to be sufficient for all practical purposes (as even the "yield from" based spelling is Python-3-only, so it's only code that still has to support 3.3, 3.4, 3.5, without needing to support 2.7, that could use "yield from + yield" comprehensions, but wouldn't have the option of just switching to async+await instead). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Allow tuple unpacking in return and yield statements
On 26 November 2017 at 09:22, Terry Reedy wrote: > Since return and yield are often the first half of a cross-namespace > assignment, requiring the () is a bit surprising. Perhaps someone else has > a good reason for the difference. These kinds of discrepancies tend to arise because there are a few different grammar nodes for "comma separated sequence of expressions", which makes it possible to miss some when enhancing the tuple syntax. Refactoring the grammar to eliminate the duplication isn't especially easy, and we don't change the syntax all that often, so it makes sense to treat cases like this one as bugs in the implementation of the original syntax change (except that the "don't change the Grammar in maintenance releases" guideline means they still need to be handled as new features when it comes to fixing them). Cheers, Nick. P.S. That said, I do wonder if it might be feasible to write a "Grammar consistency check" test that ensured the known duplicate nodes at least have consistent definitions, such that missing one in a syntax update will cause an automated test failure. Unfortunately, the nodes typically haven't been combined because they have some *intentional* differences in exactly what they allow, so I also suspect that this is easier said than done. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Second post: PEP 557, Data Classes
Hi Eric, Really excited about this PEP, thanks for working on it. A couple minor questions: > If compare is True, then eq is ignored, and __eq__ and __ne__ will be automatically generated. IMO it's generally preferable to make nonsensical parameter combinations an immediate error, rather than silently ignore one of them. Is there a strong reason for letting nonsense pass silently here? (I reviewed the previous thread; there was a lot of discussion about enums/flags vs two boolean params, but I didn't see explicit discussion of this issue; the only passing references I noticed said the invalid combo should be "disallowed", e.g. Guido in [1], which to me implies "an error.") > isdataclass(instance): Returns True if instance is an instance of a Data Class, otherwise returns False. Something smells wrong with the naming here. If I have @dataclass class Person: name: str I think it would be considered obvious and undeniable (in English prose, anyway) that Person is a dataclass. So it seems wrong to have `isdataclass(Person)` return `False`. Is there a reason not to let it handle either a class or an instance (looks like it would actually simplify the implementation)? Carl [1] https://mail.python.org/pipermail/python-dev/2017-September/149505.html On 11/25/2017 01:06 PM, Eric V. Smith wrote: > The updated version should show up at > https://www.python.org/dev/peps/pep-0557/ shortly. > > The major changes from the previous version are: > > - Add InitVar to specify initialize-only fields. > - Renamed __dataclass_post_init__() to __post_init(). > - Rename cmp to compare. > - Added eq, separate from compare, so you can test > unorderable items for equality. > - Flushed out asdict() and astuple(). > - Changed replace() to just call __init__(), and dropped > the complex post-create logic. > > The only open issues I know of are: > - Should object comparison require an exact match on the type? > https://github.com/ericvsmith/dataclasses/issues/51 > - Should the replace() function be renamed to something else? > https://github.com/ericvsmith/dataclasses/issues/77 > > Most of the items that were previously discussed on python-dev were > discussed in detail at https://github.com/ericvsmith/dataclasses. Before > rehashing an old discussion, please check there first. > > Also at https://github.com/ericvsmith/dataclasses is an implementation, > with tests, that should work with 3.6 and 3.7. The only action item for > the code is to clean up the implementation of InitVar, but that's > waiting for PEP 560. Oh, and if PEP 563 is accepted I'll also need to do > some work. > > Feedback is welcomed! > > Eric. > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/carl%40oddbird.net signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tricky way of of creating a generator via a comprehension expression
On 11/25/2017 04:20 PM, David Mertz wrote: On Sat, Nov 25, 2017 at 3:37 PM, Guido van Rossum wrote: Maybe you didn't realize async/await don't need an event loop? Driving an async/await-based coroutine is just as simple as driving a yield-from-based one (`await` does exactly the same thing as `yield from`). I realize I *can*, but it seems far from straightforward. I guess this is really a python-list question or something, but what is the async/await spelling of something toy like: In [1]: def fib(): ...: a, b = 1, 1 ...: while True: ...: yield a ...: a, b = b, a+b ...: In [2]: from itertools import takewhile In [3]: list(takewhile(lambda x: x<200, fib())) Out[3]: [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144] Maybe the rest of the discussion should be about deprecation vs. SyntaxError in Python 3.7. I vote SyntaxError, of course. :-) Given the recent thread about the difficulty of noticing DeprecationWarnings, I also vote SyntaxError. On the other hand, if we have a change in 3.7 about the visibility of DeprecationWarnings, this would make an excellent test case. -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
