Re: [Python-Dev] PEP 479 (Change StopIteration handling inside generators) -- hopefully final text
On 6 December 2014 at 04:42, Guido van Rossum gu...@python.org wrote: For those who haven't followed along, here's the final text of PEP 479, with a brief Acceptance section added. The basic plan hasn't changed, but there's a lot more clarifying text and discussion of a few counter-proposals. Please send suggestions for editorial improvements to p...@python.org. The official reference version of the PEP is at https://www.python.org/dev/peps/pep-0479/; the repo is https://hg.python.org/peps/ (please check out the repo and send diffs relative to the repo if you have edits). Thanks Guido, that explanation of the change looks great to me. And thanks also to Chris and everyone else that helped with the rather involved discussions! Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 27 November 2014 at 09:50, Guido van Rossum gu...@python.org wrote: On Wed, Nov 26, 2014 at 3:15 PM, Nick Coghlan ncogh...@gmail.com wrote: This is actually the second iteration of this bug: the original implementation *always* suppressed StopIteration. PJE caught that one before Python 2.5 was released, but we didn't notice that 3.3 had brought it back in a new, more subtle form :( It's worth noting that my allow_implicit_stop idea in the other thread wouldn't affect subgenerators - those would still convert StopIteration to RuntimeError unless explicitly silenced. You've lost me in this subthread. Am I right to conclude that the PEP change doesn't cause problems for contextlib(*), but that the PEP change also probably wouldn't have helped diagnose any contextlib bugs? I think the PEP 479 semantics would have made both bugs (the one PJE found in 2.5, and the newer one Isaac pointed out here) less cryptic, in that they would have caused RuntimeError to be raised, rather than silently consuming the StopIteration and continuing execution after the with statement body. With the new semantics, contextlib just needs to be updated to cope with the StopIteration - RuntimeError conversion, and Isaac's spurious success bug will be fixed*. Without PEP 479, I believe my only recourse to systematically eliminate the risk of generator based context managers silently consuming StopIteration would be to implement the StopIteration - gen.close() workaround, and that would be a backwards incompatible change in its own right. Cheers, Nick. P.S. *(This does mean I was wrong about allow_implicit_stop being useful to contextlib, but I still think the decorator is useful for cases where StopIteration is being used to terminate the generator on purpose) -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/26/14, Nick Coghlan wrote: On 26 November 2014 at 04:04, Guido van Rossum gu...@python.org wrote: On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico ros...@gmail.com wrote: On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher ischwabac...@wisc.edu wrote: Yield can also raise StopIteration, if it's thrown in. The current interaction of generator.throw(StopIteration) with yield from can't be emulated under the PEP's behavior, though it's not clear that that's a problem. Hrm. I have *absolutely* no idea when you would use that, and how you'd go about reworking it to fit this proposal. Do you have any example code (production or synthetic) which throws StopIteration into a generator? Sounds like a good one for the obfuscated Python contest. :-) Unless the generator has a try/except surrounding the yield point into which the exception is thrown, it will bubble right out, and PEP 479 will turn this into a RuntimeError. I'll clarify this in the PEP (even though it logically follows from the proposal) -- I don't think there's anything to worry about. This is actually the edge case that needs to be handled in contextlib - a StopIteration raised by the with statement body gets thrown into the generator implementing the context manager. My current porting recommendation is to catch the RuntimeError look at __cause__ to see if it's the StopIteration instance that was thrown in, but an alternative option would be to just call gen.close() in that case, rather than gen.throw(exc). If this is the current contextlib implementation, does it break if the yield statement is replaced with yield from another context manager generator? ijs ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/26/14, Nick Coghlan wrote:
On 26 November 2014 at 04:04, Guido van Rossum gu...@python.org wrote:
On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico ros...@gmail.com wrote:
On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher
ischwabac...@wisc.edu wrote:
Yield can also raise StopIteration, if it's thrown in. The current
interaction of generator.throw(StopIteration) with yield from can't be
emulated under the PEP's behavior, though it's not clear that that's a
problem.
Hrm. I have *absolutely* no idea when you would use that, and how
you'd go about reworking it to fit this proposal. Do you have any
example code (production or synthetic) which throws StopIteration into
a generator?
Sounds like a good one for the obfuscated Python contest. :-)
Unless the generator has a try/except surrounding the yield point into which
the exception is thrown, it will bubble right out, and PEP 479 will turn
this into a RuntimeError. I'll clarify this in the PEP (even though it
logically follows from the proposal) -- I don't think there's anything to
worry about.
This is actually the edge case that needs to be handled in contextlib
- a StopIteration raised by the with statement body gets thrown into
the generator implementing the context manager. My current porting
recommendation is to catch the RuntimeError look at __cause__ to see
if it's the StopIteration instance that was thrown in, but an
alternative option would be to just call gen.close() in that case,
rather than gen.throw(exc).
This actually leads to a good example of why the PEP is necessary:
```
In [1]: import contextlib
In [2]: @contextlib.contextmanager
...: def transact():
...: print('setup transaction')
...: try:
...: yield from subgenerator()
...: except:
...: print('rollback transaction')
...: raise
...: else:
...: print('commit transaction')
...: finally:
...: print('clean up transaction')
...:
In [3]: def subgenerator():
...: print('setup subgenerator')
...: try:
...: yield
...: except:
...: print('subgenerator failed')
...: raise
...: else:
...: print('subgenerator succeeded')
...: finally:
...: print('clean up subgenerator')
...:
In [4]: with transact():
...: next(iter([]))
...:
setup transaction
setup subgenerator
subgenerator failed
clean up subgenerator
commit transaction # BAD NOT GOOD BIG FAIL
clean up transaction
```
ijs
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 26 November 2014 at 16:24, Isaac Schwabacher ischwabac...@wisc.edu wrote: This actually leads to a good example of why the PEP is necessary: [...] Oh! If that's the current behaviour, then it probably needs to go into the PEP as a motivating example. It's far more convincing than most of the other arguments I've seen. Just one proviso - is it fixable in contextlib *without* a language change? If so, then it loses a lot of its value. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Wed, Nov 26, 2014 at 8:54 AM, Paul Moore p.f.mo...@gmail.com wrote: On 26 November 2014 at 16:24, Isaac Schwabacher ischwabac...@wisc.edu wrote: This actually leads to a good example of why the PEP is necessary: [...] Oh! If that's the current behaviour, then it probably needs to go into the PEP as a motivating example. It's far more convincing than most of the other arguments I've seen. Just one proviso - is it fixable in contextlib *without* a language change? If so, then it loses a lot of its value. It's hard to use as an example because the behavior of contextlib is an integral part of it -- currently for me the example boils down to there is a bug in contextlib. Maybe it would have been caught earlier with the change in the PEP, but when using it as a motivating example you have to show the code containing the bug, not just a demonstration. If you want to try though, I'm happy to entertain a pull request for the PEP. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 26 November 2014 at 17:19, Guido van Rossum gu...@python.org wrote: It's hard to use as an example because the behavior of contextlib is an integral part of it -- currently for me the example boils down to there is a bug in contextlib Hmm, fair point. I was assuming that the bug in contextlib can't be fixed with the current language behaviour (and I'd personally be OK with the example simply adding a comment this can't be fixed without changing Python as proposed in the PEP). But I'm not sure how true that is, so maybe it's not quite as compelling as it seemed to me at first. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/26/2014 08:54 AM, Paul Moore wrote: On 26 November 2014 at 16:24, Isaac Schwabacher ischwabac...@wisc.edu wrote: This actually leads to a good example of why the PEP is necessary: [...] Oh! If that's the current behaviour, then it probably needs to go into the PEP as a motivating example. It's far more convincing than most of the other arguments I've seen. Just one proviso - is it fixable in contextlib *without* a language change? If so, then it loses a lot of its value. No value is lost. The PEP exists because this mistake is so easy to make, and can be so hard to track down. -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
Can you summarize that in a self-contained form for inclusion in the PEP? (That was a rhetorical question. :-) On Wed, Nov 26, 2014 at 12:17 PM, Isaac Schwabacher ischwabac...@wisc.edu wrote: On 14-11-26, Guido van Rossum wrote: On Wed, Nov 26, 2014 at 8:54 AM, Paul Moore wrote: On 26 November 2014 at 16:24, Isaac Schwabacher wrote: This actually leads to a good example of why the PEP is necessary: [...] Oh! If that's the current behaviour, then it probably needs to go into the PEP as a motivating example. It's far more convincing than most of the other arguments I've seen. Just one proviso - is it fixable in contextlib *without* a language change? If so, then it loses a lot of its value. It's hard to use as an example because the behavior of contextlib is an integral part of it -- currently for me the example boils down to there is a bug in contextlib. Maybe it would have been caught earlier with the change in the PEP, but when using it as a motivating example you have to show the code containing the bug, not just a demonstration. How is this a bug in contextlib? The example behaves the way it does because gen.throw(StopIteration) behaves differently depending on whether gen is paused at a yield or a yield from. What *should* contextlib.contextmanager do in this instance? It has faithfully forwarded the StopIteration raised in the protected block to the generator, and the generator has forwarded this to the subgenerator, which has elected to fail and report success. The bug is in the subgenerator, because it fails to treat StopIteration as an error. But the subgenerator can't in general be converted to treat StopIteration as an error, because clearly it's used in other places than as a nested context manager (otherwise, it would itself be decorated with @contextlib.contextmanager and accessed as such, instead of yielded from). And in those places, perhaps it needs to simply allow StopIteration to bubble up. And can we factor out the error checking so that we don't have to duplicate subgenerator? Well... yes, but it's tricky because we'll introduce an extra yield from in the process, so we have to put the handling in the subgenerator itself and wrap the *non*-context-manager uses. ijs If you want to try though, I'm happy to entertain a pull request for the PEP. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 14-11-26, Guido van Rossum wrote: On Wed, Nov 26, 2014 at 8:54 AM, Paul Moore wrote: On 26 November 2014 at 16:24, Isaac Schwabacher wrote: This actually leads to a good example of why the PEP is necessary: [...] Oh! If that's the current behaviour, then it probably needs to go into the PEP as a motivating example. It's far more convincing than most of the other arguments I've seen. Just one proviso - is it fixable in contextlib *without* a language change? If so, then it loses a lot of its value. It's hard to use as an example because the behavior of contextlib is an integral part of it -- currently for me the example boils down to there is a bug in contextlib. Maybe it would have been caught earlier with the change in the PEP, but when using it as a motivating example you have to show the code containing the bug, not just a demonstration. How is this a bug in contextlib? The example behaves the way it does because gen.throw(StopIteration) behaves differently depending on whether gen is paused at a yield or a yield from. What *should* contextlib.contextmanager do in this instance? It has faithfully forwarded the StopIteration raised in the protected block to the generator, and the generator has forwarded this to the subgenerator, which has elected to fail and report success. The bug is in the subgenerator, because it fails to treat StopIteration as an error. But the subgenerator can't in general be converted to treat StopIteration as an error, because clearly it's used in other places than as a nested context manager (otherwise, it would itself be decorated with @contextlib.contextmanager and accessed as such, instead of yielded from). And in those places, perhaps it needs to simply allow StopIteration to bubble up. And can we factor out the error checking so that we don't have to duplicate subgenerator? Well... yes, b ut it's tricky because we'll introduce an extra yield from in the process, so we have to put the handling in the subgenerator itself and wrap the *non*-context-manager uses. ijs If you want to try though, I'm happy to entertain a pull request for the PEP. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 27 Nov 2014 03:58, Paul Moore p.f.mo...@gmail.com wrote: On 26 November 2014 at 17:19, Guido van Rossum gu...@python.org wrote: It's hard to use as an example because the behavior of contextlib is an integral part of it -- currently for me the example boils down to there is a bug in contextlib Hmm, fair point. I was assuming that the bug in contextlib can't be fixed with the current language behaviour (and I'd personally be OK with the example simply adding a comment this can't be fixed without changing Python as proposed in the PEP). But I'm not sure how true that is, so maybe it's not quite as compelling as it seemed to me at first. The contextlib only change would be to map StopIteration in the body of the with statement to gen.close() on the underlying generator rather than gen.throw(StopIteration). (That's backwards incompatible in its own way, since it means you *can't* suppress StopIteration via a generator based context manager any more) This is actually the second iteration of this bug: the original implementation *always* suppressed StopIteration. PJE caught that one before Python 2.5 was released, but we didn't notice that 3.3 had brought it back in a new, more subtle form :( It's worth noting that my allow_implicit_stop idea in the other thread wouldn't affect subgenerators - those would still convert StopIteration to RuntimeError unless explicitly silenced. Regards, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
You can use the README here: https://github.com/Rosuav/GenStopIter On Wed, Nov 26, 2014 at 1:57 PM, Isaac Schwabacher ischwabac...@wisc.edu wrote: Can you summarize that in a self-contained form for inclusion in the PEP? (That was a rhetorical question. :-) Sure. Is it on GitHub? ;D ijs -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Wed, Nov 26, 2014 at 3:15 PM, Nick Coghlan ncogh...@gmail.com wrote: On 27 Nov 2014 03:58, Paul Moore p.f.mo...@gmail.com wrote: On 26 November 2014 at 17:19, Guido van Rossum gu...@python.org wrote: It's hard to use as an example because the behavior of contextlib is an integral part of it -- currently for me the example boils down to there is a bug in contextlib Hmm, fair point. I was assuming that the bug in contextlib can't be fixed with the current language behaviour (and I'd personally be OK with the example simply adding a comment this can't be fixed without changing Python as proposed in the PEP). But I'm not sure how true that is, so maybe it's not quite as compelling as it seemed to me at first. The contextlib only change would be to map StopIteration in the body of the with statement to gen.close() on the underlying generator rather than gen.throw(StopIteration). (That's backwards incompatible in its own way, since it means you *can't* suppress StopIteration via a generator based context manager any more) This is actually the second iteration of this bug: the original implementation *always* suppressed StopIteration. PJE caught that one before Python 2.5 was released, but we didn't notice that 3.3 had brought it back in a new, more subtle form :( It's worth noting that my allow_implicit_stop idea in the other thread wouldn't affect subgenerators - those would still convert StopIteration to RuntimeError unless explicitly silenced. You've lost me in this subthread. Am I right to conclude that the PEP change doesn't cause problems for contextlib(*), but that the PEP change also probably wouldn't have helped diagnose any contextlib bugs? (*) Except perhaps that some old 3rd party copy of contextlib may eventually break if it's not updated. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
Can you summarize that in a self-contained form for inclusion in the PEP? (That was a rhetorical question. :-) Sure. Is it on GitHub? ;D ijs ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Thu, Nov 27, 2014 at 8:57 AM, Isaac Schwabacher ischwabac...@wisc.edu wrote: Can you summarize that in a self-contained form for inclusion in the PEP? (That was a rhetorical question. :-) Sure. Is it on GitHub? ;D Thanks Isaac, I've incorporated your edits. https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Mon, Nov 24, 2014 at 10:22:54AM +1100, Chris Angelico wrote: My point is that doing the same errant operation on a list or a dict will give different exceptions. In the same way, calling next() on an empty iterator will raise StopIteration normally, but might raise RuntimeError instead. It's still an exception, it still indicates a place where code needs to be changed I wouldn't interpret it like that. Calling next() on an empty iterator raises StopIteration. That's not a bug indicating a failure, it's the protocol working as expected. Your response to that may be to catch the StopIteration and ignore it, or to allow it to bubble up for something else to deal with it. Either way, next() raising StopIteration is not a bug, it is normal behaviour. (Failure to deal with any such StopIteration may be a bug.) However, if next() raises RuntimeError, that's not part of the protocol for iterators, so it is almost certainly a bug to be fixed. (Probably coming from an explicit raise StopIteration inside a generator function.) Your fix for the bug may be to refuse to fix it and just catch the exception and ignore it, but that's kind of nasty and hackish and shouldn't be considered good code. Do you agree this is a reasonable way to look at it? -- Steven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Wed, Nov 26, 2014 at 2:20 AM, Steven D'Aprano st...@pearwood.info wrote: I wouldn't interpret it like that. Calling next() on an empty iterator raises StopIteration. That's not a bug indicating a failure, it's the protocol working as expected. Your response to that may be to catch the StopIteration and ignore it, or to allow it to bubble up for something else to deal with it. Either way, next() raising StopIteration is not a bug, it is normal behaviour. (Failure to deal with any such StopIteration may be a bug.) However, if next() raises RuntimeError, that's not part of the protocol for iterators, so it is almost certainly a bug to be fixed. (Probably coming from an explicit raise StopIteration inside a generator function.) Your fix for the bug may be to refuse to fix it and just catch the exception and ignore it, but that's kind of nasty and hackish and shouldn't be considered good code. Do you agree this is a reasonable way to look at it? Yes. Specifically, your parenthesis in the middle is the important bit. If you have a csv.DictReader, KeyError might be an important part of your protocol (maybe you have an optional column in the CSV file), but it should be caught before it crosses the boundary of part of your protocol. At some point, it needs to be converted into ValueError, perhaps, or replaced with a default value, or some other coping mechanism is used. Failure to deal with StopIteration when calling next() is failure to cope with all of that function's protocol, and that is most likely to be a bug. (There are times, and some of them have been mentioned in these discussion threads, where calling next() can never raise StopIteration, so there need be no try/except - eg it=iter(string.split( )) - but that just means that a StopIteration from that call is an error somewhere else. I'm definitely happy for that kind of shouldn't happen to turn into a RuntimeError rather than being left as an unexpectedly-short generator.) ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher ischwabac...@wisc.edu wrote: Yield can also raise StopIteration, if it's thrown in. The current interaction of generator.throw(StopIteration) with yield from can't be emulated under the PEP's behavior, though it's not clear that that's a problem. Hrm. I have *absolutely* no idea when you would use that, and how you'd go about reworking it to fit this proposal. Do you have any example code (production or synthetic) which throws StopIteration into a generator? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico ros...@gmail.com wrote: On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher ischwabac...@wisc.edu wrote: Yield can also raise StopIteration, if it's thrown in. The current interaction of generator.throw(StopIteration) with yield from can't be emulated under the PEP's behavior, though it's not clear that that's a problem. Hrm. I have *absolutely* no idea when you would use that, and how you'd go about reworking it to fit this proposal. Do you have any example code (production or synthetic) which throws StopIteration into a generator? Sounds like a good one for the obfuscated Python contest. :-) Unless the generator has a try/except surrounding the yield point into which the exception is thrown, it will bubble right out, and PEP 479 will turn this into a RuntimeError. I'll clarify this in the PEP (even though it logically follows from the proposal) -- I don't think there's anything to worry about. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/25/14, Guido van Rossum wrote:
On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico ischwabac...@wisc.edu
ros...@gmail.com') target=1ros...@gmail.com wrote:
On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher
python.org/~guido(javascript:main.compose('new',
't=ischwabac...@wisc.edu wrote:
Yield can also raise StopIteration, if its thrown in. The current
interaction of generator.throw(StopIteration) with yield from cant be
emulated under the PEPs behavior, though its not clear that thats a
problem.
Hrm. I have *absolutely* no idea when you would use that,
To close the innermost generator in a yield-from chain. No, I don't know why
you'd want to do that, either.
and how
you'd go about reworking it to fit this proposal. Do you have any
example code (production or synthetic) which throws StopIteration into
a generator?
No.
Sounds like a good one for the obfuscated Python contest. :-)
I'm just playing with my food now. :)
Unless the generator has a try/except surrounding the yield point into which
the exception is thrown, it will bubble right out, and PEP 479 will turn this
into a RuntimeError. I'll clarify this in the PEP (even though it logically
follows from the proposal) -- I don't think there's anything to worry about.
--
--Guido van Rossum (a href=))
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Tue, Nov 25, 2014 at 10:12 AM, Isaac Schwabacher ischwabac...@wisc.edu
wrote:
On 11/25/14, Guido van Rossum wrote:
On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico ischwabac...@wisc.edu
ros...@gmail.com') target=1ros...@gmail.com wrote:
On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher
python.org/~guido(javascript:main.compose('new', 't=
ischwabac...@wisc.edu wrote:
Yield can also raise StopIteration, if its thrown in. The current
interaction of generator.throw(StopIteration) with yield from cant be
emulated under the PEPs behavior, though its not clear that thats a problem.
Hrm. I have *absolutely* no idea when you would use that,
To close the innermost generator in a yield-from chain. No, I don't know
why you'd want to do that, either.
For that purpose you should call the generator's close() method. This
throws a GeneratorExit into the generator to give the generator a chance of
cleanup (typically using try/finally). Various reasonable things happen if
the generator misbehaves at this point -- if you want to learn what, read
the code or experiment a bit on the command line (that's what I usually do).
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/25/14, Chris Angelico wrote: On Wed, Nov 26, 2014 at 2:20 AM, Steven D'Aprano st...@pearwood.info wrote: I wouldn't interpret it like that. Calling next() on an empty iterator raises StopIteration. That's not a bug indicating a failure, it's the protocol working as expected. Your response to that may be to catch the StopIteration and ignore it, or to allow it to bubble up for something else to deal with it. Either way, next() raising StopIteration is not a bug, it is normal behaviour. (Failure to deal with any such StopIteration may be a bug.) However, if next() raises RuntimeError, that's not part of the protocol for iterators, so it is almost certainly a bug to be fixed. (Probably coming from an explicit raise StopIteration inside a generator function.) Your fix for the bug may be to refuse to fix it and just catch the exception and ignore it, but that's kind of nasty and hackish and shouldn't be considered good code. Do you agree this is a reasonable way to look at it? Yes. Specifically, your parenthesis in the middle is the important bit. If you have a csv.DictReader, KeyError might be an important part of your protocol (maybe you have an optional column in the CSV file), but it should be caught before it crosses the boundary of part of your protocol. At some point, it needs to be converted into ValueError, perhaps, or replaced with a default value, or some other coping mechanism is used. Failure to deal with StopIteration when calling next() is failure to cope with all of that function's protocol, and that is most likely to be a bug. (There are times, and some of them have been mentioned in these discussion threads, where calling next() can never raise StopIteration, so there need be no try/except - eg it=iter(string.split( )) - but that just means that a StopIteration from that call is an error somewhere else. I'm definitely happy for that kind of shouldn't happen to turn into a RuntimeError rather than being left as an unexpectedly-short generator.) Yield can also raise StopIteration, if it's thrown in. The current interaction of generator.throw(StopIteration) with yield from can't be emulated under the PEP's behavior, though it's not clear that that's a problem. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 26 November 2014 at 04:04, Guido van Rossum gu...@python.org wrote: On Tue, Nov 25, 2014 at 9:49 AM, Chris Angelico ros...@gmail.com wrote: On Wed, Nov 26, 2014 at 4:45 AM, Isaac Schwabacher ischwabac...@wisc.edu wrote: Yield can also raise StopIteration, if it's thrown in. The current interaction of generator.throw(StopIteration) with yield from can't be emulated under the PEP's behavior, though it's not clear that that's a problem. Hrm. I have *absolutely* no idea when you would use that, and how you'd go about reworking it to fit this proposal. Do you have any example code (production or synthetic) which throws StopIteration into a generator? Sounds like a good one for the obfuscated Python contest. :-) Unless the generator has a try/except surrounding the yield point into which the exception is thrown, it will bubble right out, and PEP 479 will turn this into a RuntimeError. I'll clarify this in the PEP (even though it logically follows from the proposal) -- I don't think there's anything to worry about. This is actually the edge case that needs to be handled in contextlib - a StopIteration raised by the with statement body gets thrown into the generator implementing the context manager. My current porting recommendation is to catch the RuntimeError look at __cause__ to see if it's the StopIteration instance that was thrown in, but an alternative option would be to just call gen.close() in that case, rather than gen.throw(exc). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Wed, Nov 19, 2014 at 3:10 PM, Guido van Rossum gu...@python.org wrote: There's a new PEP proposing to change how to treat StopIteration bubbling up out of a generator frame (not caused by a return from the frame). The proposal is to replace such a StopIteration with a RuntimeError (chained to the original StopIteration), so that only *returning* from a generator (or falling off the end) causes the iteration to terminate. I think the PEP should also specify what will happen if the generator's __next__() method is called again after RuntimeError is handled. The two choices are: 1. Raise StopIteration (current behavior for all exceptions). 2. Raise RuntimeError (may be impossible without gi_frame). I think choice 1 is implied by the PEP. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Mon, Nov 24, 2014 at 4:21 PM, Alexander Belopolsky alexander.belopol...@gmail.com wrote: On Wed, Nov 19, 2014 at 3:10 PM, Guido van Rossum gu...@python.org wrote: There's a new PEP proposing to change how to treat StopIteration bubbling up out of a generator frame (not caused by a return from the frame). The proposal is to replace such a StopIteration with a RuntimeError (chained to the original StopIteration), so that only *returning* from a generator (or falling off the end) causes the iteration to terminate. I think the PEP should also specify what will happen if the generator's __next__() method is called again after RuntimeError is handled. The two choices are: 1. Raise StopIteration (current behavior for all exceptions). 2. Raise RuntimeError (may be impossible without gi_frame). I think choice 1 is implied by the PEP. Good catch. It has to be #1 because the generator object doesn't retain exception state. I am behind with updating the PEP but I promise I won't mark it as Accepted without adding this, the transition plan, and a discussion of some of the objections that were raised. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 5:23 PM, Chris Angelico wrote: On Sun, Nov 23, 2014 at 8:03 AM, Ron Adam ron3...@gmail.com wrote: Making comprehensions work more like generator expressions would, IMO, imply making the same change to all for loops: having a StopIteration raised by the body of the loop quietly terminate the loop. I'm not suggesting making any changes to generator expressions or for loops at all. They would continue to work like they currently do. But if you're suggesting making list comps react to StopIteration raised by their primary expressions, then to maintain the correspondence between a list comp and its expanded form, for loops would have to change too. Or should that correspondence be broken, in that single-expression loop constructs become semantically different from statement loop constructs? The 2.x correspondence between list comp and for loops was *already broken* in 3.0 by moving execution to a new function to prevent name binding in comprehension from leaking the way they did in 2.x. We did not change for loops to match. The following, which is an example of equivalence in 2.7, raises in 3.4 because 'itertools' is not defined. def binder(ns, name, ob): ns[name] = ob for mod in ['sys']: binder(locals(), mod, __import__(mod)) print(sys) [binder(locals(), mod, __import__(mod)) for mod in ['itertools']] print(itertools) Ceasing to leak *any* local bindings in 3.0 broke equivalence and code depending on such bindings. Ceasing to leak StopIteration would break equivalence a bit more, but only a bit, and code dependent on such leakage, which I expect is extremely rare. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 08:53 PM, Guido van Rossum wrote: In order to save everyone's breath, I am *accepting* the proposal of PEP 479. Excellent. Chris, thank you for your time, effort, and thoroughness in championing this PEP. -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Mon, Nov 24, 2014 at 2:11 AM, Ethan Furman et...@stoneleaf.us wrote: On 11/22/2014 08:53 PM, Guido van Rossum wrote: In order to save everyone's breath, I am *accepting* the proposal of PEP 479. Excellent. Chris, thank you for your time, effort, and thoroughness in championing this PEP. Thank you, it's nice to have a successful one to counterbalance the failure of PEP 463. (Which, incidentally, never actually got a resolution. It's still showing up as 'Draft' status.) I'm actually quite impressed with how on-topic the discussion threads went. They didn't ramble nearly as much as I thought they would. Thank you to all who participated, contributed suggestions, complained about the existing text, and generally worked hard to make sure I had more than enough material to draw on! Particular thanks to Guido, pushing changes through and being patient with my mistakes in markup, and adding content directly to the document. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 23 November 2014 at 15:25, Chris Angelico ros...@gmail.com wrote: Thank you, it's nice to have a successful one to counterbalance the failure of PEP 463. (Which, incidentally, never actually got a resolution. It's still showing up as 'Draft' status.) I think it's worth pointing out that both this and PEP 463 were complex an contentious topics, and the discussion in both cases was well-focused and civil. I don't think the fact that you were the author of both PEPs is unrelated to this. Thanks for taking on *both* of these PEPs and handling them so well. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 12:30 PM, Raymond Hettinger wrote: Pre-PEP 479: --- def middleware_generator(source_generator): it = source_generator() input_value = next(it) output_value = do_something_interesting(input_value) yield output_value Post-PEP 479: def middleware_generator(source_generator): it = source_generator() try: input_value = next(it) except StopIteration: return # This causes StopIteration to be reraised output_value = do_something_interesting(input_value) yield output_value While I am in favor of PEP 479, and I have to agree with Raymond that this isn't pretty. Currently, next() accepts an argument of what to return if the iterator is empty. Can we enhance that in some way so that the overall previous behavior could be retained? Something like: def middleware_generator(source_generator): it = source_generator() input_value = next(it, gen_exit=True) # or exc_type=GeneratorExit ? output_value = do_something_interesting(input_value) yield output_value Then, if the iterator is empty, instead of raising StopIteration, or returning some value that would then have to be checked, it could raise some other exception that is understood to be normal generator termination. -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/23/2014 04:08 AM, Terry Reedy wrote: On 11/22/2014 5:23 PM, Chris Angelico wrote: On Sun, Nov 23, 2014 at 8:03 AM, Ron Adam ron3...@gmail.com wrote: Making comprehensions work more like generator expressions would, IMO, imply making the same change to all for loops: having a StopIteration raised by the body of the loop quietly terminate the loop. I'm not suggesting making any changes to generator expressions or for loops at all. They would continue to work like they currently do. But if you're suggesting making list comps react to StopIteration raised by their primary expressions, then to maintain the correspondence between a list comp and its expanded form, for loops would have to change too. Or should that correspondence be broken, in that single-expression loop constructs become semantically different from statement loop constructs? The 2.x correspondence between list comp and for loops was *already broken* in 3.0 by moving execution to a new function to prevent name binding in comprehension from leaking the way they did in 2.x. We did not change for loops to match. The following, which is an example of equivalence in 2.7, raises in 3.4 because 'itertools' is not defined. def binder(ns, name, ob): ns[name] = ob for mod in ['sys']: binder(locals(), mod, __import__(mod)) print(sys) [binder(locals(), mod, __import__(mod)) for mod in ['itertools']] print(itertools) Ceasing to leak *any* local bindings in 3.0 broke equivalence and code depending on such bindings. Ceasing to leak StopIteration would break equivalence a bit more, but only a bit, and code dependent on such leakage, which I expect is extremely rare. I think they would be rare too. With the passage of the PEP, it will change what is different about them once it's in full effect. The stop hack won't work in both, and you may get a RuntimeError in generator expressions where you would get StopIteration in list-comps. (Is this correct?) Cheers, Ron ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Mon, Nov 24, 2014 at 5:28 AM, Ron Adam ron3...@gmail.com wrote:
With the passage of the PEP, it will change what is different about them
once it's in full effect. The stop hack won't work in both, and you may get
a RuntimeError in generator expressions where you would get StopIteration in
list-comps. (Is this correct?)
them being list comps and generator expressions?
The stop hack won't work in either (currently it does work in
genexps), but you'd get a different exception type if you attempt it.
This is correct. It's broadly similar to this distinction:
{1:2,3:4}[50]
Traceback (most recent call last):
File stdin, line 1, in module
KeyError: 50
[1,2,3,4][50]
Traceback (most recent call last):
File stdin, line 1, in module
IndexError: list index out of range
In both lists and dicts, you can't look up something that isn't there.
But you get a slightly different exception type (granted, these two do
have a common superclass) depending on the exact context. But the
behaviour will be effectively the same.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/23/2014 04:15 PM, Chris Angelico wrote:
On Mon, Nov 24, 2014 at 5:28 AM, Ron Adamron3...@gmail.com wrote:
With the passage of the PEP, it will change what is different about them
once it's in full effect. The stop hack won't work in both, and you may get
a RuntimeError in generator expressions where you would get StopIteration in
list-comps. (Is this correct?)
them being list comps and generator expressions?
Yes
The stop hack won't work in either (currently it does work in
genexps), but you'd get a different exception type if you attempt it.
This is correct. It's broadly similar to this distinction:
{1:2,3:4}[50]
Traceback (most recent call last):
File stdin, line 1, in module
KeyError: 50
[1,2,3,4][50]
Traceback (most recent call last):
File stdin, line 1, in module
IndexError: list index out of range
Comprehensions insert/append items, so you wouldn't get those unless you
are reading from another object and would bubble out of generators too.
Which is good because it's most likely an error that needs fixing.
In both lists and dicts, you can't look up something that isn't there.
But you get a slightly different exception type (granted, these two do
have a common superclass) depending on the exact context. But the
behaviour will be effectively the same.
I think the difference is very specific to StopIteration.
And I think we need to wait and see what happens in real code.
Cheers,
Ron
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Mon, Nov 24, 2014 at 10:18 AM, Ron Adam ron3...@gmail.com wrote:
The stop hack won't work in either (currently it does work in
genexps), but you'd get a different exception type if you attempt it.
This is correct. It's broadly similar to this distinction:
{1:2,3:4}[50]
Traceback (most recent call last):
File stdin, line 1, in module
KeyError: 50
[1,2,3,4][50]
Traceback (most recent call last):
File stdin, line 1, in module
IndexError: list index out of range
Comprehensions insert/append items, so you wouldn't get those unless you are
reading from another object and would bubble out of generators too. Which is
good because it's most likely an error that needs fixing.
My point is that doing the same errant operation on a list or a dict
will give different exceptions. In the same way, calling next() on an
empty iterator will raise StopIteration normally, but might raise
RuntimeError instead. It's still an exception, it still indicates a
place where code needs to be changed (unlike, say, a ValueError when
doing type conversions, which often means *data* needs to be changed),
so it doesn't hugely matter _which_ exception is raised. Also, I'm
hoping that someone can improve on my patch by having the full
traceback from the original StopException become visible - then, the
RuntimeError's __context__ will give all the info you would want.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 08:17:00AM -0800, Ethan Furman wrote: While I am in favor of PEP 479, and I have to agree with Raymond that this isn't pretty. Currently, next() accepts an argument of what to return if the iterator is empty. Can we enhance that in some way so that the overall previous behavior could be retained? [...] Then, if the iterator is empty, instead of raising StopIteration, or returning some value that would then have to be checked, it could raise some other exception that is understood to be normal generator termination. We *already* have an exception that is understood to be normal generator termination. It is called StopIteration. Removing the long-standing ability to halt generators with StopIteration, but then recreating that ability under a different name is the worst of both worlds: - working code is still broken; - people will complain that the new exception X is silently swallowed by generators, just as they complained about StopIteration; - it is yet another subtle difference between Python 2 and 3; - it involves a code smell (no constant arguments to functions); - and does nothing to help generators that don't call next(). The current behaviour is nice and clean and has worked well for over a decade. The new behaviour exchanges consistency in one area (generators behave like all other iterators) for consistency in another (generator expressions will behave like comprehensions in the face of StopIteration). But trying to have both at the same time via a new exception adds even more complexity and would leave everyone unhappy. -- Steven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 22 Nov 2014 02:51, Antoine Pitrou solip...@pitrou.net wrote: On Fri, 21 Nov 2014 05:47:58 -0800 Raymond Hettinger raymond.hettin...@gmail.com wrote: Another issue is that it breaks the way I and others have taught for years that generators are a kind of iterator (an object implementing the iterator protocol) and that a primary motivation for generators is to provide a simpler and more direct way of creating iterators. However, Chris explained that, This proposal causes a separation of generators and iterators, so it's no longer possible to pretend that they're the same thing. That is a major and worrisome conceptual shift. I agree with Raymond on this point. A particularly relevant variant of the idiom is the approach of writing __iter__ directly as a generator, rather than creating a separate custom iterator class. In that context, the similarities between the __iter__ implementation and the corresponding explicit __next__ implementation is a beneficial feature. I'm definitely coming around to the point of view that, even if we wouldn't design it the way it currently works given a blank slate, the alternative design doesn't provide sufficient benefit to justify the cost of changing the behaviour getting people to retrain their brains. Cheers, Nick. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 08:31 AM, Nick Coghlan wrote: On 22 Nov 2014 02:51, Antoine Pitrou solip...@pitrou.net mailto:solip...@pitrou.net wrote: On Fri, 21 Nov 2014 05:47:58 -0800 Raymond Hettinger raymond.hettin...@gmail.com mailto:raymond.hettin...@gmail.com wrote: Another issue is that it breaks the way I and others have taught for years that generators are a kind of iterator (an object implementing the iterator protocol) and that a primary motivation for generators is to provide a simpler and more direct way of creating iterators. However, Chris explained that, This proposal causes a separation of generators and iterators, so it's no longer possible to pretend that they're the same thing. That is a major and worrisome conceptual shift. I agree with Raymond on this point. A particularly relevant variant of the idiom is the approach of writing __iter__ directly as a generator, rather than creating a separate custom iterator class. In that context, the similarities between the __iter__ implementation and the corresponding explicit __next__ implementation is a beneficial feature. I'm definitely coming around to the point of view that, even if we wouldn't design it the way it currently works given a blank slate, the alternative design doesn't provide sufficient benefit to justify the cost of changing the behaviour getting people to retrain their brains. This all seems more complex than it should be to me. The way I tend to think about it is simply for loops in one form or another, catch StopIteration. So if you iterate an iterator manually rather than using it as a for iterator, you need to catch StopIteration. If we write a function to act as an iterator, like __next__, we need to raise StopIteration from somewhere in it, or let one bubble out from a generator if we are manually iterating it on each call... next(g). It's possible we may need to do either or both conditionally. That could mean we need to think about refactoring some part of a program, but it doesn't mean Python needs to be fixed. So the lines that split them isn't quite as clear cut as it may seem they should be. That may just be a misplaced ideal. Any time a StopIteration is raised.. either manually with raise, or at the end of a generator, it should bubble up until a for loop iterating over that bit of code, catches it, or a try-except catches it, or fail loudly. I think it does do this in normal generators, so I don't see an issue with how StopIteration works. Which gets us back to generator expressions and comprehensions. Let me know if I got some part of this wrong... :-) Comprehensions are used as a convenient way to create an object. The expression parts of the comprehension define the *body* of a loop, so a StopIteration raised in it will bubble out. As it would in any other case where it is raised in the body of a loop. Generator exprssions on the other hand define the *iterator* to be used in a for loop. A StopIteration raised in it is caught by the for loop. So they both work as they are designed, but they look so similar, it looks like one is broken. It looks to me that there are three options... OPTION 1: Make comprehensions act more like generator expressions. It would mean a while loop in the object creation point is converted to a for loop. (or something equivalent.) Then both a comprehension and a generator expressions can be viewed as defining iterators, with the same behaviour, rather than comprehensions defining the body of the loop, which has the different but valid behaviour of StopIteration escaping. This would make explaining them a *lot* easier as they become the same thing used in a different context, rather than two different things used in what appears to be similar contexts. I think this fits with what Guido wants, but does so in a narrower scope, only effecting comprehensions. StopIteration is less likely to leak out. But it also allows the use of the stop() hack to raise StopIteration in comprehensions and terminate them early. Currently it doesn't work as it does in generator expressions. If the stop() hack works in both comprehensions and generator expressions the same way, then maybe we can view it as less of a hack. OPTION 2: Make generator expressions more like comprehensions. This would mean StopIteration would bubble out of generator expressions as the person who posted the original topic on python ideas wanted. And the stop hack would no longer work. Both generator expressions and comprehensions could be viewed as supplying the body in a loop. This is inconsistent with defining generators that act as iterators. So I'm definitely -1 on this option. OPTION 3: Document the differences better. Cheers, Ron ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 06:31 AM, Nick Coghlan wrote: A particularly relevant variant of the idiom is the approach of writing __iter__ directly as a generator, rather than creating a separate custom iterator class. In that context, the similarities between the __iter__ implementation and the corresponding explicit __next__ implementation is a beneficial feature. https://docs.python.org/3/reference/datamodel.html?highlight=__iter__#object.__iter__ -- This method is called when an iterator is required for a container. This method should return a new iterator object that can iterate over all the objects in the container. For mappings, it should iterate over the keys of the container, and should also be made available as the method keys(). Iterator objects also need to implement this method; they are required to return themselves. For more information on iterator objects, see Iterator Types. Unless the object is itself at iterator, the __iter__ method is allowed to return any iterator object; whether that iterator is constructed by a separate class entirely, or by using the iter() function, or by writing a generator, should have no bearing on how we write generators themselves. -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 6:49 AM, Ron Adam ron3...@gmail.com wrote: OPTION 1: Make comprehensions act more like generator expressions. It would mean a while loop in the object creation point is converted to a for loop. (or something equivalent.) Then both a comprehension and a generator expressions can be viewed as defining iterators, with the same behaviour, rather than comprehensions defining the body of the loop, which has the different but valid behaviour of StopIteration escaping. This would make explaining them a *lot* easier as they become the same thing used in a different context, rather than two different things used in what appears to be similar contexts. I think this fits with what Guido wants, but does so in a narrower scope, only effecting comprehensions. StopIteration is less likely to leak out. A list comp is usually compared to a statement-form loop. def stop(): raise StopIteration lst = [stop() for x in (1,2,3)] lst = [] for x in (1,2,3): lst.append(stop()) At the moment, these are identical (virtually; the pedantic will point out that 'x' doesn't leak out of the comprehension) - in each case, the exception raised by the body will bubble up and be printed to the console. But a generator expression is different: lst = list(stop() for x in (1,2,3)) In this form, lst is an empty list, and nothing is printed to the console. Making comprehensions work more like generator expressions would, IMO, imply making the same change to all for loops: having a StopIteration raised by the body of the loop quietly terminate the loop. This is backwards. Most of Python is about catching exceptions as narrowly as possible: you make your try blocks small, you use specific exception types rather than bare except: clauses, etc, etc, etc. You do your best to catch ONLY those exceptions that you truly understand, unless you're writing a catch, log, and reraise or catch, log, and go back for more work generic handler. A 'for' loop currently is written more-or-less like this: for var in expr: body it = iter(expr) while True: try: var = next(it) except StopIteration: break body But this suggestion of yours would make it become: it = iter(expr) while True: try: var = next(it) body except StopIteration: break I believe this to be the wrong direction to make the change. Instead, generator expressions should be the ones to change, because that's a narrowing of exception-catching scope. Currently, every generator function is implicitly guarded by try: .. except StopIteration: pass. This is allowing errors to pass silently, to allow a hack involving non-local control flow (letting a chained function call terminate a generator) - violating the Zen of Python. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Nov 22, 2014, at 6:31 AM, Nick Coghlan ncogh...@gmail.com wrote: I'm definitely coming around to the point of view that, even if we wouldn't design it the way it currently works given a blank slate, the alternative design doesn't provide sufficient benefit to justify the cost of changing the behaviour getting people to retrain their brains. Thanks Nick. That was well said. After a couple more days of thinking about PEP 455 and reading many of the mailing list posts, I am also still flatly opposed to the proposal and wanted to summarize my thoughts for everyone's consideration. It looks like the only point in favor of the PEP is makes the internal semantics of genexps appear to more closely match list comprehensions. However, it does so not by fixing anything or adding a capability; rather, it disallows a coding pattern that potentially exposes the true differences between genexps and list comps. Even then, the difference isn't hidden; instead, the proposal just breaks the code loudly by raising a RuntimeError. AFAICT, the problem it solves isn't really a problem in practice. (I do code reviews and teach Python for living, so I get broad exposure to how python is used in practice). As collateral damage, the PEP breaks code that is otherwise well designed, beautiful looking, and functioning correctly. Even worse, the damage will be long lasting. In introduces a new special case in the current clean design of generators. And, it creates an additional learning burden (we would now have to teach the special case and how to work around it with a try: v=next(it) except StopIteration: return. I realize these are sweeping statements, so I elaborate with more detail and examples below. If you're not interested in the details, feel free to skip the rest of the post; you've already gotten the keys points. New Special Case By design, exceptions raised in generators are passed through to the caller. This includes IndexErrors, ValueErrors, StopIteration, and PEP 342's GeneratorExit. Under the proposed PEP, there general rule (exceptions are passed through) is broken and there is a new special case: StopIteration exceptions are caught and reraised as RuntimeErrors. This is a surprising new behavior. Breaks well established, documented, and tested behaviors - From the first day generators were introduced 13 years ago, we have documented and promised that you can end a terminate a generator by raising StopIteration or by a return-statment (that is in PEP 255, in the whatsnew document for 2.2, in the examples we provided for the myriad of ways to use generators, in standard library code, in the Martelli's Python Cookbook example, in the documention for itertools, etc.) Legitimate Use Cases for Raising StopIteration in a Generator In a producer/consumer generator chain, the input generator signals it is done by raising StopIteration and the output generator signals that it is done by raising StopIteration (this isn't in dispute). That use case is currently implemented something like this: def middleware_generator(source_generator): it = source_generator() input_value = next(it) output_value = do_something_interesting(input_value) yield output_value Termination of the input stream will then terminate middleware stream. You can see several real world examples of this code pattern in Fredrik Lundh's pure python verion of ElementTree (prepare_descendant, prepare_predicate, and iterfind). Under the proposal, the new logic would be to: 1) catch input stream StopIteration 2) return from the generator 3) which in turn raises StopIteration yet again. This doesn't make the code better in any way. The new code is wordy, slow, and unnecessarily convoluted: def middleware_generator(source_generator): it = source_generator() try: input_value = next(it) except StopIteration: return # This causes StopIteration to be reraised output_value = do_something_interesting(input_value) yield output_value I don't look forward to teaching people to have to write code like this and to have to remember yet another special case rule for Python. Is next() Surprising? - The PEP author uses the word surprise many times in describing the motivation for the PEP. In the context of comparing generator expressions to list comprehenions, I can see where someone might be surprised that though similar in appearance, their implementations are quite different and that some of those differences might not expected. However, I believe this is where the surprise ends. The behavior of next(it) is to return a value or raise StopIteration. That is fundamental to what is does (what else could it do?). This is as basic as list indexing returning a value or
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 2:49 PM, Ron Adam wrote:
On 11/22/2014 08:31 AM, Nick Coghlan wrote:
I'm definitely coming around to the point of view that, even if we
wouldn't
design it the way it currently works given a blank slate, the alternative
design doesn't provide sufficient benefit to justify the cost of changing
the behaviour getting people to retrain their brains.
Me too.
Which gets us back to generator expressions and comprehensions.
Comprehensions are used as a convenient way to create an object. The
expression parts of the comprehension define the *body* of a loop, so a
StopIteration raised in it will bubble out. As it would in any other
case where it is raised in the body of a loop.
Generator expressions on the other hand define the *iterator* to be used
in a for loop. A StopIteration raised in it is caught by the for loop.
So they both work as they are designed, but they look so similar, it
looks like one is broken.
It looks to me that there are three options...
OPTION 1:
Make comprehensions act more like generator expressions.
I was thinking about this also.
It would mean a while loop in the object creation point is converted to
a for loop. (or something equivalent.)
The code for [e(i) for i in it] already uses a for loop. The issue is
when e(i) raise StopIteration. The solution to accomplish the above is
to wrap the for loop in try: ... except StopIteration: pass.
def _list_comp_temp():
ret = []
try:
for i in it:
ret.append[e(i)]
except StopIteration:
pass
return ret
I believe this would make list(or set)(genexp) == [genexp] (or {genexp})
as desired. In 2.x, there was a second difference, the leaking of 'i'.
3.x eliminated one difference, the leaking of the iteration variable,
but not the other, the leaking of StopIteration.
The document about execution of comprehensions says the comprehension
is executed in a separate scope, so names assigned to in the target list
don’t “leak”. What we would be adding for comprensions (but not genexps)
is and StopIteration raised in evaluating the expression before
'leak'. We could then document the equality above.
Then both a comprehension and a generator expressions can be viewed as
defining iterators,
A comprehension is not an iterator. The above would make a list or set
comprehension the same as feeding a genexp to list() or set().
I think this fits with what Guido wants, but does so in a narrower
scope, only effecting comprehensions. StopIteration is less likely to
leak out.
StopIteration would *not* leak out ever. This is half of what Guido
wants, the other half being the issue of obscure bugs with genrators in
general.
But it also allows the use of the stop() hack to raise StopIteration in
comprehensions and terminate them early.
Yep. There is no good way to have next(it) work as intended, including
in Raymond's example, where it should work, and not let stop() work. I
think this falls under our 'consenting adults' principle.
Currently it doesn't work as it does in generator expressions.
If the stop() hack works in both comprehensions and generator
expressions the same way, then maybe we can view it as less of a hack.
--
Terry Jan Reedy
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 02:16 PM, Chris Angelico wrote: On Sun, Nov 23, 2014 at 6:49 AM, Ron Adamron3...@gmail.com wrote: OPTION 1: Make comprehensions act more like generator expressions. It would mean a while loop in the object creation point is converted to a for loop. (or something equivalent.) Then both a comprehension and a generator expressions can be viewed as defining iterators, with the same behaviour, rather than comprehensions defining the body of the loop, which has the different but valid behaviour of StopIteration escaping. This would make explaining them a*lot* easier as they become the same thing used in a different context, rather than two different things used in what appears to be similar contexts. I think this fits with what Guido wants, but does so in a narrower scope, only effecting comprehensions. StopIteration is less likely to leak out. A list comp is usually compared to a statement-form loop. def stop(): raise StopIteration lst = [stop() for x in (1,2,3)] lst = [] for x in (1,2,3): lst.append(stop()) At the moment, these are identical (virtually; the pedantic will point out that 'x' doesn't leak out of the comprehension) - in each case, the exception raised by the body will bubble up and be printed to the console. This is what I meant by leaking out. A StopIteration bubbles up. And your examples match my understanding. :-) But a generator expression is different: Yes, but they work as I expect them to. lst = list(stop() for x in (1,2,3)) In this form, lst is an empty list, and nothing is printed to the console. I think that is what it should do. I think of it this way... def stop(): raise StopIteration ... def ge(): ...for value in (stop() for x in (1,2,3)): ... yield value ... list(ge()) [] Notice the entire body of the comprehension is in the for loop header, and no parts of it are in the body except the reference to the already assigned value. The StopIteration is caught by the outer for loop. Not the for loop in the generator expression, or iterator part. Making comprehensions work more like generator expressions would, IMO, imply making the same change to all for loops: having a StopIteration raised by the body of the loop quietly terminate the loop. I'm not suggesting making any changes to generator expressions or for loops at all. They would continue to work like they currently do. Cheers, Ron This is backwards. Most of Python is about catching exceptions as narrowly as possible: you make your try blocks small, you use specific exception types rather than bare except: clauses, etc, etc, etc. You do your best to catch ONLY those exceptions that you truly understand, unless you're writing a catch, log, and reraise or catch, log, and go back for more work generic handler. A 'for' loop currently is written more-or-less like this: for var in expr: body it = iter(expr) while True: try: var = next(it) except StopIteration: break body But this suggestion of yours would make it become: it = iter(expr) while True: try: var = next(it) body except StopIteration: break I believe this to be the wrong direction to make the change. Instead, generator expressions should be the ones to change, because that's a narrowing of exception-catching scope. Currently, every generator function is implicitly guarded by try: .. except StopIteration: pass. This is allowing errors to pass silently, to allow a hack involving non-local control flow (letting a chained function call terminate a generator) - violating the Zen of Python. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 03:01 PM, Terry Reedy wrote: Then both a comprehension and a generator expressions can be viewed as defining iterators, A comprehension is not an iterator. The above would make a list or set comprehension the same as feeding a genexp to list() or set(). Correct, but we could think and reason about them in the same way. Cheers, Ron ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 8:03 AM, Ron Adam ron3...@gmail.com wrote: Making comprehensions work more like generator expressions would, IMO, imply making the same change to all for loops: having a StopIteration raised by the body of the loop quietly terminate the loop. I'm not suggesting making any changes to generator expressions or for loops at all. They would continue to work like they currently do. But if you're suggesting making list comps react to StopIteration raised by their primary expressions, then to maintain the correspondence between a list comp and its expanded form, for loops would have to change too. Or should that correspondence be broken, in that single-expression loop constructs become semantically different from statement loop constructs? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 7:30 AM, Raymond Hettinger raymond.hettin...@gmail.com wrote: Legitimate Use Cases for Raising StopIteration in a Generator In a producer/consumer generator chain, the input generator signals it is done by raising StopIteration and the output generator signals that it is done by raising StopIteration (this isn't in dispute). That use case is currently implemented something like this: def middleware_generator(source_generator): it = source_generator() input_value = next(it) output_value = do_something_interesting(input_value) yield output_value Does your middleware_generator work with just a single element, yielding either one output value or none? Or is it more likely to be iterating over the source generator: def middleware_generator(source_generator): for input_value in source_generator: yield do_something_interesting(input_value) MUCH tidier code, plus it's safe against unexpected StopIterations. Even if you can't do it this way, all you need is to stick a 'break' at the end of the loop, if the try/except is so abhorrent. Wouldn't be the first time I've seen a loop with a hard break at the end of it. This doesn't make the code better in any way. The new code is wordy, slow, and unnecessarily convoluted: def middleware_generator(source_generator): it = source_generator() try: input_value = next(it) except StopIteration: return # This causes StopIteration to be reraised output_value = do_something_interesting(input_value) yield output_value The raising of StopIteration here is an implementation detail; you might just as well have a comment saying This causes the function to set an exception state and return NULL, which is what happens at the C level. What happens if do_something_interesting happens to raise StopIteration? Will you be surprised that this appears identical to the source generator yielding nothing? That's current behaviour. You don't have the option of narrowing the try/except scope as you've done above. Is next() Surprising? - If someone in this thread says that they were suprised that next() could raise StopIteration, I don't buy it. Agreed, I don't think that's surprising to anyone. Being able to consume a value from an iterator stream is a fundamental skill and not hard to learn (when I teach iterators and generators, the operation of next() has never been a stumbling block). In anything other than a generator, you're expected to cope with two possible results from next(): a returned value or a raised StopIteration. Suppose you want to read a file with a header - you'd need to do something like this: def process_file(f): f = iter(f) try: header=next(f) except StopIteration: cope_somehow_maybe_return for line in f: process_line_with_headers(line, header) Currently, *if and only if* you're writing a generator, you have an implicit except StopIteration: return there. Anywhere else, you need to catch that exception. Iterators are an implementation detail of generators. There is no particular reason for a generator author to be aware of iterator protocol, any more than this class needs to be: class X: def __iter__(self): return iter([1,2,3,4]) It's perfectly iterable, just as a generator is, and it knows nothing about StopIteration. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 04:23 PM, Chris Angelico wrote:
On Sun, Nov 23, 2014 at 8:03 AM, Ron Adamron3...@gmail.com wrote:
Making comprehensions work more like generator expressions
would, IMO, imply making the same change to all for loops: having a
StopIteration raised by the body of the loop quietly terminate the
loop.
I'm not suggesting making any changes to generator expressions or for loops
at all. They would continue to work like they currently do.
But if you're suggesting making list comps react to StopIteration
raised by their primary expressions, then to maintain the
correspondence between a list comp and its expanded form, for loops
would have to change too.
Or should that correspondence be broken, in
that single-expression loop constructs become semantically different
from statement loop constructs?
Se we have these...
Tuple Comprehension (...)
List Comprehension [...]
Dict Comprehension {...} Colon make's it different from sets.
Set Comprehension {...}
I don't think there is any other way to create them. And they don't
actually expand to any python code that is visible to a programmer. They
are self contained literal expressions for creating these few objects.
The expanded form you are referring to is just how we currently explain
them. And yes, we will need to alter how we currently explain
Comprehensions a bit if this is done.
Here is what I think(?) list comps do currently.
lst = [expr for items in itr if cond]
Is the same as.
lst = []
for x in itr: # StopIteration caught here.
if cond: # StopIteration bubbles here.
lst.append(expr) # StopIteration bubbles here.
And it would be changed to this...
def comp_expression():
for x in itr: # StopIteration caught here.
if cond:
list.append(expr)
# StopIteration from cond and expr caught here.
lst = list(x for x in comp_expression())
So
[expr for itr if cond]
Becomes a literal form of:
list(expr for itr if cond)
And I believe that is how they are explained quite often. :-)
(Although It's not currently quite true, is it?)
Cheers,
Ron
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 11:05 AM, Ron Adam ron3...@gmail.com wrote:
Se we have these...
Tuple Comprehension (...)
List Comprehension [...]
Dict Comprehension {...} Colon make's it different from sets.
Set Comprehension {...}
I don't think there is any other way to create them. And they don't actually
expand to any python code that is visible to a programmer. They are self
contained literal expressions for creating these few objects.
Hmmm, there's no such thing as tuple comprehensions. Are you confusing
comprehension syntax (which has a 'for' loop in it) with
tuple/list/etc display, which doesn't?
lst = [1,2,3,4] # not a comprehension
even = [n*2 for n in lst] # comprehension
Here is what I think(?) list comps do currently.
lst = [expr for items in itr if cond]
Is the same as.
lst = []
for x in itr: # StopIteration caught here.
if cond: # StopIteration bubbles here.
lst.append(expr) # StopIteration bubbles here.
Pretty much. It's done in a nested function (so x doesn't leak), but
otherwise, yes.
And it would be changed to this...
def comp_expression():
for x in itr: # StopIteration caught here.
if cond:
list.append(expr)
# StopIteration from cond and expr caught here.
lst = list(x for x in comp_expression())
That wouldn't quite work, but this would:
def listcomp():
ret = []
try:
for x in itr:
if cond:
ret.append(x)
except StopIteration:
pass
return ret
lst = listcomp()
(And yes, the name's syntactically illegal, but if you look at a
traceback, that's what is used.)
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 06:20 PM, Chris Angelico wrote:
On Sun, Nov 23, 2014 at 11:05 AM, Ron Adamron3...@gmail.com wrote:
Se we have these...
Tuple Comprehension (...)
List Comprehension [...]
Dict Comprehension {...} Colon make's it different from sets.
Set Comprehension {...}
I don't think there is any other way to create them. And they don't actually
expand to any python code that is visible to a programmer. They are self
contained literal expressions for creating these few objects.
Hmmm, there's no such thing as tuple comprehensions.
Just didn't think it through quite well enough. But you are correct, that
would be a generator expression.
One less case to worry about. :-)
lst = [1,2,3,4] # not a comprehension
even = [n*2 for n in lst] # comprehension
Here is what I think(?) list comps do currently.
lst = [expr for items in itr if cond]
Is the same as.
lst = []
for x in itr: # StopIteration caught here.
if cond: # StopIteration bubbles here.
lst.append(expr) # StopIteration bubbles here.
Pretty much. It's done in a nested function (so x doesn't leak), but
otherwise, yes.
And it would be changed to this...
def comp_expression():
for x in itr: # StopIteration caught here.
if cond:
list.append(expr)
# StopIteration from cond and expr caught here.
lst = list(x for x in comp_expression())
That wouldn't quite work, but this would:
Right, the list.append() should be a yield(expr).
def listcomp():
ret = []
try:
for x in itr:
if cond:
ret.append(x)
except StopIteration:
pass
return ret
lst = listcomp()
(And yes, the name's syntactically illegal, but if you look at a
traceback, that's what is used.)
That's fine too.
The real question is how much breakage would such a change make? That will
probably need a patch in order to test it out.
Cheers,
Ron
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 11:51 AM, Ron Adam ron3...@gmail.com wrote: On 11/22/2014 06:20 PM, Chris Angelico wrote: Hmmm, there's no such thing as tuple comprehensions. Just didn't think it through quite well enough. But you are correct, that would be a generator expression. One less case to worry about. :-) Ah right, no probs. And it would be changed to this... def comp_expression(): for x in itr: # StopIteration caught here. if cond: list.append(expr) # StopIteration from cond and expr caught here. lst = list(x for x in comp_expression()) Right, the list.append() should be a yield(expr). In that case, your second generator expression is entirely redundant; all you want is list(comp_expression()). But the example doesn't say *why* this version should terminate on a StopIteration raised by expr, when the statement form would print an exception traceback. The real question is how much breakage would such a change make? That will probably need a patch in order to test it out. There's one attached here: http://bugs.python.org/issue22906 It doesn't create a __future__ directive, just applies the change globally. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Nov 22, 2014, at 2:45 PM, Chris Angelico ros...@gmail.com wrote:
Does your middleware_generator work with just a single element,
yielding either one output value or none?
I apologize if I didn't make the point clearly. The middleware example was
just simple outline of calling next(), doing some processing, and yielding a
result while letting the StopIteration float through from the next() call.
It was meant to show in summary form a pattern for legitimate uses of next()
inside a generator. Some of those uses benefit from letting their
StopIteration
pass through rather than being caught, returning, and reraising the
StopIteration.
The worry is that your proposal intentionally breaks that code which is
currently
bug free, clean, fast, stable, and relying on a part of the API that has been
guaranteed and documented from day one.
Since the middleware() example was ineffective in communicating the need,
here are some real-world examples.
Here's one from Fredrick Lundh's ElementTree code in the standard library
(there are several other examples besides this one in his code are well):
def iterfind(elem, path, namespaces=None):
# compile selector pattern
cache_key = (path, None if namespaces is None
else tuple(sorted(namespaces.items(
if path[-1:] == /:
path = path + * # implicit all (FIXME: keep this?)
try:
selector = _cache[cache_key]
except KeyError:
if len(_cache) 100:
_cache.clear()
if path[:1] == /:
raise SyntaxError(cannot use absolute path on element)
next = iter(xpath_tokenizer(path, namespaces)).__next__
token = next()
selector = []
while 1:
try:
selector.append(ops[token[0]](next, token))
except StopIteration:
raise SyntaxError(invalid path)
try:
token = next()
if token[0] == /:
token = next()
except StopIteration:
break
_cache[cache_key] = selector
# execute selector pattern
result = [elem]
context = _SelectorContext(elem)
for select in selector:
result = select(context, result)
return result
And here is an example from the pure python version of one of the itertools:
def accumulate(iterable, func=operator.add):
'Return running totals'
# accumulate([1,2,3,4,5]) -- 1 3 6 10 15
# accumulate([1,2,3,4,5], operator.mul) -- 1 2 6 24 120
it = iter(iterable)
total = next(it)
yield total
for element in it:
total = func(total, element)
yield total
And here is an example from Django:
def _generator():
it = iter(text.split(' '))
word = next(it)
yield word
pos = len(word) - word.rfind('\n') - 1
for word in it:
if \n in word:
lines = word.split('\n')
else:
lines = (word,)
pos += len(lines[0]) + 1
if pos width:
yield '\n'
pos = len(lines[-1])
else:
yield ' '
if len(lines) 1:
pos = len(lines[-1])
yield word
return ''.join(_generator())
I could scan for even more examples, but I think you get the gist.
All I'm asking is that you consider that your proposal will do more
harm than good. It doesn't add any new capability at all.
It just kills some code that currently works.
Raymond
(the author of the generator expressions pep)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sun, Nov 23, 2014 at 12:11 PM, Raymond Hettinger
raymond.hettin...@gmail.com wrote:
The worry is that your proposal intentionally breaks that code which is
currently
bug free, clean, fast, stable, and relying on a part of the API that has
been
guaranteed and documented from day one.
(I'd just like to mention that this isn't my proposal, beyond that
I'm doing the summarizing and PEP writing. The proposal itself is
primarily derived from one of Guido's posts on -ideas.)
Here's one from Fredrick Lundh's ElementTree code in the standard library
(there are several other examples besides this one in his code are well):
def iterfind(elem, path, namespaces=None):
# compile selector pattern
cache_key = (path, None if namespaces is None
else tuple(sorted(namespaces.items(
if path[-1:] == /:
path = path + * # implicit all (FIXME: keep this?)
try:
selector = _cache[cache_key]
except KeyError:
if len(_cache) 100:
_cache.clear()
if path[:1] == /:
raise SyntaxError(cannot use absolute path on element)
next = iter(xpath_tokenizer(path, namespaces)).__next__
token = next()
selector = []
while 1:
try:
selector.append(ops[token[0]](next, token))
except StopIteration:
raise SyntaxError(invalid path)
try:
token = next()
if token[0] == /:
token = next()
except StopIteration:
break
_cache[cache_key] = selector
# execute selector pattern
result = [elem]
context = _SelectorContext(elem)
for select in selector:
result = select(context, result)
return result
Most of the next() calls are already guarded with try/except; from
what I can see, only the first one isn't. So this wouldn't be much of
a change here.
And here is an example from the pure python version of one of the itertools:
def accumulate(iterable, func=operator.add):
'Return running totals'
# accumulate([1,2,3,4,5]) -- 1 3 6 10 15
# accumulate([1,2,3,4,5], operator.mul) -- 1 2 6 24 120
it = iter(iterable)
total = next(it)
yield total
for element in it:
total = func(total, element)
yield total
Again, only the first one needs protection, and all that happens is
that there's clearly a control flow possibility here (that the first
yield total might not be reached). Currently, *any* function call
has the potential to be a silent control flow event.
And here is an example from Django:
def _generator():
it = iter(text.split(' '))
word = next(it)
yield word
pos = len(word) - word.rfind('\n') - 1
for word in it:
if \n in word:
lines = word.split('\n')
else:
lines = (word,)
pos += len(lines[0]) + 1
if pos width:
yield '\n'
pos = len(lines[-1])
else:
yield ' '
if len(lines) 1:
pos = len(lines[-1])
yield word
return ''.join(_generator())
When you split a string, you're guaranteed at least one result, ergo
'it' is guaranteed to yield at least one word. So this one wouldn't
need to be changed - it can't possibly raise RuntimeError.
I could scan for even more examples, but I think you get the gist.
All I'm asking is that you consider that your proposal will do more
harm than good. It doesn't add any new capability at all.
It just kills some code that currently works.
I have considered it, and I'm not convinced that it will. I see lots
of people saying code will have to be changed, but that's exactly
the same concern that people raise about switching from the sloppy Py2
merging of text and bytes to the strict Py3 separation - yes, code has
to be changed, but it's definitely much better to have immediate and
obvious failures when something's wrong than to have subtle behavioral
changes.
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 05:11 PM, Raymond Hettinger wrote: On Nov 22, 2014, at 2:45 PM, Chris Angelico wrote: Does your middleware_generator work with just a single element, yielding either one output value or none? I apologize if I didn't make the point clearly. The middleware example was just simple outline of calling next(), doing some processing, and yielding a result while letting the StopIteration float through from the next() call. [middleware example] def middleware_generator(source_generator): it = source_generator() input_value = next(it) output_value = do_something_interesting(input_value) yield output_value The point that Chris made that you should be refuting is this one: What happens if do_something_interesting happens to raise StopIteration? Will you be surprised that this appears identical to the source generator yielding nothing? -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/22/2014 07:06 PM, Chris Angelico wrote: On Sun, Nov 23, 2014 at 11:51 AM, Ron Adamron3...@gmail.com wrote: On 11/22/2014 06:20 PM, Chris Angelico wrote: Hmmm, there's no such thing as tuple comprehensions. Just didn't think it through quite well enough. But you are correct, that would be a generator expression. One less case to worry about.:-) Ah right, no probs. And it would be changed to this... def comp_expression(): for x in itr: # StopIteration caught here. if cond: list.append(expr) # StopIteration from cond and expr caught here. lst = list(x for x in comp_expression()) Right, the list.append() should be a yield(expr). In that case, your second generator expression is entirely redundant; all you want is list(comp_expression()). Yes, and that is good. Simplies it even more. But the example doesn't say *why* this version should terminate on a StopIteration raised by expr, when the statement form would print an exception traceback. I presume you are asking why do this? And not why the example does that? There has been a desires expressed, more than a few times, to make comprehensions more like generator expressions in the past. It looks to me that that desire is still true. I also think there has been quite a bit of confusion in these discussions that could be reduced substantially by making Comprehensions work a bit more like generator expressions. As to why the example does that.. a list constructor iterates over a generator expression in a way that follows the iterator protocol. If you make comprehsionsons work as if they are a generator expression fed to a constructor... then it too should follow the itorator protocol. Do you agree? The real question is how much breakage would such a change make? That will probably need a patch in order to test it out. There's one attached here: http://bugs.python.org/issue22906 Doesn't that patch effect generators and not comprehensions? If so, it wouldn't do what we are talking about here. Ron ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
In order to save everyone's breath, I am *accepting* the proposal of PEP 479. The transition plan is: - from __future__ import generator_stop in 3.5, and a silent deprecation if StopIteration is allowed to bubble out of a generator (i.e. no warning is printed unless you explicitly turn it on) - non-silent deprecation in 3.6 - feature enabled by default in 3.7 The PEP hasn't been updated to include this and it also could use some more editing -- I'll try to get to that Monday. But the specification of the proposal is crystal-clear and I have no doubt that this is the right thing going forward. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Nov 19, 2014, at 12:10 PM, Guido van Rossum gu...@python.org wrote: There's a new PEP proposing to change how to treat StopIteration bubbling up out of a generator frame (not caused by a return from the frame). The proposal is to replace such a StopIteration with a RuntimeError (chained to the original StopIteration), so that only *returning* from a generator (or falling off the end) causes the iteration to terminate. The proposal unifies the behavior of list comprehensions and generator expressions along the lines I had originally in mind when they were introduced. It renders useless/illegal certain hacks that have crept into some folks' arsenal of obfuscated Python tools. I strongly recommend against accepting this PEP. The PEP itself eloquently articulates an important criticism, Unofficial and apocryphal statistics suggest that this is seldom, if ever, a problem. [4] https://www.python.org/dev/peps/pep-0479/#id16Code does exist which relies on the current behaviour (e.g. [2] https://www.python.org/dev/peps/pep-0479/#id14, [5] https://www.python.org/dev/peps/pep-0479/#id17, [6] https://www.python.org/dev/peps/pep-0479/#id18), and there is the concern that this would be unnecessary code churn to achieve little or no gain.. Another issue is that it breaks the way I and others have taught for years that generators are a kind of iterator (an object implementing the iterator protocol) and that a primary motivation for generators is to provide a simpler and more direct way of creating iterators. However, Chris explained that, This proposal causes a separation of generators and iterators, so it's no longer possible to pretend that they're the same thing. That is a major and worrisome conceptual shift. Also, the proposal breaks a reasonably useful pattern of calling next(subiterator) inside a generator and letting the generator terminate when the data stream ends. Here is an example that I have taught for years: def izip(iterable1, iterable2): it1 = iter(iterable1) it2 = iter(iterable2) while True: v1 = next(it1) v2 = next(it2) yield v1, v2 The above code is not atypical. Several of the pure python equivalents in the itertools docs have documented this pattern to the world for over a decade. I have seen it other people's code bases as well (in several contexts including producer/consumer chains, generators that use next() to fetch initial values from a stream, and generators that have multiple subiterators). This behavior was guaranteed from day one in PEP 255, so we would be breaking a long-standing, published rule. Adding a try/except to catch the StopIteration make the above code compliant with the new PEP, but it wouldn't make the code better in any way. And after that fix, the code would be less beautiful that it is now, and I think that matters. Lastly, as I mentioned on python-ideas, if we really want people to migrate to Python 3, there should be a strong aversion to further increasing the semantic difference between Python 2 and Python 3 without a really good reason. Raymond P.S. The PEP 255 promise was also announced as a practice in the WhatsNew 2.2 document (where most people first learned what generators are and how to use them), The end of the generator’s results can also be indicated by raising StopIteration https://docs.python.org/3/library/exceptions.html#StopIteration manually, or by just letting the flow of execution fall off the bottom of the function. The technique was also used in the generator tutorial (which was tested Lib/test/test_generators.py). One other thought: A number of other languages have added generators modeled on the Python implementation. It would be worthwhile to check to see what the prevailing wisdom is regarding whether it should be illegal to raise StopIteration inside a generator. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sat, Nov 22, 2014 at 12:47 AM, Raymond Hettinger raymond.hettin...@gmail.com wrote: Also, the proposal breaks a reasonably useful pattern of calling next(subiterator) inside a generator and letting the generator terminate when the data stream ends. Here is an example that I have taught for years: def izip(iterable1, iterable2): it1 = iter(iterable1) it2 = iter(iterable2) while True: v1 = next(it1) v2 = next(it2) yield v1, v2 Is it obvious to every user that this will consume an element from it1, then silently terminate if it2 no longer has any content? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 21 November 2014 13:53, Chris Angelico ros...@gmail.com wrote: On Sat, Nov 22, 2014 at 12:47 AM, Raymond Hettinger raymond.hettin...@gmail.com wrote: Also, the proposal breaks a reasonably useful pattern of calling next(subiterator) inside a generator and letting the generator terminate when the data stream ends. Here is an example that I have taught for years: def izip(iterable1, iterable2): it1 = iter(iterable1) it2 = iter(iterable2) while True: v1 = next(it1) v2 = next(it2) yield v1, v2 Is it obvious to every user that this will consume an element from it1, then silently terminate if it2 no longer has any content? It is to me, certainly. I'm mostly with Raymond on this. The main exception is that there does seem to be a trend towards more tricky uses of this feature, typically around ideas such as a stop() function to break out of generator expressions. I do think this is bad practice and shouldn't be allowed. But given that it's possible to write bad code in plenty of ways, and changing the language to protect people from themselves is not a good idea, I don't think the benefits justify the change [1]. Paul [1] Probably simply telling people not to try to cram over-complex code into a genexp would address the worst aspects of the problem... ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sat, Nov 22, 2014 at 12:53:41AM +1100, Chris Angelico wrote:
On Sat, Nov 22, 2014 at 12:47 AM, Raymond Hettinger
raymond.hettin...@gmail.com wrote:
Also, the proposal breaks a reasonably useful pattern of calling
next(subiterator) inside a generator and letting the generator terminate
when the data stream ends. Here is an example that I have taught for
years:
def izip(iterable1, iterable2):
it1 = iter(iterable1)
it2 = iter(iterable2)
while True:
v1 = next(it1)
v2 = next(it2)
yield v1, v2
Is it obvious to every user that this will consume an element from
it1, then silently terminate if it2 no longer has any content?
Every user? Of course not. But it should be obvious to those who think
carefully about the specification of zip() and what is available to
implement it.
zip() can't detect that the second argument is empty except by calling
next(), which it doesn't do until after it has retrieved a value from
the first argument. If it turns out the second argument is empty, what
can it do with that first value? It can't shove it back into the
iterator. It can't return a single value, or pad it with some sentinel
value (that's what izip_longest does). Since zip() is documented as
halting on the shorter argument, it can't raise an exception. So what
other options are there apart from silently consuming the value?
Indeed that is exactly what the built-in zip does:
py a = iter(abcdef)
py b = iter(abc)
py list(zip(a, b))
[('a', 'a'), ('b', 'b'), ('c', 'c')]
py next(a)
'e'
--
Steven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sat, Nov 22, 2014 at 2:58 AM, Steven D'Aprano st...@pearwood.info wrote: Since zip() is documented as halting on the shorter argument, it can't raise an exception. So what other options are there apart from silently consuming the value? Sure, it's documented as doing that. But imagine something that isn't a well-known function - all you have is someone writing a generator that calls next() in multiple places. Is it obvious that it it'll silently terminate as soon as any one of those iterators is exhausted? In many cases, an exception (probably ValueError?) would be the most obvious response. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 21 November 2014 15:58, Steven D'Aprano st...@pearwood.info wrote: def izip(iterable1, iterable2): it1 = iter(iterable1) it2 = iter(iterable2) while True: v1 = next(it1) v2 = next(it2) yield v1, v2 Is it obvious to every user that this will consume an element from it1, then silently terminate if it2 no longer has any content? Every user? Of course not. But it should be obvious to those who think carefully about the specification of zip() and what is available to implement it. zip() can't detect that the second argument is empty except by calling next(), which it doesn't do until after it has retrieved a value from the first argument. If it turns out the second argument is empty, what can it do with that first value? It can't shove it back into the iterator. It can't return a single value, or pad it with some sentinel value (that's what izip_longest does). Since zip() is documented as halting on the shorter argument, it can't raise an exception. So what other options are there apart from silently consuming the value? Interestingly, although I said yes, it's obvious, I'd missed this subtlety. But I don't consider it unexpected or a gotcha, just subtle. I certainly don't consider it to be the *wrong* behaviour - on the contrary, I'd be more surprised to get a RuntimeError when the second iterator was shorter. What I understand to be the recommended alternative: def izip(iterable1, iterable2): it1 = iter(iterable1) it2 = iter(iterable2) while True: try: # Is it OK to cover both statements with one try...except? # I think it should be, but it's a pattern I try to avoid v1 = next(it1) v2 = next(it2) except StopIteration: return yield v1, v2 looks less obvious to me, and obscures the intent a little. (Note that I understand this is only one example and that others present a much more compelling case in favour of the explicit return form) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/21/2014 05:47 AM, Raymond Hettinger wrote: Also, the proposal breaks a reasonably useful pattern of calling next(subiterator) inside a generator and letting the generator terminate when the data stream ends. Here is an example that I have taught for years: def [...] it1 = iter(iterable1) it2 = iter(iterable2) while True: v1 = next(it1) v2 = next(it2) yield v1, v2 Stepping back a little and looking at this code, sans header, let's consider the possible desired behaviors: - have an exact match-up between the two iterators, error otherwise - stop when one is exhausted - pad shorter one to longer one Two of those three possible options are going to require dealing with the StopIteration that shouldn't escape -- is the trade of keeping one option short and simple worth the pain caused by the error-at-a-distance bugs caused when a StopIteration does escape that shouldn't have? -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Nov 21, 2014, at 11:31 AM, Ethan Furman et...@stoneleaf.us wrote: On 11/21/2014 05:47 AM, Raymond Hettinger wrote: Also, the proposal breaks a reasonably useful pattern of calling next(subiterator) inside a generator and letting the generator terminate when the data stream ends. Here is an example that I have taught for years: def [...] it1 = iter(iterable1) it2 = iter(iterable2) while True: v1 = next(it1) v2 = next(it2) yield v1, v2 Stepping back a little and looking at this code, sans header, let's consider the possible desired behaviors: - have an exact match-up between the two iterators, error otherwise - stop when one is exhausted - pad shorter one to longer one Two of those three possible options are going to require dealing with the StopIteration that shouldn't escape -- is the trade of keeping one option short and simple worth the pain caused by the error-at-a-distance bugs caused when a StopIteration does escape that shouldn't have? I don’t have an opinion on whether it’s enough of a big deal to actually change it, but I do find wrapping it with a try: except block and returning easier to understand. If you showed me the current code unless I really thought about it I wouldn't think about the fact that the next() calls can cause the generator to terminate. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Sat, Nov 22, 2014 at 3:37 AM, Donald Stufft don...@stufft.io wrote: I don’t have an opinion on whether it’s enough of a big deal to actually change it, but I do find wrapping it with a try: except block and returning easier to understand. If you showed me the current code unless I really thought about it I wouldn't think about the fact that the next() calls can cause the generator to terminate. And don't forget, by the way, that you can always request the current behaviour by explicitly wrapping the body of the function in try/except: def izip(iterable1, iterable2): try: it1 = iter(iterable1) it2 = iter(iterable2) while True: v1 = next(it1) v2 = next(it2) yield v1, v2 except StopIteration: pass That's exactly what current behaviour does, and if you think that that try block is too broad and should be narrowed, then you should support this proposal :) ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Fri, 21 Nov 2014 05:47:58 -0800 Raymond Hettinger raymond.hettin...@gmail.com wrote: Another issue is that it breaks the way I and others have taught for years that generators are a kind of iterator (an object implementing the iterator protocol) and that a primary motivation for generators is to provide a simpler and more direct way of creating iterators. However, Chris explained that, This proposal causes a separation of generators and iterators, so it's no longer possible to pretend that they're the same thing. That is a major and worrisome conceptual shift. I agree with Raymond on this point. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Thu, Nov 20, 2014 at 11:36:54AM -0800, Guido van Rossum wrote: [...] That said, I think for most people the change won't matter, some people will have to apply one of a few simple fixes, and a rare few will have to rewrite their code in a non-trivial way (sometimes this will affect clever libraries). I wonder if the PEP needs a better transition plan, e.g. - right now, start an education campaign - with Python 3.5, introduce from __future__ import generator_return, and silent deprecation warnings - with Python 3.6, start issuing non-silent deprecation warnings - with Python 3.7, make the new behavior the default (subject to some kind of review) I fear that there is one specific corner case that will be impossible to deal with in a backwards-compatible way supporting both Python 2 and 3 in one code base: the use of `return value` in a generator. In Python 2.x through 3.1, `return value` is a syntax error inside generators. Currently, the only way to handle this case in 2+3 code is by using `raise StopIteration(value)` but if that changes in 3.6 or 3.7 then there will be no (obvious?) way to deal with this case. -- Steven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Fri, Nov 21, 2014 at 8:47 AM, Antoine Pitrou solip...@pitrou.net wrote: On Fri, 21 Nov 2014 05:47:58 -0800 Raymond Hettinger raymond.hettin...@gmail.com wrote: Another issue is that it breaks the way I and others have taught for years that generators are a kind of iterator (an object implementing the iterator protocol) and that a primary motivation for generators is to provide a simpler and more direct way of creating iterators. However, Chris explained that, This proposal causes a separation of generators and iterators, so it's no longer possible to pretend that they're the same thing. That is a major and worrisome conceptual shift. I agree with Raymond on this point. Pretending they're the same thing has always been fraught with subtle errors. From the *calling* side a generator implements the same protocol as any other iterator (though it also has a few others -- send(), throw(), close()). However *inside* they are not at all similar -- generators produce a value is done through yield, __next__() methods use return. Even if we end up rejecting the PEP we should campaign for better understanding of generators. Raymond may just have to fix some of his examples. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Fri, Nov 21, 2014 at 9:18 AM, Steven D'Aprano st...@pearwood.info wrote: I fear that there is one specific corner case that will be impossible to deal with in a backwards-compatible way supporting both Python 2 and 3 in one code base: the use of `return value` in a generator. In Python 2.x through 3.1, `return value` is a syntax error inside generators. Currently, the only way to handle this case in 2+3 code is by using `raise StopIteration(value)` but if that changes in 3.6 or 3.7 then there will be no (obvious?) way to deal with this case. Note that using StopIteration for this purpose is a recent invention (I believe I invented it for the Google App Engine NDB library). Before Python 3.3 this had to be essentially a private protocol implemented by the framework, and typically the framework defines a custom exception for this purpose -- either an alias for StopIteration, or a subclass of it, or a separate exception altogether. I did a little survey: - ndb uses return Return(v) where Return is an alias for StopIteration. - monocle uses yield Return(v), so it doesn't even use an exception. - In Twisted you write returnValue(v) -- IMO even more primitive since it's not even a control flow statement. - In Tornado you write raise tornado.gen.Return(v), where Return does not inherit from StopIteration. In Python 3.3 and later you can also write return v, and the framework treats Return and StopIteration the same -- but there is no mention of raise StopIteration(v) in the docs and given that they have Return there should be no need for it, ever. - In Trollius (the backport of asyncio) you write raise Return(v), where Return is currently a subclass of StopIteration -- but it doesn't really have to be, it could be a different exception (like in Tornado). So I haven't found any framework that recommends raise StopIteration(v). Sure, some frameworks will have to be changed, but they have until Python 3.6 or 3.6, and the changes can be made to work all the way back to Python 2.7. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 20 November 2014 06:48, MRAB pyt...@mrabarnett.plus.com wrote: On 2014-11-19 20:10, Guido van Rossum wrote: There's a new PEP proposing to change how to treat StopIteration bubbling up out of a generator frame (not caused by a return from the frame). The proposal is to replace such a StopIteration with a RuntimeError (chained to the original StopIteration), so that only *returning* from a generator (or falling off the end) causes the iteration to terminate. The PEP says any generator that depends on an implicitly-raised StopIteration to terminate it will have to be rewritten to either catch that exception or use a for-loop Shouldn't that be ... explicitly-raised ..., because returning raises StopIteration implicitly? (raise StopIteration is explicit) The ways a generator can currently be terminated: return [value] raise StopIteration[([value])] any other expression that may raise StopIteration The first case is unchanged, and (as Chris noted), the second case can be trivially converted to the first case. It's the third implicit case which would no longer be allowed, which means no more or stop() tricks to get generator expressions to terminate early. (If folks wanted to restore that capability, they'd instead need to propose new syntax that works the same way in both comprehensions and generator expressions, rather than relying on the lazy evaluation of generator expressions) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 20 November 2014 06:15, Benjamin Peterson benja...@python.org wrote: On Wed, Nov 19, 2014, at 15:10, Guido van Rossum wrote: There's a new PEP proposing to change how to treat StopIteration bubbling up out of a generator frame (not caused by a return from the frame). The proposal is to replace such a StopIteration with a RuntimeError (chained to the original StopIteration), so that only *returning* from a generator (or falling off the end) causes the iteration to terminate. The proposal unifies the behavior of list comprehensions and generator expressions along the lines I had originally in mind when they were introduced. It renders useless/illegal certain hacks that have crept into some folks' arsenal of obfuscated Python tools. In Python 3.5 the proposed change is conditional on: from __future__ import replace_stopiteration_in_generators Drive-by comment: This seems like a terribly awkward name. Could a shorter and sweeter name not be found? I think my suggestion was something like from __future__ import generator_return. I saw that style as somewhat similar to from __future__ import division - it just tells you what the change affects (in this case, returning from generators), while requiring folks to look up the documentation to find out the exact details of the old behaviour and the new behaviour. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
I've made some updates to the PEP: - added 19-Nov-2014 to Post-History - removed implicitly-raised from the abstract - changed the __future__ thing to generator_return - added a clarifying paragraph that Chris added to his own draft - added a link to http://bugs.python.org/issue22906 which has a proof-of-concept patch There's still a lively discussion on python-ideas; Steven D'Aprano has dug up quite a bit of evidence that StopIteration is used quite a bit in ways that will break under the new behavior, and there also seems to be quite a bit of third-party information that recommends StopIteration over return to terminate a generator early. However I don't see much evidence that the current behavior is *better* than the proposal -- I see the current behavior as a definite wart, and I have definitely seen people struggle to debug silent early loop termination due to an escaped StopIteration. That said, I think for most people the change won't matter, some people will have to apply one of a few simple fixes, and a rare few will have to rewrite their code in a non-trivial way (sometimes this will affect clever libraries). I wonder if the PEP needs a better transition plan, e.g. - right now, start an education campaign - with Python 3.5, introduce from __future__ import generator_return, and silent deprecation warnings - with Python 3.6, start issuing non-silent deprecation warnings - with Python 3.7, make the new behavior the default (subject to some kind of review) It would also be useful if we could extend the PEP with some examples of the various categories of fixes that can be applied easily, e.g. a few examples of raise StopIteration directly in a generator that can be replaced with return (or omitted, if it's at the end); a few examples of situations where yield from can supply an elegant fix (and an alternative for code that needs to be backward compatible with Python 3.2 or 2.7); and finally (to be honest) an example of code that will require being made more complicated. Oh, and it would also be nice if the PEP included some suggested words that 3rd party educators can use to explain the relationship between StopIteration and generators in a healthier way (preferably a way that also applies to older versions). Chris, are you up to drafting these additions? On Thu, Nov 20, 2014 at 2:05 AM, Nick Coghlan ncogh...@gmail.com wrote: On 20 November 2014 06:15, Benjamin Peterson benja...@python.org wrote: On Wed, Nov 19, 2014, at 15:10, Guido van Rossum wrote: There's a new PEP proposing to change how to treat StopIteration bubbling up out of a generator frame (not caused by a return from the frame). The proposal is to replace such a StopIteration with a RuntimeError (chained to the original StopIteration), so that only *returning* from a generator (or falling off the end) causes the iteration to terminate. The proposal unifies the behavior of list comprehensions and generator expressions along the lines I had originally in mind when they were introduced. It renders useless/illegal certain hacks that have crept into some folks' arsenal of obfuscated Python tools. In Python 3.5 the proposed change is conditional on: from __future__ import replace_stopiteration_in_generators Drive-by comment: This seems like a terribly awkward name. Could a shorter and sweeter name not be found? I think my suggestion was something like from __future__ import generator_return. I saw that style as somewhat similar to from __future__ import division - it just tells you what the change affects (in this case, returning from generators), while requiring folks to look up the documentation to find out the exact details of the old behaviour and the new behaviour. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Thu, 20 Nov 2014 11:36:54 -0800 Guido van Rossum gu...@python.org wrote: I've made some updates to the PEP: - added 19-Nov-2014 to Post-History - removed implicitly-raised from the abstract - changed the __future__ thing to generator_return To me generator_return sounds like the addition to generator syntax allowing for return statements (which was done as part of the yield from PEP). How about generate_escape? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 20.11.14 21:58, Antoine Pitrou wrote: To me generator_return sounds like the addition to generator syntax allowing for return statements (which was done as part of the yield from PEP). How about generate_escape? Or may be generator_stop_iteration? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Thu, Nov 20, 2014 at 12:13 PM, Serhiy Storchaka storch...@gmail.com wrote: On 20.11.14 21:58, Antoine Pitrou wrote: To me generator_return sounds like the addition to generator syntax allowing for return statements (which was done as part of the yield from PEP). How about generate_escape? Or may be generator_stop_iteration? Or just generator_stop? -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/20/2014 5:04 PM, Guido van Rossum wrote: On Thu, Nov 20, 2014 at 12:13 PM, Serhiy Storchaka storch...@gmail.com mailto:storch...@gmail.com wrote: On 20.11.14 21:58, Antoine Pitrou wrote: To me generator_return sounds like the addition to generator syntax allowing for return statements (which was done as part of the yield from PEP). How about generate_escape? Or may be generator_stop_iteration? Or just generator_stop? +1 About as sort as we could get for the informed user reader and neutrally cryptic so a naive reader will know to look it up rather than guess (wrongly) -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Thu, 20 Nov 2014 14:04:24 -0800 Guido van Rossum gu...@python.org wrote: On Thu, Nov 20, 2014 at 12:13 PM, Serhiy Storchaka storch...@gmail.com wrote: On 20.11.14 21:58, Antoine Pitrou wrote: To me generator_return sounds like the addition to generator syntax allowing for return statements (which was done as part of the yield from PEP). How about generate_escape? Or may be generator_stop_iteration? Or just generator_stop? That sounds good. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 11/20/2014 2:36 PM, Guido van Rossum wrote: There's still a lively discussion on python-ideas; Steven D'Aprano has dug up quite a bit of evidence that StopIteration is used quite a bit in ways that will break under the new behavior, and there also seems to be quite a bit of third-party information that recommends StopIteration over return to terminate a generator early. However I don't see much evidence that the current behavior is *better* than the proposal -- I see the current behavior as a definite wart, and I have definitely seen people struggle to debug silent early loop termination due to an escaped StopIteration. That said, I think for most people the change won't matter, some people will have to apply one of a few simple fixes, and a rare few will have to rewrite their code in a non-trivial way (sometimes this will affect clever libraries). I wonder if the PEP needs a better transition plan, e.g. - right now, start an education campaign For StackOverflow, someone with a high-enough rep (you, Guido?) should add a tag for StopIteration with text like The Python `StopIteration` exception. (Copied from the current AttributeError tag, typeerror and NameError also exist.). If and when the PEP is accepted, I would be willing to add comments and the new tag to existing and future questions/answers. - with Python 3.5, introduce from __future__ import generator_return, and silent deprecation warnings - with Python 3.6, start issuing non-silent deprecation warnings I agree that two deprecation periods are needed. - with Python 3.7, make the new behavior the default (subject to some kind of review) It would also be useful if we could extend the PEP with some examples of the various categories of fixes that can be applied easily, e.g. a few examples of raise StopIteration directly in a generator that can be replaced with return (or omitted, if it's at the end); a few examples of situations where yield from can supply an elegant fix (and an alternative for code that needs to be backward compatible with Python 3.2 or 2.7); and finally (to be honest) an example of code that will require being made more complicated. Agree that explicit fixes are needed. Deprecation message could refer to section of PEP. Oh, and it would also be nice if the PEP included some suggested words that 3rd party educators can use to explain the relationship between StopIteration and generators in a healthier way (preferably a way that also applies to older versions). Chris, are you up to drafting these additions? -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Fri, Nov 21, 2014 at 6:36 AM, Guido van Rossum gu...@python.org wrote: It would also be useful if we could extend the PEP with some examples of the various categories of fixes that can be applied easily, e.g. a few examples of raise StopIteration directly in a generator that can be replaced with return (or omitted, if it's at the end); a few examples of situations where yield from can supply an elegant fix (and an alternative for code that needs to be backward compatible with Python 3.2 or 2.7); and finally (to be honest) an example of code that will require being made more complicated. Oh, and it would also be nice if the PEP included some suggested words that 3rd party educators can use to explain the relationship between StopIteration and generators in a healthier way (preferably a way that also applies to older versions). Chris, are you up to drafting these additions? Sure, no problem. Will knock something up shortly. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Fri, Nov 21, 2014 at 9:04 AM, Guido van Rossum gu...@python.org wrote: On Thu, Nov 20, 2014 at 12:13 PM, Serhiy Storchaka storch...@gmail.com wrote: On 20.11.14 21:58, Antoine Pitrou wrote: To me generator_return sounds like the addition to generator syntax allowing for return statements (which was done as part of the yield from PEP). How about generate_escape? Or may be generator_stop_iteration? Or just generator_stop? Unrelated to the GeneratorExit exception. I don't think that'll be a problem though. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Thu, Nov 20, 2014 at 3:13 PM, Antoine Pitrou solip...@pitrou.net wrote: On Thu, 20 Nov 2014 14:04:24 -0800 Guido van Rossum gu...@python.org wrote: On Thu, Nov 20, 2014 at 12:13 PM, Serhiy Storchaka storch...@gmail.com wrote: On 20.11.14 21:58, Antoine Pitrou wrote: To me generator_return sounds like the addition to generator syntax allowing for return statements (which was done as part of the yield from PEP). How about generate_escape? Or may be generator_stop_iteration? Or just generator_stop? That sounds good. OK, updated the PEP. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Fri, Nov 21, 2014 at 10:34 AM, Chris Angelico ros...@gmail.com wrote: On Fri, Nov 21, 2014 at 6:36 AM, Guido van Rossum gu...@python.org wrote: It would also be useful if we could extend the PEP with some examples of the various categories of fixes that can be applied easily, e.g. a few examples of raise StopIteration directly in a generator that can be replaced with return (or omitted, if it's at the end); a few examples of situations where yield from can supply an elegant fix (and an alternative for code that needs to be backward compatible with Python 3.2 or 2.7); and finally (to be honest) an example of code that will require being made more complicated. Oh, and it would also be nice if the PEP included some suggested words that 3rd party educators can use to explain the relationship between StopIteration and generators in a healthier way (preferably a way that also applies to older versions). Chris, are you up to drafting these additions? Sure, no problem. Will knock something up shortly. Examples and explanatory text added. https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Wed, Nov 19, 2014, at 15:10, Guido van Rossum wrote: There's a new PEP proposing to change how to treat StopIteration bubbling up out of a generator frame (not caused by a return from the frame). The proposal is to replace such a StopIteration with a RuntimeError (chained to the original StopIteration), so that only *returning* from a generator (or falling off the end) causes the iteration to terminate. The proposal unifies the behavior of list comprehensions and generator expressions along the lines I had originally in mind when they were introduced. It renders useless/illegal certain hacks that have crept into some folks' arsenal of obfuscated Python tools. In Python 3.5 the proposed change is conditional on: from __future__ import replace_stopiteration_in_generators Drive-by comment: This seems like a terribly awkward name. Could a shorter and sweeter name not be found? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On 2014-11-19 20:10, Guido van Rossum wrote: There's a new PEP proposing to change how to treat StopIteration bubbling up out of a generator frame (not caused by a return from the frame). The proposal is to replace such a StopIteration with a RuntimeError (chained to the original StopIteration), so that only *returning* from a generator (or falling off the end) causes the iteration to terminate. The PEP says any generator that depends on an implicitly-raised StopIteration to terminate it will have to be rewritten to either catch that exception or use a for-loop Shouldn't that be ... explicitly-raised ..., because returning raises StopIteration implicitly? (raise StopIteration is explicit) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators
On Thu, Nov 20, 2014 at 7:48 AM, MRAB pyt...@mrabarnett.plus.com wrote: The PEP says any generator that depends on an implicitly-raised StopIteration to terminate it will have to be rewritten to either catch that exception or use a for-loop Shouldn't that be ... explicitly-raised ..., because returning raises StopIteration implicitly? (raise StopIteration is explicit) The point here is primarily about some other function (maybe a next(iter), or maybe something else entirely) raising StopIteration. (If it explicitly raises StopIteration right there in the generator, it can be trivially converted into a return statement, anyway.) The return statement is an explicit indication that the generator should now return; permitting a StopIteration to bubble up through and out is the implicit option; but the 'plicitness' isn't necessarily obvious. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
