Re: [Python-Dev] problem with recursive yield from delegation
Stefan Behnel wrote: Hi, I found a problem in the current yield from implementation that I think is worth discussing: http://bugs.python.org/issue14220 [snip] I've been experimenting with the implementation of PEP 380, and I found a couple of interesting things. First of all, the semantics described in the PEP do not match the tests. If you substitute the supposedly semantically equivalent code based on normal yields for each yield from in the test code (Lib/test/test_pep380.py) and run it, then it fails. My second experiment involved stripping away all the code relating to yield from outside of the interpreter and changing the YIELD_FROM bytecode to repeat itself, by setting the last instruction to the instruction immediately before itself. To do this I added 4 lines of code and removed over 120 lines :) This fails many of the tests*, but works for the most straightforward use cases. Many of these failures seem to me to be more 'natural' than the current behaviour It might be possible to fix most, or all?, of the other failures by compiling yield from into a two opcode sequence: YIELD_FROM_START and YIELD_FROM_REPEAT. Both opcodes should be reasonably simple and __next__() and send() would not have to worry about the subiterator, although close() and throw() might (the subiterator would be top-of-stack in the generator's frame) Overall the semantics of PEP 380 seem far too complex, trying to do several things at once. An example: Plain yield makes no distinction between a receiving a None and any other value, so send(None) and __next__() are the same. Yield from makes this distinction so has to test for None, meaning the semantics of send now changes with its argument. I would recommend changing one of two things in the PEP: Either, close and throw should not close/throw in subiterators (this would simplify the semantics and implementation immensely) Or, only allow subgenerators, not subiterators (this would fix the next/send problem). I would also suggest a change in implementation to perform all yields within the interpreter. A simpler implementation is likely to be a more reliable one. Finally, the PEP itself makes no mention of coroutines, stackless or greenlet in the alternatives or criticisms section. Perhaps it should. Cheers, Mark. *Tests in PEP 380. It passes all other tests. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
On Thu, Mar 8, 2012 at 9:52 PM, Mark Shannon m...@hotpy.org wrote: I would recommend changing one of two things in the PEP: Either, close and throw should not close/throw in subiterators (this would simplify the semantics and implementation immensely) Or, only allow subgenerators, not subiterators (this would fix the next/send problem). Either of those changes would completely defeat the point of the PEP. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
On Thu, Mar 8, 2012 at 9:52 PM, Mark Shannon m...@hotpy.org wrote: First of all, the semantics described in the PEP do not match the tests. If you substitute the supposedly semantically equivalent code based on normal yields for each yield from in the test code (Lib/test/test_pep380.py) and run it, then it fails. What's more important is whether or not it matches the semantics of inlining the subgenerator bodies. The expansion in the PEP was an attempt to define a way to achieve that in current Python without interpreter support. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
Nick Coghlan wrote: On Thu, Mar 8, 2012 at 9:52 PM, Mark Shannon m...@hotpy.org wrote: First of all, the semantics described in the PEP do not match the tests. If you substitute the supposedly semantically equivalent code based on normal yields for each yield from in the test code (Lib/test/test_pep380.py) and run it, then it fails. What's more important is whether or not it matches the semantics of inlining the subgenerator bodies. The expansion in the PEP was an attempt to define a way to achieve that in current Python without interpreter support. So yield from X means inline X here, if X is a generator. That is much, much easier to understand than the big block of code in the PEP. It really ought say that yield from is equivalent to inlining in the PEP. Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
On Thu, Mar 8, 2012 at 11:45 PM, Mark Shannon m...@hotpy.org wrote: It really ought say that yield from is equivalent to inlining in the PEP. That's what the motivation section is about. There's also an entire subsection called The Refactoring Principle that describes this intent. However, we needed something more concrete to flesh out the original implementation details, which is what the code equivalent was designed to provide (it was also designed to point out that the days of you can just use a simple for loop actually went away when PEP 342 was implemented). Now, it may be that we fixed things during implementation that should be reflected back into the formal semantic spec in the PEP - so if you can point out specific cases where what we implemented doesn't match the nominal behaviour, I'm open to updating the PEP accordingly. (Of course, if there are any tests that fail solely due to the two known differences in semantics that are already noted in the PEP, they don't count). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
Stefan Behnel, 07.03.2012 21:40:
I found a problem in the current yield from implementation
... and here's another one, also in genobject.c:
int
PyGen_FetchStopIterationValue(PyObject **pvalue) {
PyObject *et, *ev, *tb;
PyObject *value = NULL;
if (PyErr_ExceptionMatches(PyExc_StopIteration)) {
PyErr_Fetch(et, ev, tb);
Py_XDECREF(et);
Py_XDECREF(tb);
if (ev) {
value = ((PyStopIterationObject *)ev)-value;
Py_INCREF(value);
Py_DECREF(ev);
}
} else if (PyErr_Occurred()) {
return -1;
}
if (value == NULL) {
value = Py_None;
Py_INCREF(value);
}
*pvalue = value;
return 0;
}
When the StopIteration was set using PyErr_SetObject(), ev points to the
value, not the exception instance, so this code lacks exception
normalisation. I use slightly different code in Cython (which needs to be
compatible with Py2.x), but CPython 3.3 could do something along these lines:
if (ev) {
if (PyObject_IsInstance(ev, PyExc_StopIteration)) {
value = ((PyStopIterationObject *)ev)-value;
Py_INCREF(value); // or maybe XINCREF()?
Py_DECREF(ev);
} else {
/* PyErr_SetObject() puts the value directly into ev */
value = ev;
}
} else ...
Would that be acceptable for CPython as well or would you prefer full
fledged normalisation?
Stefan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
2012/3/8 Stefan Behnel stefan...@behnel.de: Would that be acceptable for CPython as well or would you prefer full fledged normalisation? I think we have to normalize for correctness. Consider that it may be some StopIteration subclass which set value on construction. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
On Thu, 8 Mar 2012 14:36:06 -0600 Benjamin Peterson benja...@python.org wrote: 2012/3/8 Stefan Behnel stefan...@behnel.de: Would that be acceptable for CPython as well or would you prefer full fledged normalisation? I think we have to normalize for correctness. Consider that it may be some StopIteration subclass which set value on construction. Perhaps it would be time to drop the whole delayed normalization thing, provided the benchmarks don't exhibit a slowdown? It complicates a lot of code paths. cheers Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
2012/3/8 Antoine Pitrou solip...@pitrou.net: On Thu, 8 Mar 2012 14:36:06 -0600 Benjamin Peterson benja...@python.org wrote: 2012/3/8 Stefan Behnel stefan...@behnel.de: Would that be acceptable for CPython as well or would you prefer full fledged normalisation? I think we have to normalize for correctness. Consider that it may be some StopIteration subclass which set value on construction. Perhaps it would be time to drop the whole delayed normalization thing, provided the benchmarks don't exhibit a slowdown? It complicates a lot of code paths. +1 Also, it delays errors from exception initialization to arbitrary points. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] problem with recursive yield from delegation
Hi, I found a problem in the current yield from implementation that I think is worth discussing: http://bugs.python.org/issue14220 Test code: def g1(): yield y1 yield from g2() yield y4 def g2(): yield y2 try: yield from gi except ValueError: pass # catch already running error yield y3 gi = g1() for y in gi: print(Yielded: %s % (y,)) This is what it currently does: 1) g1() delegates to a new g2(), propagates its y2 value and asks for the next value 2) g2 delegates back to the g1 instance and asks for its next value 3) Python sees the active delegation in g1 and asks g2 for its next value 4) g2 sees that it's already running and throws an exception Ok so far. Now: 5) the exception is propagated into g1 at call level 3) instead of the original requestor g2 one level above 6) g1 undelegates and terminates by the exception 7) g2 catches the exception, yields y3 and then terminates normally 8) g1 gets control back but has already terminated and does nothing Effect: g1 does not yield y4 anymore. The problem is in steps 5) and 6), which are handled by g1 at the wrong call level. They shouldn't lead to undelegation and termination in g1, just to an exception being raised in g2. I ran into this while trying to adapt the implementation for Cython, which has a different generator type implementation but otherwise uses more or less the same code now. But I'm not sure how to fix this one without major changes to the implementation, especially not without special casing the generator type on delegation (which won't work because CPython doesn't know about Cython generators). Any ideas? Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
2012/3/7 Stefan Behnel stefan...@behnel.de: The problem is in steps 5) and 6), which are handled by g1 at the wrong call level. They shouldn't lead to undelegation and termination in g1, just to an exception being raised in g2. That looks wrong indeed. I ran into this while trying to adapt the implementation for Cython, which has a different generator type implementation but otherwise uses more or less the same code now. But I'm not sure how to fix this one without major changes to the implementation Cython's or CPython's implementation? -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
On Thu, Mar 8, 2012 at 6:40 AM, Stefan Behnel stefan...@behnel.de wrote: I ran into this while trying to adapt the implementation for Cython, which has a different generator type implementation but otherwise uses more or less the same code now. But I'm not sure how to fix this one without major changes to the implementation, especially not without special casing the generator type on delegation (which won't work because CPython doesn't know about Cython generators). Any ideas? After tinkering with it a bit, a couple of my original guesses as to the underlying problem were clearly wrong. I'm moving house next week, so it'll be a while before I get to look at it in detail, but I added Mark Shannon to the issue's nosy list. He's been working on a few patches lately to clean up generator related state handling in general, so he may have some insight into this. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
2012/3/7 Benjamin Peterson benja...@python.org: 2012/3/7 Stefan Behnel stefan...@behnel.de: The problem is in steps 5) and 6), which are handled by g1 at the wrong call level. They shouldn't lead to undelegation and termination in g1, just to an exception being raised in g2. That looks wrong indeed. Fixed as of 3357eac1ba62 -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
On Thu, Mar 8, 2012 at 10:00 AM, Benjamin Peterson benja...@python.org wrote: 2012/3/7 Benjamin Peterson benja...@python.org: 2012/3/7 Stefan Behnel stefan...@behnel.de: The problem is in steps 5) and 6), which are handled by g1 at the wrong call level. They shouldn't lead to undelegation and termination in g1, just to an exception being raised in g2. That looks wrong indeed. Fixed as of 3357eac1ba62 Thanks. And, since the fix was entirely internal to the generator implementation, Stefan should be right for the Cython generators, too. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] problem with recursive yield from delegation
http://mail.python.org/pipermail/python-dev/2012-March/117396.html Stefan Behnel posted: I found a problem in the current yield from implementation ... [paraphrasing] g1 yields from g2 g2 yields from g1 XXX python follows the existing delegation without checking re-entrancy g2 (2nd call) checks re-entrancy, and raises an exception g1 (2nd call) gets to handle the exception, and doesn't g2 (1st call) gets to handle the exception, and does How is this a problem? Re-entering a generator is a bug. Python caught it and raised an appropriate exception. It would be nice if python caught the generator cycle as soon as it was created, just as it would be nice if reference cycles were collected as soon as they became garbage. But python doesn't promise to catch cycles immediately, and the checks required to do so would slow down all code, so in practice the checks are delayed. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] problem with recursive yield from delegation
On Thu, Mar 8, 2012 at 10:32 AM, Jim J. Jewett jimjjew...@gmail.com wrote: How is this a problem? Re-entering a generator is a bug. Python caught it and raised an appropriate exception. No, the problem was that the interpreter screwed up the state of the generators while attempting to deal with the erroneous reentry. The ValueError *should* just be caught and completely suppressed by the try/except block, but that wasn't quite happening properly - the failed attempt at reentry left the generators in a dodgy state (which is why the subsequent 3 was being produced, but then the expected final 4 vanished into the electronic ether). Benjamin figured out where the generator's reentrancy check was going wrong, so Stefan's example should do the right thing in the next alpha (i.e. the ValueError will still get raised and suppressed by the try/except block, the inner generator will complete, but the outer generator will also continue on to produce the final value). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
