Re: Different behaviour in list comps and generator expressions
Steven D'Aprano wrote: The following list comprehension and generator expression are almost, but not quite, the same: [expr for x in iterable] list(expr for x in iterable) The difference is in the handling of StopIteration raised inside the expr. [...] Thanks to Roy and Wolfgang for their comments. If anyone is interested, Guido will soon be ruling on a PEP which will change the behaviour of generators and generator expressions, and he has asked for feedback. In particular, examples of code which will be broken by this change. Read the PEP here: https://www.python.org/dev/peps/pep-0479 If you post comments here, I will see that they get forwarded on. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Different behaviour in list comps and generator expressions
On Mon, Nov 17, 2014 at 3:30 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: The following list comprehension and generator expression are almost, but not quite, the same: [expr for x in iterable] list(expr for x in iterable) The difference is in the handling of StopIteration raised inside the expr. [...] Thanks to Roy and Wolfgang for their comments. If anyone is interested, Guido will soon be ruling on a PEP which will change the behaviour of generators and generator expressions, and he has asked for feedback. In particular, examples of code which will be broken by this change. Read the PEP here: https://www.python.org/dev/peps/pep-0479 If you post comments here, I will see that they get forwarded on. I'm inclined to say that list comprehensions and generator expressions should be different. I don't really think they should be identical, one being eager and one being lazy. Why let the implementation detail of one impact the other? -- https://mail.python.org/mailman/listinfo/python-list
Re: Different behaviour in list comps and generator expressions
On 11/17/2014 03:38 PM, Dan Stromberg wrote: I'm inclined to say that list comprehensions and generator expressions should be different. I don't really think they should be identical, one being eager and one being lazy. Why let the implementation detail of one impact the other? It's not the eagerness vs. laziness that's being changed, but rather what happens in a generator when something inside the generator raises StopIteration (as opposed to the generator itself simply return'ing and thereby causing a StopIteration to be generated). -- ~Ethan~ signature.asc Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: Different behaviour in list comps and generator expressions
On 08.11.2014 02:50, Steven D'Aprano wrote: The following list comprehension and generator expression are almost, but not quite, the same: [expr for x in iterable] list(expr for x in iterable) The difference is in the handling of StopIteration raised inside the expr. Generator expressions consume them and halt, while comprehensions allow them to leak out. This is not the right description of what's happening. It is not the generator expression that consumes it, but the list constructor and, of course, list can't tell at which level in the inside code StopIteration got raised. So, yes this had me confused some times, too, but it is really not surprising. A simple example: iterable = [iter([])] list(next(x) for x in iterable) = returns [] But: [next(x) for x in iterable] = raises StopIteration Yes, but the equivalent to list(next(x) for x in iterable) is: l = [] for item in (next(x) for x in iterable): l.append(item) = returns [] because the for consumes the StopIteration this, on the other hand raises StopIteration l = [] for item in [next(x) for x in iterable]: l.append(item) because the error gets raised already when for causes iter() to be called on the list comprehension. Has anyone come across this difference in the wild? Was it a problem? Do you rely on that difference, or is it a nuisance? Has it caused difficulty in debugging code? If you had to keep one behaviour, which would you keep? The point is: there is no difference in the behavior of comprehensions and generator expressions, it is just how you access them: for anything that follows the iterator protocol, a comprehension (since it is evaluated immediately) raises during the iter() phase, while a generator expression (which gets evaluated lazily) raises during the next() phase where it gets swallowed. So in your example you would have to use the silly list([next(x) for x in iterable]) if you want the error to get raised. I agree this is all rather non-intuitive, but how would you change it ? -- https://mail.python.org/mailman/listinfo/python-list
Re: Different behaviour in list comps and generator expressions
On 08.11.2014 22:31, Wolfgang Maier wrote: On 08.11.2014 02:50, Steven D'Aprano wrote: The following list comprehension and generator expression are almost, but not quite, the same: [expr for x in iterable] list(expr for x in iterable) The difference is in the handling of StopIteration raised inside the expr. Generator expressions consume them and halt, while comprehensions allow them to leak out. This is not the right description of what's happening. It is not the generator expression that consumes it, but the list constructor and, of course, list can't tell at which level in the inside code StopIteration got raised. So, yes this had me confused some times, too, but it is really not surprising. A simple example: iterable = [iter([])] list(next(x) for x in iterable) = returns [] But: [next(x) for x in iterable] = raises StopIteration Yes, but the equivalent to list(next(x) for x in iterable) is: l = [] for item in (next(x) for x in iterable): l.append(item) = returns [] because the for consumes the StopIteration this, on the other hand raises StopIteration l = [] for item in [next(x) for x in iterable]: l.append(item) because the error gets raised already when for causes iter() to be called on the list comprehension. Has anyone come across this difference in the wild? Was it a problem? Do you rely on that difference, or is it a nuisance? Has it caused difficulty in debugging code? If you had to keep one behaviour, which would you keep? The point is: there is no difference in the behavior of comprehensions and generator expressions, it is just how you access them: for anything that follows the iterator protocol, a comprehension (since it is evaluated immediately) raises during the iter() phase, while a generator expression (which gets evaluated lazily) raises during the next() phase where it gets swallowed. So in your example you would have to use the silly list([next(x) for x in iterable]) if you want the error to get raised. I agree this is all rather non-intuitive, but how would you change it ? Ah, I came across the related thread on python-ideas only now and from this I can see that, as I expected, you know everything I've written above already. In light of the discussion on the other list: I did find it annoying occasionally that raising StopIteration inside a generator expression conveys a different behavior than elsewhere. It did take me quite a while to understand why that is so, but after that it did not cause me much of a headache anymore. I would say that Guido's suggestion of transforming StopIteration raised inside a generator into some other error to eliminate ambiguity would really help in some situations, but then I'm not sure whether that's worth breaking existing code. Best, Wolfgang -- https://mail.python.org/mailman/listinfo/python-list
Different behaviour in list comps and generator expressions
The following list comprehension and generator expression are almost, but not quite, the same: [expr for x in iterable] list(expr for x in iterable) The difference is in the handling of StopIteration raised inside the expr. Generator expressions consume them and halt, while comprehensions allow them to leak out. A simple example: iterable = [iter([])] list(next(x) for x in iterable) = returns [] But: [next(x) for x in iterable] = raises StopIteration Has anyone come across this difference in the wild? Was it a problem? Do you rely on that difference, or is it a nuisance? Has it caused difficulty in debugging code? If you had to keep one behaviour, which would you keep? -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Different behaviour in list comps and generator expressions
In article 545d76fe$0$12980$c3e8da3$54964...@news.astraweb.com, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: The following list comprehension and generator expression are almost, but not quite, the same: [expr for x in iterable] list(expr for x in iterable) The difference is in the handling of StopIteration raised inside the expr. Generator expressions consume them and halt, while comprehensions allow them to leak out. A simple example: iterable = [iter([])] list(next(x) for x in iterable) = returns [] But: [next(x) for x in iterable] = raises StopIteration Has anyone come across this difference in the wild? Was it a problem? Do you rely on that difference, or is it a nuisance? Has it caused difficulty in debugging code? If you had to keep one behaviour, which would you keep? Wow, that's really esoteric. I can't imagine this happening in real-life code (but I'm sure somebody will come up with an example :-)) My inclination is that a list comprehension should stop if StopIteration is raised by the comprehension body. I can't come up with a good argument to support that, other than it seems like the right thing to do. -- https://mail.python.org/mailman/listinfo/python-list
