[issue17468] Generator memory leak

Vikas added the comment: I see this comment : >> When iteration over a queryset raised an exception, the result cache >> remained initialized with an empty list, so subsequent iterations returned >> an empty list instead of raising an exception>> What if we catch the exceptions? Will that caus

[issue17468] Generator memory leak

Antoine Pitrou added the comment: The issue17807 patch has now been committed and should solve this issue. I would like to encourage anyone to test this change, as it is quite a significant departure from the previous semantics: http://mail.python.org/pipermail/python-dev/2013-May/126102.html

[issue17468] Generator memory leak

Antoine Pitrou added the comment: I've opened issue17807 for an alternative generator cleanup scheme which solves the present issue. -- ___ Python tracker ___ __

[issue17468] Generator memory leak

Antoine Pitrou added the comment: > We *don't care* if the generator *would* have caught the thrown > GeneratorExit, we only care about ensuring that finally blocks are > executed (including those implied by with statements). So if there > aren't any finally clauses or with statements in the bloc

[issue17468] Generator memory leak

Nick Coghlan added the comment: To get back to Anssi's original suggestion... I think Anssi's proposal to allow finalisation to be skipped for try/except/else is legitimate. It's only finally clauses that we try to guarantee will execute, there's no such promise implied for ordinary except cla

[issue17468] Generator memory leak

Nick Coghlan added the comment: We can't make ordinary generators innately context managers, as it makes the error too hard to detect when you accidentally leave out @contextmanager when using a generator to write a custom one. You can already use contextlib.closing to forcibly close them when

[issue17468] Generator memory leak

Benjamin Peterson added the comment: I realize, but if people were responsible and closed their generators, the second one would be as much of a problem. -- ___ Python tracker _

[issue17468] Generator memory leak

Antoine Pitrou added the comment: Those are two different issues: - not calling the finalizer in a timely manner - never calling the finalizer and creating a memory leak through gc.garbage -- ___ Python tracker ___

[issue17468] Generator memory leak

Benjamin Peterson added the comment: In a sense, doing something like "with self.lock" in a generator is already a leak. Even if there wasn't a cycle, collection could be arbitrarily delayed (in partincular on non-CPython VMs). I wonder if making generators context managers which call close()

[issue17468] Generator memory leak

Anssi Kääriäinen added the comment: I was imagining that the collection should happen in two passes. First check for tp_dels and call them if safe (single tp_del in cycle allowed). Then free the memory. The first pass is already there, it just doesn't collect callable tp_dels. If it would be

[issue17468] Generator memory leak

Antoine Pitrou added the comment: > I wonder if it would be better to reword the garbage collection docs > to mention that Python can't collect objects if they are part of a > reference cycle, and some of the objects in the reference cycle need > to run code at gc time. Then mention that such obj

[issue17468] Generator memory leak

Anssi Kääriäinen added the comment: I wonder if it would be better to reword the garbage collection docs to mention that Python can't collect objects if they are part of a reference cycle, and some of the objects in the reference cycle need to run code at gc time. Then mention that such object

[issue17468] Generator memory leak

Nick Coghlan added the comment: Issue 17800 is anyone wants to weigh in on the tp_del -> __del__ question (I ended up not adding Guido back to that one either, since the original design was based on an assumption that's now demonstrably false, so it makes sense to update the behaviour) --

[issue17468] Generator memory leak

Nick Coghlan added the comment: I'll create a separate issue for the tp_del -> __del__ question, since that's a language design decision that *does* need Guido's input, but doesn't relate to the broader question of generators, cycles and potential memory leaks. --

[issue17468] Generator memory leak

Changes by Benjamin Peterson : -- nosy: -gvanrossum ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mai

[issue17468] Generator memory leak

Guido van Rossum added the comment: Unless I'm the only person on earth who can understand this I beg to be left out of this bug. I just don't have the bandwidth right now to look into it. If it's really about GC and __del__ and generators, maybe Tim Peters would be willing to look into it? Or

[issue17468] Generator memory leak

Changes by Martin Morrison : -- nosy: +pconnell ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyt

[issue17468] Generator memory leak

Antoine Pitrou added the comment: > Yes. Hopefully, the async framework using generators can properly can > close() on them, though. With yield from-based generators, you don't need a trampoline anymore, so the close() call is now left to the application developers (if I'm not mistaken). --

[issue17468] Generator memory leak

Benjamin Peterson added the comment: Yes. Hopefully, the async framework using generators can properly can close() on them, though. -- ___ Python tracker ___ ___

[issue17468] Generator memory leak

Benjamin Peterson added the comment: I think the creation of __del__ wrappers for extension types is separate from this issue of generator finalization. -- ___ Python tracker __

[issue17468] Generator memory leak

Antoine Pitrou added the comment: With the advent of yield-based asynchronous programming, it is going to be problematic if a generator caught in a reference cycle can create a memory leak. -- nosy: +pitrou ___ Python tracker

[issue17468] Generator memory leak

Nick Coghlan added the comment: Adding Guido because this appears to be due to a longstanding difference between the handling of tp_del and most other slots Specifically, the reason for the odd behaviour appears to be that generator objects define tp_del [1] (which is what the cyclic gc *reall

[issue17468] Generator memory leak

Benjamin Peterson added the comment: A documentation patch sounds good to me. -- assignee: -> docs@python components: +Documentation nosy: +docs@python type: resource usage -> behavior ___ Python tracker _

[issue17468] Generator memory leak

Anssi Kääriäinen added the comment: True, except GeneratorExit will run at garbage collection time and this will result in reference cycle problems. Checking if there is no except GeneratorExit clause might be too complicated. I still think this is worth a brief note in the gc docs. The gc doc

[issue17468] Generator memory leak

Benjamin Peterson added the comment: Generators exit using GeneratorExit, which you could possible catch. -- ___ Python tracker ___ __

[issue17468] Generator memory leak

Changes by Antoine Pitrou : -- nosy: +benjamin.peterson, ncoghlan ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe

[issue17468] Generator memory leak

Changes by Martin Morrison : -- nosy: +isoschiz ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyt

[issue17468] Generator memory leak

Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue17468] Generator memory leak

Anssi Kääriäinen added the comment: I am trying to read the code, and it seems objects of type generator are uncollectable if the code of the generator has a block of other type than SETUP_LOOP. I can see how try-finally for example in the generator would cause problems - the finally might ref

[issue17468] Generator memory leak

Amaury Forgeot d'Arc added the comment: Well, there is a __del__: the one from the generator function. And it's part of the cycle. -- nosy: +amaury.forgeotdarc ___ Python tracker __

[issue17468] Generator memory leak

New submission from Anssi Kääriäinen: A generator is leaked to gc.garbage in a situation where `__del__` isn't defined. See the attached file for as-minimalistic test case as I could make. Tested on Python 3.3.0, 3.2.3 and 2.7.3. Note that if the try-except is removed from iterator(), then the