[issue17468] Generator memory leak
Anssi Kääriäinen added the comment: I wonder if it would be better to reword the garbage collection docs to mention that Python can't collect objects if they are part of a reference cycle, and some of the objects in the reference cycle need to run code at gc time. Then mention that such objects include objects with __del__ and also generators if they do have some other blocks than loops in them (for example try or with blocks). To me it seems this issue could be mitigated somewhat by collecting the objects if there is just one object with finalizer code in the cycle. It should be safe to run the single finalizer, then collect the whole cycle, right? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17468 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17468] Generator memory leak
Anssi Kääriäinen added the comment: I was imagining that the collection should happen in two passes. First check for tp_dels and call them if safe (single tp_del in cycle allowed). Then free the memory. The first pass is already there, it just doesn't collect callable tp_dels. If it would be possible to turn an object into inaccessible state after tp_del was called it would be possible to call multiple tp_dels in a cycle. If the del tries to use already deleted object you will get some sort of runtime exception indicating access of already collected object (another option is to allow access to object which has already had __del__ called, but that seems problematic). Now, both of the above are likely way more complicated to implement than what I imagine. I have very little knowledge of the technical details involved. If I understand correctly your idea was to do something similar to above, but only for generators. The current situation is somewhat ugly. First, I can imagine people wishing to do try-finally or something with self.lock: in a generator. These will leak in circular reference cases. The generator case is surprising, so surprising that even experienced programmers will do such mistakes (see the django ticket for one example). Second, from application programmers perspective the fact that __del__ wasn't called is usually similar to having a silent failure in their code - for example this can result in leaking database connections or other resources, not to mention possible problems if one has a with self.lock: block in a generator. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17468 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17468] Generator memory leak
Anssi Kääriäinen added the comment: True, except GeneratorExit will run at garbage collection time and this will result in reference cycle problems. Checking if there is no except GeneratorExit clause might be too complicated. I still think this is worth a brief note in the gc docs. The gc docs talk only about __del__ methods, and there isn't one defined for the example generator. The generator does have something technically equivalent, but I don't spot any actual __del__ methods in the reference loop. A docs mention will not help much in avoiding this problem, but it will at least make the code behaviour equivalent to documentation. A mention in the gc docs would have helped me when trying to debug the leak in Django. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17468 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17468] Generator memory leak
Anssi Kääriäinen added the comment: I am trying to read the code, and it seems objects of type generator are uncollectable if the code of the generator has a block of other type than SETUP_LOOP. I can see how try-finally for example in the generator would cause problems - the finally might reference objects which are already collected. But plain try-except? What is the problem with that? (Not saying there isn't one, I am new to the code...). I tried to change the code to also allow collecting generators with plain try-except (add in a check for SETUP_EXCEPTION in addition to SETUP_LOOP in PyGen_NeedsFinalizing()). This resolves the leak, and all tests pass. I can't tell if it is actually safe to return 0 in SETUP_EXCEPTION case. After the change try-finally will still leak but that seems correct. Maybe this limitation should be documented? I think I understand now why finally or with statement inside a cyclic referenced generator will end up in gc.garbage. However the docs tell that only objects with __del__ defined will leak. From user perspective generator doesn't have __del__, at least it is not to be found with pdb. Generators have potentially unsafe finalizer code but that isn't technically equivalent to having __del__ method defined as far as I can tell. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17468 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17468] Generator memory leak
New submission from Anssi Kääriäinen: A generator is leaked to gc.garbage in a situation where `__del__` isn't defined. See the attached file for as-minimalistic test case as I could make. Tested on Python 3.3.0, 3.2.3 and 2.7.3. Note that if the try-except is removed from iterator(), then there is no leak. This is related to Django bug #19895 (https://code.djangoproject.com/ticket/19895). -- files: test.py messages: 184515 nosy: Anssi.Kääriäinen priority: normal severity: normal status: open title: Generator memory leak type: resource usage versions: Python 2.7, Python 3.3 Added file: http://bugs.python.org/file29455/test.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17468 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
