This problem is caused by caching GeneratorExit exception:
https://bitbucket.org/pypy/pypy/src/8b43b50fbf61f46701d398bf514cf540201ffd03/pypy/interpreter/generator.py?at=py3.5&fileviewer=file-view-default#generator.py-454:457
In Python 3, an exception saves its traceback in __traceback__ attribute. When
an exception object with __traceback__ is raised again, current frame is
appended to the __traceback__. When the GeneratorExit is raised inside a
iterator, the __traceback__ attribute of this object saves the frame, which
prevents the iterator from been collected by GC.
Since __traceback__ attribute is mutable, it is generally a dangerous idea to
reuse an exception object in Python 3, the problem includes: creating memory
leaks with frames, wrong traceback, etc. I think remove the caching will solve
the problem.
It's quite easy to be validated with the following script:
def test():
try:
yield 1
except BaseException as e:
global ex
ex = e
for _ in range(10):
t = test()
next(t)
t.close()
import traceback
traceback.print_tb(ex.__traceback__)
You will see many frames in the traceback.
Removing the __traceback__ attribute before each usage is not acceptable,
because the exception may be used concurrently in different threads, or nested
in a finalizer when GC is triggered inside the generator close.
2018-06-28
hubo
发件人："hubo" <h...@jiedaibao.com>
发送时间：2018-06-28 13:06
主题：[pypy-dev] Memory leaking with iterator.close() in PyPy3
收件人："PyPy Developer Mailing List"<pypy-dev@python.org>
抄送：
In PyPy3, when an iterator is closed with "close()" method, the iterator leaks
and cannot be collected.
Execute the following script in PyPy3, the memory usage is increasing very
fast, and gc.collect() cannot collect the memory
def test():
yield 1
while True:
t = test()
t.close()
The tested version:
Python 3.5.3 (fdd60ed87e94, Apr 24 2018, 06:10:04)
[PyPy 6.0.0 with GCC 6.2.0 20160901]
This is not reproduced in CPython 3.5 and PyPy2.
2018-06-28
hubo _______________________________________________
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev
