New submission from Douglas Raillard <douglas.raill...@arm.com>:
The following script exhibits the memory leak. It only happens if
"dispatch_table" is set _before_ calling super().__init__, which is pretty
unexpected.
import pickle
import io
import gc
import tracemalloc
tracemalloc.start(10)
snap = tracemalloc.take_snapshot()
class MyPickler(pickle.Pickler):
def __init__(self, *args, **kwargs):
# Swapping the next 2 lines "solves" the memory leak for some reason
self.dispatch_table = dict()
super().__init__(*args, **kwargs)
l=[]
for i in range(10000):
if i % 1000 == 0:
print('='*80)
snap2 = tracemalloc.take_snapshot()
stats=snap2.compare_to(snap, 'lineno')
for s in stats[:10]:
print(s)
snap = snap2
f = io.BytesIO()
MyPickler(f)
gc.collect()
The output of the last iteration is as follow. The leak of 562 kiB is apparent:
testmem.py:12: size=562 KiB (+62.5 KiB), count=9000 (+1000), average=64
B
/usr/lib/python3.10/tracemalloc.py:125: size=2376 B (-72 B), count=33
(-1), average=72 B
/usr/lib/python3.10/tracemalloc.py:129: size=72 B (+72 B), count=1
(+1), average=72 B
/usr/lib/python3.10/tracemalloc.py:502: size=252 B (+28 B), count=9
(+1), average=28 B
/usr/lib/python3.10/tracemalloc.py:498: size=2104 B (+0 B), count=36
(+0), average=58 B
/home/dourai01/Work/lisa/lisa/testmem.py:10: size=1844 B (+0 B),
count=9 (+0), average=205 B
/usr/lib/python3.10/tracemalloc.py:193: size=1680 B (+0 B), count=35
(+0), average=48 B
/usr/lib/python3.10/tracemalloc.py:547: size=1256 B (+0 B), count=3
(+0), average=419 B
/usr/lib/python3.10/tracemalloc.py:226: size=832 B (+0 B), count=2
(+0), average=416 B
/usr/lib/python3.10/tracemalloc.py:173: size=800 B (+0 B), count=2
(+0), average=400 B
If "dispatch_table" is set after calling super().__init__, there is no leak
anymore:
/usr/lib/python3.10/tracemalloc.py:135: size=740 B (+740 B), count=7
(+7), average=106 B
/usr/lib/python3.10/tracemalloc.py:125: size=2088 B (-656 B), count=29
(-4), average=72 B
/usr/lib/python3.10/tracemalloc.py:136: size=320 B (+320 B), count=1
(+1), average=320 B
/usr/lib/python3.10/tracemalloc.py:132: size=0 B (-256 B), count=0 (-1)
/usr/lib/python3.10/tracemalloc.py:129: size=72 B (+72 B), count=1
(+1), average=72 B
/usr/lib/python3.10/tracemalloc.py:498: size=2008 B (+48 B), count=34
(+1), average=59 B
/usr/lib/python3.10/tracemalloc.py:193: size=1584 B (+48 B), count=33
(+1), average=48 B
/usr/lib/python3.10/tracemalloc.py:502: size=196 B (-28 B), count=7
(-1), average=28 B
/usr/lib/python3.10/tracemalloc.py:126: size=84 B (+28 B), count=3
(+1), average=28 B
----------
components: Library (Lib)
messages: 406478
nosy: douglas-raillard-arm
priority: normal
severity: normal
status: open
title: Custom pickler memory leak
type: resource usage
versions: Python 3.10
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue45830>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
