New submission from Adam <adam.jgr...@gmail.com>:
When running 3.7, we noticed a memory leak in threading._shutdown_locks when
non-deamon threads are started but "join()" or "is_alive()" is never called.
Here's a test to illustrate the growth:
=========
import threading
import time
import tracemalloc
def test_leaking_locks():
tracemalloc.start(10)
snap1 = tracemalloc.take_snapshot()
def print_things():
print('.', end='')
for x in range(500):
t = threading.Thread(target=print_things)
t.start()
time.sleep(5)
print('')
gc.collect()
snap2 = tracemalloc.take_snapshot()
filters = []
for stat in
snap2.filter_traces(filters).compare_to(snap1.filter_traces(filters),
'traceback')[:10]:
print("New Bytes: {}\tTotal Bytes {}\tNew blocks: {}\tTotal blocks: {}:
".format(stat.size_diff, stat.size, stat.count_diff ,stat.count))
for line in stat.traceback.format():
print(line)
=========
=========
Output in v3.6.8:
New Bytes: 840 Total Bytes 840 New blocks: 1 Total blocks: 1:
File "/usr/local/lib/python3.6/threading.py", line 884
self._bootstrap_inner()
New Bytes: 608 Total Bytes 608 New blocks: 4 Total blocks: 4:
File "/usr/local/lib/python3.6/tracemalloc.py", line 387
self.traces = _Traces(traces)
File "/usr/local/lib/python3.6/tracemalloc.py", line 524
return Snapshot(traces, traceback_limit)
File "/gems/tests/integration/endpoint_connection_test.py", line 856
snap1 = tracemalloc.take_snapshot()
File "/usr/local/lib/python3.6/site-packages/_pytest/python.py", line 198
testfunction(**testargs)
File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 187
res = hook_impl.function(*args)
File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 87
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 93
return self._inner_hookexec(hook, methods, kwargs)
File "/usr/local/lib/python3.6/site-packages/pluggy/hooks.py", line 286
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/usr/local/lib/python3.6/site-packages/_pytest/python.py", line 1459
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/usr/local/lib/python3.6/site-packages/_pytest/runner.py", line 111
item.runtest()
==========
Output in v3.7.4:
New Bytes: 36000 Total Bytes 36000 New blocks: 1000 Total
blocks: 1000:
File "/usr/local/lib/python3.7/threading.py", line 890
self._bootstrap_inner()
File "/usr/local/lib/python3.7/threading.py", line 914
self._set_tstate_lock()
File "/usr/local/lib/python3.7/threading.py", line 904
self._tstate_lock = _set_sentinel()
New Bytes: 32768 Total Bytes 32768 New blocks: 1 Total blocks:
1:
File "/usr/local/lib/python3.7/threading.py", line 890
self._bootstrap_inner()
File "/usr/local/lib/python3.7/threading.py", line 914
self._set_tstate_lock()
File "/usr/local/lib/python3.7/threading.py", line 909
_shutdown_locks.add(self._tstate_lock)
=================
It looks like this commit didn't take into account the tstate_lock cleanup that
happens in the C code, and it's not removing the _tstate_lock of completed
threads from the _shutdown_locks once the thread finishes, unless the code
manually calls "join()" or "is_alive()" on the thread:
https://github.com/python/cpython/commit/468e5fec8a2f534f1685d59da3ca4fad425c38dd
Let me know if I can provide more clarity on this!
----------
messages: 358551
nosy: krypticus
priority: normal
severity: normal
status: open
title: Threading memory leak in _shutdown_locks for non-daemon threads
type: resource usage
versions: Python 3.7, Python 3.8, Python 3.9
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue39074>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
