New submission from Or Yahalom <o...@parametrix-ins.com>:
I've been debugging a high memory consumption in one of my scripts and traced it back to the `concurrent.futures.ThreadPoolExecutor`. When further investigating and playing around, I found out that when using `concurrent.futures.ThreadPoolExecutor` with the map function, and passing a dictionary to the map's function as an argument, the memory used by the pool won't be freed and as a result the total memory consumption will continue to rise. (Seems like it also happens when passing a list and maybe even other types). Here is an example of a code to recreate this issue: ``` #!/usr/bin/env python3 import os import time import psutil import random import concurrent.futures from memory_profiler import profile as mem_profile p = psutil.Process(os.getpid()) def do_magic(values): return None @mem_profile def foo(): a = {i: chr(i) for i in range(1024)} with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool: proccessed_data = pool.map(do_magic, a) def fooer(): while True: foo() time.sleep(1) fooer() ``` ---------- components: Extension Modules messages: 375647 nosy: or12 priority: normal severity: normal status: open title: Potential Memory leak with concurrent.futures.ThreadPoolExecutor's map type: resource usage versions: Python 3.7 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue41588> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com