whitespacer <sila...@gmail.com> added the comment: pitrou, thanks for great diagnosing program! You are right that there is no memory leak.
We have investigated this issue a little bit more - it looks real reason for large memory consuming in the end is not fragmentation, just glibc doesn't release all free memory. So what happens in this example (according to http://man7.org/linux/man-pages/man3/mallopt.3.html): 1) More threads leads to creating of more arenas (e.g. to more sub-heaps). It can be tested with setting environment variable MALLOC_ARENA_MAX to 1 - memory consumption in example will be reduced significantly. But this can be used in real code, performance will degrade. 2) Heap trim threshold (M_TRIM_THRESHOLD) is calculated dynamically. Creation and deletion of small arrays leads to M_MMAP_THRESHOLD=2*4*1024*1024*sizeof(long) on 64 bit systems. So each sub-heap in each arena can have up to 64 mb (if long is 8 bytes) of untrimmed space. It can be tested with setting environment variable MALLOC_TRIM_THRESHOLD_ to 128*1024 - memory consumption in example will be reduced significantly because dynamic calculation of trim threshold will be turned off. Again I doubt this can be used in real code. 3) Maxiumum number of arenas vary on number of cpu's. On my system it looks like it is set to 16. So I get around 16*64 of untrimmed space. To conclude: There is similar bug in python tracker: https://bugs.python.org/issue11849. Message https://bugs.python.org/msg134992 states that nothing can be done in some cases (like in attached example). One possible solution is to use jemalloc - we have tested that it doesn't have such problems. Limiting maximum number of arenas also helps, but can degrade performance. ---------- nosy: +whitespacer _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33444> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
