[issue26415] Fragmentation of the heap memory in the Python parser

A. Skrobov added the comment: My current patch avoids the memory peak *and* doesn't add any memory fragmentation on top of whatever is already there. In other words, it makes the parser better in this one aspect, and it doesn't make it worse in any aspect. --

[issue26415] Fragmentation of the heap memory in the Python parser

STINNER Victor added the comment: Benjamin Peterson: "It seems to me a simpler solution would be allocate all nodes for a parse tree in an arena." Exactly, that's the real fix. Just make sure that deallocating this arena does punch holes in the "heap memory". For example, on Linux, it means

[issue26415] Fragmentation of the heap memory in the Python parser

A. Skrobov added the comment: An arena might help reclaim the memory once the parsing is complete, but it wouldn't reduce the peak memory consumption by the parser, and so it wouldn't prevent a MemoryError when parsing a 35MB source on a PC with 2GB of RAM. --

[issue26415] Fragmentation of the heap memory in the Python parser

Benjamin Peterson added the comment: It seems to me a simpler solution would be allocate all nodes for a parse tree in an arena. -- ___ Python tracker

[issue26415] Fragmentation of the heap memory in the Python parser

A. Skrobov added the comment: Now that #26526 landed (thanks to everybody involved!), I'm requesting a review on an updated version of my patch, which addresses the excessive memory consumption by the parser. The description of my original patch still applies: > The attached patch for the

[issue26415] Fragmentation of the heap memory in the Python parser

A. Skrobov added the comment: Ping? This patch is two months old now. -- ___ Python tracker ___ ___

[issue26415] Fragmentation of the heap memory in the Python parser

A. Skrobov added the comment: I've now tried it with "perf.py -r -m", and the memory savings are as follows: ### 2to3 ### Mem max: 45976.000 -> 47440.000: 1.0318x larger ### chameleon_v2 ### Mem max: 436968.000 -> 401088.000: 1.0895x smaller ### django_v3 ### Mem max: 23808.000 -> 22584.000:

[issue26415] Fragmentation of the heap memory in the Python parser

Changes by Serhiy Storchaka : -- stage: -> patch review ___ Python tracker ___ ___

[issue26415] Fragmentation of the heap memory in the Python parser

A. Skrobov added the comment: The attached patch for the parser reduces "Maximum resident set size (kbytes)" threefold, for the degenerate example of 'import ast; ast.parse("0,"*100, mode="eval")', by eliminating many CST nodes that have a single child. According to the comment in node.c

[issue26415] Fragmentation of the heap memory in the Python parser

A. Skrobov added the comment: @Serhiy: if your build is 32-bit, then every node is half the size, as it mostly consists of pointers. The amount of heap fragmentation can also depend on gcc/glibc version. -- ___ Python tracker

[issue26415] Fragmentation of the heap memory in the Python parser

Serhiy Storchaka added the comment: On my computer peak memory usage in non-debug build is about 450 MB. Peak memory usage in debug build is about 730 MB. I'm not sure this is due to memory fragmentation. -- ___ Python tracker

[issue26415] Fragmentation of the heap memory in the Python parser

STINNER Victor added the comment: libapr library of Apache is designed to group all memory allocations required to handle an HTTP request. It is possible to release *all* memory allocations of the request at once. It looks like the the "pool" object: *

[issue26415] Fragmentation of the heap memory in the Python parser

STINNER Victor added the comment: My misc notes about memory fragmentation: https://haypo-notes.readthedocs.org/heap_fragmentation.html -- ___ Python tracker

[issue26415] Fragmentation of the heap memory in the Python parser

Changes by STINNER Victor : -- title: Out of memory, trying to parse a 35MB dict -> Fragmentation of the heap memory in the Python parser ___ Python tracker