[issue26601] Use new madvise()'s MADV_FREE on the private heap

Changes by Antoine Pitrou : -- stage: -> needs patch versions: +Python 3.7 -Python 3.6 ___ Python tracker ___

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Antoine Pitrou added the comment: All this discussion is in the context of the GNU libc allocator, but please remember that Python works on many platforms, including OS X, Windows, the *BSDs... -- ___ Python tracker

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Julian Taylor added the comment: which is exactly what malloc is already doing for, thus my point is by using malloc we would fullfill your request. But do you have an actual real work application where this would help? it is pretty easy to figure out, just run the application under perf and

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Antti Haapala added the comment: mmap is not the problem, the eagerness of munmap is a source of possible problem. The munmap eagerness does not show problems in all programs because the arena allocation heuristics do not work as intended. A proper solution in Linux and other operating

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Julian Taylor added the comment: I know one can change the allocator, but the default is mmap which I don't think is a very good choice for the current arena size. All the arguments about fragmentation and memory space also apply to pythons arena allocator itself and I am not convinced that

[issue26601] Use new madvise()'s MADV_FREE on the private heap

STINNER Victor added the comment: I'm not sure that I understood correctly, but if you are proposing to use malloc()/free() instead of mmap()/munmap() to allocate arenas in pymalloc, you have to know that we already different allocators depending on the platform:

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Charles-François Natali added the comment: The heap on Linux is still a linear contiguous *address space*. I agree that MADV_DONTNEED allow's returning committed memory back to the VM subsystem, but it is still using a large virtual memory area. Not everyone runs on 64-bit, or can waste address

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Julian Taylor added the comment: glibcs malloc is not obstack, its not a simple linear heap where one object on top means everything below is not freeable. It also uses MADV_DONTNEED give sbrk'd memory back to the system. This is the place where MADV_FREE can now be used now as the latter

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Charles-François Natali added the comment: > Julian Taylor added the comment: > > it defaulted to 128kb ten years ago, its a dynamic threshold since ages. Indeed, and that's what encouraged switching the allocator to use mmap. The problem with dynamic mmap threshold is that since the Python

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Julian Taylor added the comment: it defaulted to 128kb ten years ago, its a dynamic threshold since ages. -- ___ Python tracker ___

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Julian Taylor added the comment: ARENA_SIZE is 256kb, the threshold in glibc is up to 32 MB -- ___ Python tracker ___

[issue26601] Use new madvise()'s MADV_FREE on the private heap

David Wilson added the comment: @Julian note that ARENA_SIZE is double the threshold after which at least glibc resorts to calling mmap directly, so using malloc in place of mmap on at least Linux would have zero effect -- nosy: +dw ___ Python

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Julian Taylor added the comment: simplest way to fix this would be to not use malloc instead of mmap in the allocator, then you also get MADV_FREE for free when malloc uses it. The rational for using mmap is kind of weak, the source just says "heap fragmentation". The usual argument for using

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Bar Harel added the comment: Any idea how to test it then? I found this happening by chance because I care about efficiency too much. We can't just stick timeit in random areas and hope to get results. -- ___ Python tracker

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Antoine Pitrou added the comment: Another question is how often this situation occurs in practice and whether it's worth spending some bits, CPU cycles and developer time on "fixing" this. -- ___ Python tracker

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Antti Haapala added the comment: Also what is important to notice is that the behaviour occurs *exactly* because the current heuristics *work*; the allocations were successfully organized so that one arena could be freed as soon as possible. The question is that is it sane to try to free the

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Antti Haapala added the comment: I said that *munmapping* is not the smart thing to do: and it is not, if you're going to *mmap* soon again. -- ___ Python tracker

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Antoine Pitrou added the comment: > ... and it turns out that munmapping is not always that smart thing to do: I don't think a silly benchmark says anything about the efficiency of our allocation strategy. If you have a real-world use case where this turns up, then please post about it.

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Changes by Bar Harel : -- nosy: +bar.harel ___ Python tracker ___ ___ Python-bugs-list

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Antti Haapala added the comment: > Maybe we need an heuristic to release the free arena after N calls to object > allocator functions which don't need this free arena. That'd be my thought; again I believe that `madvise` could be useful there; now `mmap`/`munmap` I believe is particularly

[issue26601] Use new madvise()'s MADV_FREE on the private heap

STINNER Victor added the comment: > ... and it turns out that munmapping is not always that smart thing to do: > http://stackoverflow.com/questions/36548518/variable-assignment-faster-than-one-liner py -3 -m timeit "tuple(range(2000)) == tuple(range(2000))" 1 loops, best of 3: 97.7 usec

[issue26601] Use new madvise()'s MADV_FREE on the private heap

STINNER Victor added the comment: Are you aware of unused memory in the heap memory? The pymalloc memory allocator uses munmap() to release a wgole arena as soon as the last memory block of an arena is freed. -- ___ Python tracker

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Changes by Antoine Pitrou : -- nosy: +neologix ___ Python tracker ___ ___ Python-bugs-list

[issue26601] Use new madvise()'s MADV_FREE on the private heap

Changes by SilentGhost : -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list

[issue26601] Use new madvise()'s MADV_FREE on the private heap

New submission from Marcos Dione: Linux kernel's new madvise() MADV_FREE[1] could be used in the memory allocator to signal unused parts of the private heap as such, allowing the kernel use those pages for resolving lowmem pressure situations. From a LWN article[2]: [...] Rather than