[issue13483] Use VirtualAlloc to allocate memory arenas

Roundup Robot added the comment: New changeset 44f455e6163d by Martin v. Löwis in branch 'default': Issue #13483: Use VirtualAlloc in obmalloc on Windows. http://hg.python.org/cpython/rev/44f455e6163d -- nosy: +python-dev ___ Python tracker

[issue13483] Use VirtualAlloc to allocate memory arenas

Changes by Martin v. Löwis : -- resolution: -> fixed status: open -> closed ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue13483] Use VirtualAlloc to allocate memory arenas

Martin v. Löwis added the comment: Ok, I'm going to commit this patch. Any further revisions (including reversions) can be done then. -- ___ Python tracker ___ _

[issue13483] Use VirtualAlloc to allocate memory arenas

STINNER Victor added the comment: >> I plan to test the Low Fragmentation Allocator, at least on Windows 7. > I don't think it can be any better than raw mmap() / VirtualAlloc()... I mean using the Low Fragmentation Allocator for PyObject_Malloc() instead of pymalloc. Martin wrote (msg148605):

[issue13483] Use VirtualAlloc to allocate memory arenas

Antoine Pitrou added the comment: > I plan to test the Low Fragmentation Allocator, at least on Windows 7. I don't think it can be any better than raw mmap() / VirtualAlloc()... -- ___ Python tracker _

[issue13483] Use VirtualAlloc to allocate memory arenas

STINNER Victor added the comment: > Ah ok. I guess tuples.py then indeed demonstrates a saving. I'll apply the > patch. According to my test, the memory usage is a little bit better with the patch. So Martin:,do you plan to commit the patch? Or is a benchmark required? Or should check first c

[issue13483] Use VirtualAlloc to allocate memory arenas

Changes by Trent Nelson : -- nosy: +trent ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.or

[issue13483] Use VirtualAlloc to allocate memory arenas

STINNER Victor added the comment: haypo> I tested VirtualAlloc/VirtualFree versus malloc/free haypo> on Windows 7 SP1 64-bit. On my small test, ... I realized that I was no precise: I tried attached va.diff patch. I didn't try to replace completly malloc(). --

[issue13483] Use VirtualAlloc to allocate memory arenas

STINNER Victor added the comment: Martin von Loewis: "If we take the route proposed by this patch, I recommend also dropping all other CRT malloc() calls in Python, and make allocations from the process heap instead (that's a separate issue, though)." => see issue #18203 -- _

[issue13483] Use VirtualAlloc to allocate memory arenas

Changes by Giampaolo Rodola' : -- nosy: +giampaolo.rodola ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:

[issue13483] Use VirtualAlloc to allocate memory arenas

STINNER Victor added the comment: I tested VirtualAlloc/VirtualFree versus malloc/free on Windows 7 SP1 64-bit. On my small test, when using VirtualAlloc/VirtualFree, the memory peak is lower (ex: 58.1 MB vs 59.0), and the memory usage is the same or sometimes lower. The difference is small, m

[issue13483] Use VirtualAlloc to allocate memory arenas

STINNER Victor added the comment: Set also issue #3329 which proposes an API to define memory allocators. -- nosy: +haypo ___ Python tracker ___ _

[issue13483] Use VirtualAlloc to allocate memory arenas

Martin v. Löwis added the comment: Ah ok. I guess tuples.py then indeed demonstrates a saving. I'll apply the patch. -- ___ Python tracker ___ __

[issue13483] Use VirtualAlloc to allocate memory arenas

Antoine Pitrou added the comment: I asked for benchmarks because I don't know anything about Windows virtual memory management, but if other people think this patch should go in then it's fine. The main point of using VirtualAlloc/VirtualFree was, in my mind, to allow *releasing* memory in mo

[issue13483] Use VirtualAlloc to allocate memory arenas

Charles-François Natali added the comment: I can't speak for Antoine, but I guess that the result of pybench would be enough to make sure it doesn't introduce any regression (which would be *really* suprising). As for the memory savings, the benchmark you posted earlier is conclusive enough IMO (

[issue13483] Use VirtualAlloc to allocate memory arenas

Martin v. Löwis added the comment: Antoine's request for benchmarks still stands. I continue to think that it should be applied even in absence of benchmarks. In the absence of third opinions on this specific aspect, I don't think it can be applied. --

[issue13483] Use VirtualAlloc to allocate memory arenas

Charles-François Natali added the comment: Martin, do you think your latest patch can be committed? -- ___ Python tracker ___ ___ Pyth

[issue13483] Use VirtualAlloc to allocate memory arenas

Martin v. Löwis added the comment: Here is an updated patch. -- Added file: http://bugs.python.org/file26070/va.diff ___ Python tracker ___ _

[issue13483] Use VirtualAlloc to allocate memory arenas

Martin v. Löwis added the comment: Here is a benchmark. Based on my assumption that this patch may reduce allocation overheads due to minimizing padding+fragmentation, it allocates a lot of memory, and then waits 20s so you can check in the process explorer what the "Commit Size" of the proce

[issue13483] Use VirtualAlloc to allocate memory arenas

Antoine Pitrou added the comment: Le mardi 29 novembre 2011 à 22:39 +, Brian Curtin a écrit : > Brian Curtin added the comment: > > > Tim, Brian, do you know anything about this? > > Unfortunately, no. It's on my todo list of things to understand but I > don't see that happening in the ne

[issue13483] Use VirtualAlloc to allocate memory arenas

Brian Curtin added the comment: > Tim, Brian, do you know anything about this? Unfortunately, no. It's on my todo list of things to understand but I don't see that happening in the near future. I'm willing to run tests or benchmarks for this issue, but that's likely the most I can provide.

[issue13483] Use VirtualAlloc to allocate memory arenas

Tim Golden added the comment: 'fraid not. I've never had to dig into the allocation stuff at this level. -- ___ Python tracker ___ __

[issue13483] Use VirtualAlloc to allocate memory arenas

Antoine Pitrou added the comment: > The patch looks good to me. > > To study Microsoft's malloc, see VC\crt\src\malloc.c. Typically, it > uses HeapAlloc from the CRT heap, unless it's in 32-bit mode, and > __active_heap is either __V6_HEAP or __V5_HEAP. This is determined at > startup by __heap

[issue13483] Use VirtualAlloc to allocate memory arenas

Martin v. Löwis added the comment: The patch looks good to me. To study Microsoft's malloc, see VC\crt\src\malloc.c. Typically, it uses HeapAlloc from the CRT heap, unless it's in 32-bit mode, and __active_heap is either __V6_HEAP or __V5_HEAP. This is determined at startup by __heap_select,

[issue13483] Use VirtualAlloc to allocate memory arenas

New submission from Antoine Pitrou : Similar to issue #11849, this patch proposes to use VirtualAlloc/VirtualFree to allocate the Python allocator's memory arenas (rather than malloc() / free()). It might help release more memory if there is some fragmentation, although I don't know how Micros