On Thu, 1 Jun 2017 20:21:12 -0700 Larry Hastings <la...@hastings.org> wrote: > On 06/01/2017 02:50 AM, Antoine Pitrou wrote: > > Another possible strategy is: allocate several arenas at once (using a > > larger mmap() call), and use MADV_DONTNEED to relinquish individual > > arenas. > > Thus adding a *fourth* layer of abstraction over memory we get from the OS? > > block -> pool -> arena -> "multi-arena" -> OS
Not a layer of abstraction, just over-allocation. You would over-allocate arenas like you over-allocate a list object's elements storage (though probably using a different over-allocation strategy ;-)). That would reduce the number of mmap() calls (though not necessarily the number of munmap() or madvise() calls), and also provide more opportunities for the kernel to allocate a large page. But you would still handle individual arenas in the allocation code. > Y'know, this might actually make things faster. These multi-arenas > could be the dynamically growing thing Victor wants to try. We allocate > 16mb, then carve it up into arenas (however big those are), then next > time allocate 32mb or what have you. I hope those are not the actual numbers you're intending to use ;-) I still think that allocating more than 1 or 2MB at once would be foolish. Remember this is data that's going to be carved up into (tens of) thousands of small objects. Large objects eschew the small object allocator (not to mention that third-party libraries like Numpy may be using different allocation routines when they allocate very large data). Regards Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com