Hi Antoine ! > I would indeed have expected jemalloc to do that (remap the pages) I have no idea about the performance gain this would provide (if any). Could be interesting to explore.
> do you know that Arrow also supports integration with another allocator, mimalloc I only tried Jemalloc and the system allocator because I assumed mimalloc was specific to windows. I'll try to look into this as well ;-) I liked the idea of looking deeper into jemalloc, as it is also the default allocator for Rust. Thanks for your insights! Regards, Remi Le ven. 5 juin 2020 à 14:32, Antoine Pitrou <anto...@python.org> a écrit : > > Le 05/06/2020 à 14:25, Rémi Dettai a écrit : > > Hi Uwe! > > > >> As your suggestions don't seem to be specific to Arrow, why not > > contribute them directly to jemalloc? They are much better in reviewing > > allocator code than we are. > > I mentioned this idea in the jemalloc gitter. The first response was that > > it should work but workloads with realloc aren't very common and these > huge > > allocations can have some impact during coredumps. It's true that > jemalloc > > is supposed to be general purpose allocator, so it has to make > compromises > > in that sense. Let's wait to see if other answers come in. You can follow > > the conversation here if you are interested > > https://gitter.im/jemalloc/jemalloc. It seemed to me while > investigating on > > Jemalloc inners that most of the conception effort is concentrated on > > small/medium allocations. This makes sense as they probably represent 99% > > of workloads. > > Thanks for the pointer. The reply is interesting: > """ > Another trick that’s possible but we don’t do is to just remap the pages > into some new space, rather than alloc-and-copy > """ > I would indeed have expected jemalloc to do that (remap the pages) when > reallocating large allocations, so I'm a bit surprised they don't. I > guess it's one more special case to worry about. > > By the way, before going further with jemalloc, do you know that Arrow > also supports integration with another allocator, mimalloc > (https://github.com/microsoft/mimalloc/)? Perhaps you should try it out > and see whether that improves performance or not. > > (and also the system allocator, by the way) > > Regards > > Antoine. >
