Le 05/06/2020 à 14:25, Rémi Dettai a écrit : > Hi Uwe! > >> As your suggestions don't seem to be specific to Arrow, why not > contribute them directly to jemalloc? They are much better in reviewing > allocator code than we are. > I mentioned this idea in the jemalloc gitter. The first response was that > it should work but workloads with realloc aren't very common and these huge > allocations can have some impact during coredumps. It's true that jemalloc > is supposed to be general purpose allocator, so it has to make compromises > in that sense. Let's wait to see if other answers come in. You can follow > the conversation here if you are interested > https://gitter.im/jemalloc/jemalloc. It seemed to me while investigating on > Jemalloc inners that most of the conception effort is concentrated on > small/medium allocations. This makes sense as they probably represent 99% > of workloads.
Thanks for the pointer. The reply is interesting: """ Another trick that’s possible but we don’t do is to just remap the pages into some new space, rather than alloc-and-copy """ I would indeed have expected jemalloc to do that (remap the pages) when reallocating large allocations, so I'm a bit surprised they don't. I guess it's one more special case to worry about. By the way, before going further with jemalloc, do you know that Arrow also supports integration with another allocator, mimalloc (https://github.com/microsoft/mimalloc/)? Perhaps you should try it out and see whether that improves performance or not. (and also the system allocator, by the way) Regards Antoine.
