Hi Antoine !
> I would indeed have expected jemalloc to do that (remap the pages)
I have no idea about the performance gain this would provide (if any).
Could be interesting to explore.

> do you know that Arrow also supports integration with another allocator,
mimalloc
I only tried Jemalloc and the system allocator because I assumed mimalloc
was specific to windows. I'll try to look into this as well ;-) I liked the
idea of looking deeper into jemalloc, as it is also the default allocator
for Rust.

Thanks for your insights!

Regards,

Remi



Le ven. 5 juin 2020 à 14:32, Antoine Pitrou <anto...@python.org> a écrit :

>
> Le 05/06/2020 à 14:25, Rémi Dettai a écrit :
> > Hi Uwe!
> >
> >> As your suggestions don't seem to be specific to Arrow, why not
> > contribute them directly to jemalloc? They are much better in reviewing
> > allocator code than we are.
> > I mentioned this idea in the jemalloc gitter. The first response was that
> > it should work but workloads with realloc aren't very common and these
> huge
> > allocations can have some impact during coredumps. It's true that
> jemalloc
> > is supposed to be general purpose allocator, so it has to make
> compromises
> > in that sense. Let's wait to see if other answers come in. You can follow
> > the conversation here if you are interested
> > https://gitter.im/jemalloc/jemalloc. It seemed to me while
> investigating on
> > Jemalloc inners that most of the conception effort is concentrated on
> > small/medium allocations. This makes sense as they probably represent 99%
> > of workloads.
>
> Thanks for the pointer.  The reply is interesting:
> """
> Another trick that’s possible but we don’t do is to just remap the pages
> into some new space, rather than alloc-and-copy
> """
> I would indeed have expected jemalloc to do that (remap the pages) when
> reallocating large allocations, so I'm a bit surprised they don't.  I
> guess it's one more special case to worry about.
>
> By the way, before going further with jemalloc, do you know that Arrow
> also supports integration with another allocator, mimalloc
> (https://github.com/microsoft/mimalloc/)?  Perhaps you should try it out
> and see whether that improves performance or not.
>
> (and also the system allocator, by the way)
>
> Regards
>
> Antoine.
>

Reply via email to