Hi guys:
refer:
https://stackoverflow.com/questions/13027475/cpu-and-memory-usage-of-jemalloc-as-compared-to-glibc-malloc
code: https://github.com/jemalloc/jemalloc-experiments
https://code.woboq.org/userspace/glibc/benchtests/
Best Regards!
Billy xie(谢志民)
On Fri, Oct 30, 2020 at 4:27 PM Yun Tang <myas...@live.com> wrote:
> Hi
>
> > Do you see a noticeable performance difference between the two?
> @ Stephan Ewen , as we already use jemalloc as default memory allocator in
> production, we do not have much experience to compare performace between
> glibc and jemalloc. And I did not take a look at the performance difference
> when I debug docker OOM. I'll have a try to run benchmark on docker with
> different allocators when I have time these days.
>
> @wang gang, yes, that is what I also observed when I pmap the memory when
> using glibc, many memory segments with 64MB size.
>
> @ Billy Xie, what kind of test case did you use? In my point of view,
> compared to who would use more memory in some cases, we should care more
> about who would behave under the max limit for most cases.
>
> Best
> Yun Tang
> ________________________________
> From: Xie Billy <billy.xiezhi...@gmail.com>
> Sent: Friday, October 30, 2020 8:46
> To: dev@flink.apache.org <dev@flink.apache.org>
> Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator
> for debian based Flink docker image
>
> After long time running test application(>96 hours) , glibc used less
> memory than jemalloc , almost half of jemalloc. Can refer this test case.
>
> Best Regards!
> Billy xie(谢志民)
>
>
> On Fri, Oct 30, 2020 at 12:38 AM 王刚 <wanggang11...@autohome.com.cn> wrote:
>
> > we also met glic memory leak .it appeared many 64 MB memory segment when
> > using pmap command . i am +1 on setting jemalloc as default
> >
> > 发送自autohome
> > ________________________________
> > 发件人: Yu Li <car...@gmail.com>
> > 发送时间: 2020-10-30 00:15:12
> > 收件人: dev <dev@flink.apache.org>
> > 抄送: n...@ververica.com <n...@ververica.com>
> > 主题: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for
> > debian based Flink docker image
> >
> > Thanks for sharing more thoughts guys.
> >
> > My experience with malloc libraries are also limited, and as mentioned in
> > my previous email [1], my major concern comes from some online
> information
> > [2] which indicates in some cases jemalloc will consume as much as twice
> > the memory than glibc, which is a huge cost from my point of view.
> However,
> > one may also argue that this post was written in 2015 and things might
> have
> > been improved a lot ever since.
> >
> > Nevertheless, as Till and Yun pointed out, I could also see the benefits
> of
> > setting jemalloc as default, so I don't have strong objections here. But
> > this actually is a change of the default memory allocator of our docker
> > image (previously glibc is the only choice) which definitely worth a big
> > release note.
> >
> > Best Regards,
> > Yu
> >
> > [1]
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-docker-Adopt-Jemalloc-as-default-memory-allocator-for-debian-based-Flink-docker-image-tp45643p45692.html
> > [2] https://stackoverflow.com/a/33993215
> >
> >
> > On Thu, 29 Oct 2020 at 20:55, Stephan Ewen <se...@apache.org<mailto:
> > se...@apache.org>> wrote:
> >
> > > I think Till has a good point. The default should work well, because
> very
> > > few users will ever change it. Even on OOMs.
> > > So I am slightly in favor of going with jemalloc.
> > >
> > > @Yun Tang - Do you see a noticeable performance difference between the
> > two?
> > >
> > > On Thu, Oct 29, 2020 at 12:32 PM Yun Tang <myas...@live.com<mailto:
> > myas...@live.com>> wrote:
> > >
> > > > Hi
> > > >
> > > > I think Till's view deserves wider discussion.
> > > > I want to give my two cents when I debug with Nico on his reported
> > > RocksDB
> > > > OOM problem.
> > > > Jemalloc has the mechanism to profile memory allocation [1] which is
> > > > widely used to analysis memory leak.
> > > > Once we set jemalloc as default memory allocator, the frequency of
> OOM
> > > > behavior decreases obviously.
> > > >
> > > > Considering the OOM killed problem in k8s, change default memory
> > > allocator
> > > > as jemalloc could be something beneficial.
> > > >
> > > > [1]
> > https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling
> > > >
> > > > Best
> > > > Yun Tang
> > > >
> > > > ________________________________
> > > > From: Till Rohrmann <trohrm...@apache.org<mailto:
> trohrm...@apache.org
> > >>
> > > > Sent: Thursday, October 29, 2020 18:34
> > > > To: dev <dev@flink.apache.org<mailto:dev@flink.apache.org>>
> > > > Cc: Yun Tang <myas...@live.com<mailto:myas...@live.com>>
> > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory
> > allocator
> > > > for debian based Flink docker image
> > > >
> > > > Hi Yu,
> > > >
> > > > I see your point why you are in favour of continuing using glibc.
> > > >
> > > > I honestly don't have a lot of experience with malloc libraries so I
> > can
> > > > only argue from a general perspective: We know that glibc has some
> > > problems
> > > > wrt to memory fragmentation which can cause processes to exceed its
> > > memory
> > > > limit. jemalloc seems to address this problem by avoiding memory
> > > > fragmentation. However, it might be the case that jemalloc introduces
> > > other
> > > > problems. In order to figure this out we would need to collect more
> > data
> > > > points/experience. So in the end we have to weigh glibc memory
> > > > fragmentation against unknown problems in jemalloc.
> > > >
> > > > Given that choosing the malloc library will be quite an expert
> > setting, I
> > > > don't expect many Flink users to use it in case they see OOMs.
> Hence, I
> > > > would be slightly in favour of choosing jemalloc as the default
> because
> > > it
> > > > would allow us to gather feedback from users and solve the existing
> > > > problem. In the best case, user's won't run into new problems. If
> they
> > > do,
> > > > then we will have to re-evaluate this decision and might have to
> switch
> > > the
> > > > malloc library back. W/o setting jemalloc as the default, I fear that
> > we
> > > > will never gather enough feedback to make jemalloc confidently the
> > > default.
> > > >
> > > > What one could argue, though, is that making jemalloc the default so
> > > > shortly before the feature freeze limits the timespan we can observe
> > how
> > > it
> > > > behaves before the actual release. Doing such a change at the
> beginning
> > > of
> > > > a release cycle, would allow us to gain more confidence.
> > > >
> > > > In any way, I believe that we should add a big release note and
> > document
> > > > what users should do in case they see memory issues (OOM kills,
> slower
> > > > performance, etc.).
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Tue, Oct 20, 2020 at 5:39 PM Yu Li <car...@gmail.com<mailto:
> > car...@gmail.com>> wrote:
> > > >
> > > > > True, thanks for the reminder Till!
> > > > >
> > > > > I suggest using glibc malloc as default. On one hand this follows
> our
> > > old
> > > > > behavior (only with glibc malloc support in the image), on the
> other
> > > > hand I
> > > > > believe glibc isn't using jemalloc as its default memory allocator
> > for
> > > > some
> > > > > reason.
> > > > >
> > > > > Please let me know your thoughts. Thanks.
> > > > >
> > > > > Best Regards,
> > > > > Yu
> > > > >
> > > > >
> > > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann <trohrm...@apache.org
> > <mailto:trohrm...@apache.org>>
> > > > wrote:
> > > > >
> > > > > > The only question left would be what will be the default value?
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <car...@gmail.com<mailto:
> > car...@gmail.com>> wrote:
> > > > > >
> > > > > > > I'm also +1 on making it configurable in the same Docker image.
> > > > > > >
> > > > > > > It seems we have reached consensus and there are already enough
> > +1s
> > > > to
> > > > > > move
> > > > > > > forward, and suggest @Yun to conclude the discussion directly
> if
> > > > there
> > > > > > are
> > > > > > > no objections.
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Yu
> > > > > > >
> > > > > > >
> > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann <
> > trohrm...@apache.org<mailto:trohrm...@apache.org>>
> > > > > > wrote:
> > > > > > >
> > > > > > > > +1 for making it configurable in the same Docker image.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Till
> > > > > > > >
> > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler <
> > > > > ches...@apache.org<mailto:<br></a>> > > ches...@apache.org>>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > If it is possible to support both allocators in a single
> > image
> > > > then
> > > > > > we
> > > > > > > > > should definitely go with that option.
> > > > > > > > >
> > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote:
> > > > > > > > > > Thanks for Yang's suggestion. I think this could be a
> > better
> > > > > > choice.
> > > > > > > > > > We could install jemalloc and only enable it in
> LD_PRELOAD
> > > when
> > > > > > user
> > > > > > > > > pass specific configuration for docker-entrypoint.sh.
> > > > > > > > > > By doing so, we could avoid to create another docker
> image
> > > tags
> > > > > and
> > > > > > > > also
> > > > > > > > > offer ability to reduce memory fragmentation problem.
> > > > > > > > > >
> > > > > > > > > > Does anyone else have other ideas?
> > > > > > > > > >
> > > > > > > > > > Best
> > > > > > > > > > Yun Tang
> > > > > > > > > > ________________________________
> > > > > > > > > > From: Yang Wang <danrtsey...@gmail.com<mailto:
> > danrtsey...@gmail.com>>
> > > > > > > > > > Sent: Thursday, October 15, 2020 14:59
> > > > > > > > > > To: dev <dev@flink.apache.org<mailto:
> dev@flink.apache.org
> > >>
> > > > > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default
> > > memory
> > > > > > > > > allocator for debian based Flink docker image
> > > > > > > > > >
> > > > > > > > > > Thanks Yun Tang for starting this discussion.
> > > > > > > > > >
> > > > > > > > > > I think this is very important when deploy Flink with
> > > container
> > > > > > > > > environment
> > > > > > > > > > in production. I just
> > > > > > > > > > have quick question. Could we have both memory
> > allocator(e.g.
> > > > > > glibc,
> > > > > > > > > > jemalloc) in the Flink
> > > > > > > > > > official image and enable a specific one by setting ENV?
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Yang
> > > > > > > > > >
> > > > > > > > > > Yu Li <car...@gmail.com<mailto:car...@gmail.com>>
> > 于2020年10月14日周三 下午12:23写道：
> > > > > > > > > >
> > > > > > > > > >> Thanks for debugging and resolving the issue and driving
> > the
> > > > > > > > discussion
> > > > > > > > > >> Yun!
> > > > > > > > > >>
> > > > > > > > > >> For the given solutions, I prefer option 1 (supply
> another
> > > > > > > Dockerfile
> > > > > > > > > using
> > > > > > > > > >> jemalloc as default memory allocator) because of the
> below
> > > > > > reasons:
> > > > > > > > > >>
> > > > > > > > > >> 1. It's hard to say jemalloc is always better than
> > ptmalloc
> > > > > (glibc
> > > > > > > > > malloc),
> > > > > > > > > >> or else glibc should have already adopted it as the
> > default
> > > > > memory
> > > > > > > > > >> allocator. And as indicated here [1], in some cases
> > jemalloc
> > > > > will
> > > > > > > > > >> consume as much as twice the memory than glibc
> > > > > > > > > >>
> > > > > > > > > >> 2. All existing Flink docker images use glibc, if we
> > change
> > > > the
> > > > > > > > default
> > > > > > > > > >> memory allocator to jemalloc and only supply one series
> of
> > > > > images,
> > > > > > > we
> > > > > > > > > will
> > > > > > > > > >> leave those having better performance with glibc no
> other
> > > > > choices
> > > > > > > but
> > > > > > > > > >> staying with old images. In another word, there's a risk
> > of
> > > > > > > > introducing
> > > > > > > > > new
> > > > > > > > > >> problems while fixing an existing one if choosing
> > option-2.
> > > > > > > > > >>
> > > > > > > > > >> And there is a third option considering the efforts of
> > > > > maintaining
> > > > > > > > more
> > > > > > > > > >> images if the memory leak issue is not widely observed,
> > that
> > > > we
> > > > > > > could
> > > > > > > > > >> document the steps of building Dockerfile with jemalloc
> as
> > > > > default
> > > > > > > > > >> allocator so users could build it when needed, which
> > leaves
> > > > the
> > > > > > > burden
> > > > > > > > > to
> > > > > > > > > >> our users so for me it's not the best option.
> > > > > > > > > >>
> > > > > > > > > >> Best Regards,
> > > > > > > > > >> Yu
> > > > > > > > > >>
> > > > > > > > > >> [1] https://stackoverflow.com/a/33993215
> > > > > > > > > >>
> > > > > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <
> myas...@live.com
> > <mailto:myas...@live.com>>
> > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >>> Hi all
> > > > > > > > > >>>
> > > > > > > > > >>> Users report they meet serious memory leak when
> > submitting
> > > > jobs
> > > > > > > > > >>> continously in session mode within k8s (please refer to
> > > > > > > > FLINK-18712[1]
> > > > > > > > > ),
> > > > > > > > > >>> and I also reproduce this to find this is caused by
> > memory
> > > > > > > > > fragmentation
> > > > > > > > > >> of
> > > > > > > > > >>> glibc [2][3] and provide solutions to fix this:
> > > > > > > > > >>>
> > > > > > > > > >>> * Quick but not very clean solution to limit the
> > > memory
> > > > > pool
> > > > > > > of
> > > > > > > > > >> glibc,
> > > > > > > > > >>> limit MALLOC_ARENA_MAX to 2
> > > > > > > > > >>>
> > > > > > > > > >>> * More general solution by rebuilding the image to
> > > > install
> > > > > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to
> > LD_PRELOAD
> > > > > > > > > >>>
> > > > > > > > > >>> The reporter adopted the 2nd solution to fix this issue
> > > > > > eventually.
> > > > > > > > > Thus,
> > > > > > > > > >>> I begin to think whether we should change our
> Dockerfile
> > to
> > > > > adopt
> > > > > > > > > >> jemalloc
> > > > > > > > > >>> as default memory allocator [4].
> > > > > > > > > >>> From my point of view, we have two choices:
> > > > > > > > > >>>
> > > > > > > > > >>> 1. Introduce another Dockerfile using jemalloc as
> > > default
> > > > > > > memory
> > > > > > > > > >>> allocator, which means Flink needs another two new
> image
> > > tags
> > > > > to
> > > > > > > > build
> > > > > > > > > >>> docker with jemalloc while default docker still use
> > glibc.
> > > > > > > > > >>> 2. Set the default memory allocator as jemalloc in
> > our
> > > > > > existing
> > > > > > > > > >>> Dockerfiles, which means Flink offer docker image with
> > > > jemalloc
> > > > > > by
> > > > > > > > > >> default.
> > > > > > > > > >>> I prefer the 2nd option as our company already use
> > jemalloc
> > > > as
> > > > > > > > default
> > > > > > > > > >>> memory allocator for JDK at our production environment
> > due
> > > to
> > > > > > > > messages
> > > > > > > > > >> from
> > > > > > > > > >>> os team warning of glibc's memory fragmentation.
> > > > > > > > > >>> Moreover, I found several open source projects adopting
> > > > > jemalloc
> > > > > > as
> > > > > > > > > >>> default memory allocator within their images to resolve
> > > > memory
> > > > > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant
> > [6].
> > > > > > > > > >>>
> > > > > > > > > >>> What do you guys think of this issue?
> > > > > > > > > >>>
> > > > > > > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712
> > > > > > > > > >>> [2]
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc
> > > > > > > > > >>> [3]
> > https://sourceware.org/bugzilla/show_bug.cgi?id=15321
> > > > > > > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125
> > > > > > > > > >>> [5]
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux
> > > > > > > > > >>> [6] https://github.com/home-assistant/core/pull/33237
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> Best
> > > > > > > > > >>> Yun Tang
> > > > > > > > > >>>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
