Re: [Proposal] Use jemalloc as default memory allocator for Mesos
Looks like we can agree that it's a useful feature to add, which is great! :) I've written up the first version of a Design Document, please feel free to take a look and comment: https://docs.google.com/document/d/1qCVK40nOKDWlLKYrYUpLDwkH654zCoatIzi4OnsW8ac/edit?usp=sharing Thanks, Benno On Tue, Sep 5, 2017 at 6:24 PM, Jeff Coffler < jeff.coff...@microsoft.com.invalid> wrote: > We're talking with the jemalloc folks. > > They did have a plan to move to cmake, but they wanted to move to cmake > for all platforms simultaneously (a tall order). My thoughts are to ask > them to move to cmake (we'll even do the work here if they'll take 'em) for > Windows only. This is manageable (although they would be using two build > systems - like Mesos DEVs don't know all about that), but jemalloc already > does that since they've checked in some Visual Studio solutions. > > In the past, I've just asked the company to buy what I need (i.e. > Smartheap), and it's worked splendidly. But that's not an option for Mesos, > as you pointed out. > > I'll talk with JohnK this morning (he's managing this), and will see where > he is in discussions with jemalloc authors. Last week, we were talking with > the guy that did a lot of the cmake porting, but we need to talk with the > jemalloc author directly. > > Given the situation, I guess we should just move with jemalloc. If worse > comes to worse, we'll maintain our own cmake build process for jemalloc > against a specific branch. We already maintain local changes for some other > 3rd party dependencies. > > /Jeff > > -Original Message----- > From: Benno Evers [mailto:bev...@mesosphere.com] > Sent: Tuesday, September 5, 2017 4:02 AM > To: dev@mesos.apache.org > Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos > > Hi Jeff, > > do you have a particular alternative in mind? Certainly SmartHeap is a > non-starter because it is proprietary. I actually did look for alternatives > before sending out this proposal, but from what I've seen, there isn't > exactly an abundance of well-tested, widely used and stable malloc > implementations with heap profiling features, i.e. I'm not aware of any > besides tcmalloc and jemalloc. > > I also don't think having a native windows build is as important as being > well-tested, because either the build just works (perfect), or it doesn't > work and the feature will be disabled in windows, then people are in > exactly the same position as they were before, in particular they can still > use windows-native heap profiling solutions if they want. On the other > hand, if we decide on some obscure malloc, there is a much higher chance of > accidentally introducing subtle bugs for the people who enable the feature. > > Best regards, > Benno > > > On Thu, Aug 31, 2017 at 5:41 PM, Jeff Coffler < jeff.coff...@microsoft.com > .invalid> wrote: > > > The fact that Firefox works with jemalloc isn't necessarily > > indicative. I, for one, would like to avoid dependencies on Cygwin for > > Mesos. We don't need it today, and we're building an awful lot. > > > > (Interesting that you brought up SASL-based auth - that's currently in > > the process of being ported - natively - to Windows.) > > > > There are many options for memory allocators that run both on Linux > > and Windows. For example, I've used SmartHeap in the past, and that > > works well on UNIX, Linux, Windows, and more. (That's commercial; I'm > > not sure it's free for open source products or not.) I'm not > > necessarily suggesting SmartHeap, I'm just pointing out that there are > > native options for both Linux and Windows that are well ported and work > everywhere. > > > > Before we decide on using jemalloc, I'd like to see someone look at > > memory allocators and see if there's one we can use (i.e. free) that's > > natively supported both on Linux and Windows without jumping through > > hoops (like Cygwin for builds). If there are native choices, we should > > look at those much more aggressively than an option that doesn't work > well on Windows. > > > > /Jeff > > > > -Original Message- > > From: Till Toenshoff [mailto:toensh...@me.com] > > Sent: Wednesday, August 30, 2017 6:27 PM > > To: dev@mesos.apache.org > > Subject: Re: [Proposal] Use jemalloc as default memory allocator for > > Mesos > > > > It appears that jemalloc does support Windows (64bit) > > See: https://na01.safelinks.protection.outlook.com/?url= > >
RE: [Proposal] Use jemalloc as default memory allocator for Mesos
We're talking with the jemalloc folks. They did have a plan to move to cmake, but they wanted to move to cmake for all platforms simultaneously (a tall order). My thoughts are to ask them to move to cmake (we'll even do the work here if they'll take 'em) for Windows only. This is manageable (although they would be using two build systems - like Mesos DEVs don't know all about that), but jemalloc already does that since they've checked in some Visual Studio solutions. In the past, I've just asked the company to buy what I need (i.e. Smartheap), and it's worked splendidly. But that's not an option for Mesos, as you pointed out. I'll talk with JohnK this morning (he's managing this), and will see where he is in discussions with jemalloc authors. Last week, we were talking with the guy that did a lot of the cmake porting, but we need to talk with the jemalloc author directly. Given the situation, I guess we should just move with jemalloc. If worse comes to worse, we'll maintain our own cmake build process for jemalloc against a specific branch. We already maintain local changes for some other 3rd party dependencies. /Jeff -Original Message- From: Benno Evers [mailto:bev...@mesosphere.com] Sent: Tuesday, September 5, 2017 4:02 AM To: dev@mesos.apache.org Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos Hi Jeff, do you have a particular alternative in mind? Certainly SmartHeap is a non-starter because it is proprietary. I actually did look for alternatives before sending out this proposal, but from what I've seen, there isn't exactly an abundance of well-tested, widely used and stable malloc implementations with heap profiling features, i.e. I'm not aware of any besides tcmalloc and jemalloc. I also don't think having a native windows build is as important as being well-tested, because either the build just works (perfect), or it doesn't work and the feature will be disabled in windows, then people are in exactly the same position as they were before, in particular they can still use windows-native heap profiling solutions if they want. On the other hand, if we decide on some obscure malloc, there is a much higher chance of accidentally introducing subtle bugs for the people who enable the feature. Best regards, Benno On Thu, Aug 31, 2017 at 5:41 PM, Jeff Coffler < jeff.coff...@microsoft.com.invalid> wrote: > The fact that Firefox works with jemalloc isn't necessarily > indicative. I, for one, would like to avoid dependencies on Cygwin for > Mesos. We don't need it today, and we're building an awful lot. > > (Interesting that you brought up SASL-based auth - that's currently in > the process of being ported - natively - to Windows.) > > There are many options for memory allocators that run both on Linux > and Windows. For example, I've used SmartHeap in the past, and that > works well on UNIX, Linux, Windows, and more. (That's commercial; I'm > not sure it's free for open source products or not.) I'm not > necessarily suggesting SmartHeap, I'm just pointing out that there are > native options for both Linux and Windows that are well ported and work > everywhere. > > Before we decide on using jemalloc, I'd like to see someone look at > memory allocators and see if there's one we can use (i.e. free) that's > natively supported both on Linux and Windows without jumping through > hoops (like Cygwin for builds). If there are native choices, we should > look at those much more aggressively than an option that doesn't work well on > Windows. > > /Jeff > > -Original Message- > From: Till Toenshoff [mailto:toensh...@me.com] > Sent: Wednesday, August 30, 2017 6:27 PM > To: dev@mesos.apache.org > Subject: Re: [Proposal] Use jemalloc as default memory allocator for > Mesos > > It appears that jemalloc does support Windows (64bit) > See: https://na01.safelinks.protection.outlook.com/?url= > https%3A%2F%2Fgithub.com%2Fjemalloc%2Fjemalloc%2Fblob% > 2Fdev%2Fmsvc%2FReadMe.txt&data=02%7C01%7CJeff.Coffler%40microsoft.com% > 7C1cb6c56ba96f401a8d6108d4f00f660e%7C72f988bf86f141af91ab2d7cd011 > db47%7C1%7C0%7C636397396315954606&sdata=e6RrlOXc% > 2B9BAY0FwBx3UMKElg3S5SCgZXKVYKGSAfQE%3D&reserved=0 < > https://na01.safelinks.protection.outlook.com/?url= > https%3A%2F%2Fgithub.com%2Fjemalloc%2Fjemalloc%2Fblob% > 2Fdev%2Fmsvc%2FReadMe.txt&data=02%7C01%7CJeff.Coffler%40microsoft.com% > 7C1cb6c56ba96f401a8d6108d4f00f660e%7C72f988bf86f141af91ab2d7cd011 > db47%7C1%7C0%7C636397396315954606&sdata=e6RrlOXc% > 2B9BAY0FwBx3UMKElg3S5SCgZXKVYKGSAfQE%3D&reserved=0> > > tcmalloc on the other hand appears to only offer a min
Re: [Proposal] Use jemalloc as default memory allocator for Mesos
Hi Jeff, do you have a particular alternative in mind? Certainly SmartHeap is a non-starter because it is proprietary. I actually did look for alternatives before sending out this proposal, but from what I've seen, there isn't exactly an abundance of well-tested, widely used and stable malloc implementations with heap profiling features, i.e. I'm not aware of any besides tcmalloc and jemalloc. I also don't think having a native windows build is as important as being well-tested, because either the build just works (perfect), or it doesn't work and the feature will be disabled in windows, then people are in exactly the same position as they were before, in particular they can still use windows-native heap profiling solutions if they want. On the other hand, if we decide on some obscure malloc, there is a much higher chance of accidentally introducing subtle bugs for the people who enable the feature. Best regards, Benno On Thu, Aug 31, 2017 at 5:41 PM, Jeff Coffler < jeff.coff...@microsoft.com.invalid> wrote: > The fact that Firefox works with jemalloc isn't necessarily indicative. I, > for one, would like to avoid dependencies on Cygwin for Mesos. We don't > need it today, and we're building an awful lot. > > (Interesting that you brought up SASL-based auth - that's currently in the > process of being ported - natively - to Windows.) > > There are many options for memory allocators that run both on Linux and > Windows. For example, I've used SmartHeap in the past, and that works well > on UNIX, Linux, Windows, and more. (That's commercial; I'm not sure it's > free for open source products or not.) I'm not necessarily suggesting > SmartHeap, I'm just pointing out that there are native options for both > Linux and Windows that are well ported and work everywhere. > > Before we decide on using jemalloc, I'd like to see someone look at memory > allocators and see if there's one we can use (i.e. free) that's natively > supported both on Linux and Windows without jumping through hoops (like > Cygwin for builds). If there are native choices, we should look at those > much more aggressively than an option that doesn't work well on Windows. > > /Jeff > > -----Original Message- > From: Till Toenshoff [mailto:toensh...@me.com] > Sent: Wednesday, August 30, 2017 6:27 PM > To: dev@mesos.apache.org > Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos > > It appears that jemalloc does support Windows (64bit) > See: https://na01.safelinks.protection.outlook.com/?url= > https%3A%2F%2Fgithub.com%2Fjemalloc%2Fjemalloc%2Fblob% > 2Fdev%2Fmsvc%2FReadMe.txt&data=02%7C01%7CJeff.Coffler%40microsoft.com% > 7C1cb6c56ba96f401a8d6108d4f00f660e%7C72f988bf86f141af91ab2d7cd011 > db47%7C1%7C0%7C636397396315954606&sdata=e6RrlOXc% > 2B9BAY0FwBx3UMKElg3S5SCgZXKVYKGSAfQE%3D&reserved=0 < > https://na01.safelinks.protection.outlook.com/?url= > https%3A%2F%2Fgithub.com%2Fjemalloc%2Fjemalloc%2Fblob% > 2Fdev%2Fmsvc%2FReadMe.txt&data=02%7C01%7CJeff.Coffler%40microsoft.com% > 7C1cb6c56ba96f401a8d6108d4f00f660e%7C72f988bf86f141af91ab2d7cd011 > db47%7C1%7C0%7C636397396315954606&sdata=e6RrlOXc% > 2B9BAY0FwBx3UMKElg3S5SCgZXKVYKGSAfQE%3D&reserved=0> > > tcmalloc on the other hand appears to only offer a minimal variant on > Windows. > See: https://na01.safelinks.protection.outlook.com/?url= > https%3A%2F%2Fgithub.com%2Fgperftools%2Fgperftools& > data=02%7C01%7CJeff.Coffler%40microsoft.com%7C1cb6c56ba96f401a8d6108d4f00f > 660e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0% > 7C636397396315954606&sdata=rz1OflD81bHoGkSpw7JinsVKLjpGK9 > Lso546GnfP5L8%3D&reserved=0 <https://na01.safelinks. > protection.outlook.com/?url=https%3A%2F%2Fgithub.com% > 2Fgperftools%2Fgperftools&data=02%7C01%7CJeff.Coffler%40microsoft.com% > 7C1cb6c56ba96f401a8d6108d4f00f660e%7C72f988bf86f141af91ab2d7cd011 > db47%7C1%7C0%7C636397396315954606&sdata=rz1OflD81bHoGkSpw7JinsVKLjpGK9 > Lso546GnfP5L8%3D&reserved=0> - grep for "COMPILING ON NON-LINUX SYSTEMS” > See: https://na01.safelinks.protection.outlook.com/?url= > https%3A%2F%2Fgithub.com%2Fgperftools%2Fgperftools% > 2Fblob%2Fmaster%2FINSTALL&data=02%7C01%7CJeff.Coffler%40microsoft.com% > 7C1cb6c56ba96f401a8d6108d4f00f660e%7C72f988bf86f141af91ab2d7cd011 > db47%7C1%7C0%7C636397396315954606&sdata=WcU72HYzVMao7yF7LJ7Ks% > 2Bdv0P7zhjE%2BM6cKIOfa488%3D&reserved=0 <https://na01.safelinks. > protection.outlook.com/?url=https%3A%2F%2Fgithub.com% > 2Fgperftools%2Fgperftools%2Fblob%2Fmaster%2FINSTALL& > data=02%7C01%7CJeff.Coffler%40microsoft.com%7C1cb6c56ba96f401a8d6108d4f00f > 660e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0% > 7C63639
RE: [Proposal] Use jemalloc as default memory allocator for Mesos
There's a jemalloc-cmake repo that has cmake support (I just tested it, and works great with VS 2017 x64). You can find it here: https://github.com/jemalloc/jemalloc-cmake However, it looks like it's a few versions behind the main jemalloc repo (4.3.1 looks to be the latest version in the jemalloc-cmake repo, while 5.0.1 is in the jemalloc repo). I'm reaching out to some of the committers as to whether there are plans to integrate these changes into the main repo, and I'll report back with what I learn. It seems they do want to support cmake in the main jemalloc repo given this discussion: https://github.com/jemalloc/jemalloc/issues/303 -Original Message- From: and...@schwartzmeyer.com [mailto:and...@schwartzmeyer.com] Sent: Thursday, August 31, 2017 11:35 AM To: dev@mesos.apache.org Cc: toenshoff=me@schwartzmeyer.com; Till Toenshoff Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos I looked at the Windows build a bit closer, and the Cygwin requirement appears to simply be for the use of autogen to generate headers. Once the headers are generated, the rest of the build is driven by a committed Visual Studio 2015 solution, which means that jemalloc can be built natively on Windows using MSVC. It should be possibly to replace autogen with some other tooling in order to generate the required headers. However, this is work we would likely have to do and of course submit upstream. Furthermore, the committed solution itself is a bad idea, as it won't "just work" for Visual Studio 2017. We would want to instead write a CMake build for jemalloc to replace the committed solution upstream, so that VS 2015, 2017, etc. solutions can just be generated. This is what we do for Mesos, and have done for other dependencies such as ZooKeeper. So in summary: it looks like jemalloc is usable on Windows, but it will require further effort to integrate into the build cleanly. On 08/30/2017 6:26 pm, Till Toenshoff wrote: > It appears that jemalloc does support Windows (64bit) > See: > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu > b.com%2Fjemalloc%2Fjemalloc%2Fblob%2Fdev%2Fmsvc%2FReadMe.txt&data=02%7 > C01%7Cjohnkord%40microsoft.com%7Ca59a5339a3514ad65d6508d4f09f0136%7C72 > f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636398013091711993&sdata=8uH6 > e5lX8ZT1VIBq8jK9A3REfUfOUHC%2FRj0GcDSX%2FOY%3D&reserved=0 > <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith > ub.com%2Fjemalloc%2Fjemalloc%2Fblob%2Fdev%2Fmsvc%2FReadMe.txt&data=02% > 7C01%7Cjohnkord%40microsoft.com%7Ca59a5339a3514ad65d6508d4f09f0136%7C7 > 2f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636398013091711993&sdata=8uH > 6e5lX8ZT1VIBq8jK9A3REfUfOUHC%2FRj0GcDSX%2FOY%3D&reserved=0> > > tcmalloc on the other hand appears to only offer a minimal variant on > Windows. > See: > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu > b.com%2Fgperftools%2Fgperftools&data=02%7C01%7Cjohnkord%40microsoft.co > m%7Ca59a5339a3514ad65d6508d4f09f0136%7C72f988bf86f141af91ab2d7cd011db4 > 7%7C1%7C0%7C636398013091711993&sdata=DAZLSajdGete70eJrqe46Aaf%2FV4oKBa > KNHfwciRo0R4%3D&reserved=0 > <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith > ub.com%2Fgperftools%2Fgperftools&data=02%7C01%7Cjohnkord%40microsoft.com%7Ca59a5339a3514ad65d6508d4f09f0136%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636398013091711993&sdata=DAZLSajdGete70eJrqe46Aaf%2FV4oKBaKNHfwciRo0R4%3D&reserved=0> > - grep for "COMPILING ON NON-LINUX SYSTEMS” > See: > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu > b.com%2Fgperftools%2Fgperftools%2Fblob%2Fmaster%2FINSTALL&data=02%7C01 > %7Cjohnkord%40microsoft.com%7Ca59a5339a3514ad65d6508d4f09f0136%7C72f98 > 8bf86f141af91ab2d7cd011db47%7C1%7C0%7C636398013091711993&sdata=4dnlI5Z > NaVq%2By%2BvJngZ7zRiSHBrwEiI8ieaIanG1xVo%3D&reserved=0 > <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith > ub.com%2Fgperftools%2Fgperftools%2Fblob%2Fmaster%2FINSTALL&data=02%7C01%7Cjohnkord%40microsoft.com%7Ca59a5339a3514ad65d6508d4f09f0136%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636398013091711993&sdata=4dnlI5ZNaVq%2By%2BvJngZ7zRiSHBrwEiI8ieaIanG1xVo%3D&reserved=0> > - grep for “Windows (MSVC, Cygwin, and MinGW)" > > So both options rely on Cygwin or MinGW for building - a requirement > that the Mesos build itself does not have and proves your point of > stuff not really “just working” at least when it comes to the build > step of those packages. > > Seems that without trying it, we won’t find out if jemalloc works as > hoped on Windows for us - the Firefox project results however are > encouraging. On the other hand, if it doesn’t work,
Re: [Proposal] Use jemalloc as default memory allocator for Mesos
I looked at the Windows build a bit closer, and the Cygwin requirement appears to simply be for the use of autogen to generate headers. Once the headers are generated, the rest of the build is driven by a committed Visual Studio 2015 solution, which means that jemalloc can be built natively on Windows using MSVC. It should be possibly to replace autogen with some other tooling in order to generate the required headers. However, this is work we would likely have to do and of course submit upstream. Furthermore, the committed solution itself is a bad idea, as it won't "just work" for Visual Studio 2017. We would want to instead write a CMake build for jemalloc to replace the committed solution upstream, so that VS 2015, 2017, etc. solutions can just be generated. This is what we do for Mesos, and have done for other dependencies such as ZooKeeper. So in summary: it looks like jemalloc is usable on Windows, but it will require further effort to integrate into the build cleanly. On 08/30/2017 6:26 pm, Till Toenshoff wrote: It appears that jemalloc does support Windows (64bit) See: https://github.com/jemalloc/jemalloc/blob/dev/msvc/ReadMe.txt <https://github.com/jemalloc/jemalloc/blob/dev/msvc/ReadMe.txt> tcmalloc on the other hand appears to only offer a minimal variant on Windows. See: https://github.com/gperftools/gperftools <https://github.com/gperftools/gperftools> - grep for "COMPILING ON NON-LINUX SYSTEMS” See: https://github.com/gperftools/gperftools/blob/master/INSTALL <https://github.com/gperftools/gperftools/blob/master/INSTALL> - grep for “Windows (MSVC, Cygwin, and MinGW)" So both options rely on Cygwin or MinGW for building - a requirement that the Mesos build itself does not have and proves your point of stuff not really “just working” at least when it comes to the build step of those packages. Seems that without trying it, we won’t find out if jemalloc works as hoped on Windows for us - the Firefox project results however are encouraging. On the other hand, if it doesn’t work, we could simply decide to disable it on Windows just like some other Mesos features will remain disabled on that platform unless someone decides to port them - e.g. SASL based authn. On Aug 25, 2017, at 3:28 PM, Benno Evers wrote: Hi Jeff, from looking around on the internet, it seems like Firefox builds with jemalloc on all platforms, and I've also seen reports of people successfully using tcmalloc heap profiling on windows. I'm afraid I don't currently have a Windows machine with development environment set up, so I can't provide direct user experience. In the worst case, we would have to disable jemalloc-support on windows, but I hope it won't be necessary. Since you probably have more experience with memory management on windows, is there a reason to suspect that it should or shouldn't work? Best regards, Benno On Wed, Aug 23, 2017 at 6:16 PM, Jeff Coffler < jeff.coff...@microsoft.com.invalid> wrote: Hi Benno, What's the availability of both jemalloc and tcmalloc on the Windows platform? Do the products work there properly? There are solutions that I know work on Windows (from past work I've done). I'm unsure about either jemalloc and tcmalloc, however. Thanks, /Jeff -Original Message- From: Benno Evers [mailto:bev...@mesosphere.com] Sent: Tuesday, August 22, 2017 3:16 AM To: dev@mesos.apache.org Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos Hi Alexander, in general, jemalloc and tcmalloc are very similar, and seem to be taking ideas from each other (in fact the jeprof executable started as a copy of pprof and there are still references the pprof documentation in some comments) From what I've seen, the main difference is that the profiling seems better-suited to multi-threaded programs, in particular the profile file format includes per-thread memory statistics and the profiling features can be turned on and off individually per thread. From an API perspective, all settings can be accessed by the mallctl() function, while it seems that tcmalloc requires some options to be set by environment variable ( https://na01.safelinks.protection.outlook.com/?url= https%3A%2F%2Fgperftools.github.io%2Fgperftools% 2Fheapprofile.html&data=02%7C01%7CJeff.Coffler%40microsoft.com% 7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011 db47%7C1%7C0%7C636389937852256730&sdata=IQeb2% 2BpcrWRQ8yvdTgOEHfyplgC36dy73nnXswdPamo%3D&reserved=0). Finally, I also found the documentation to be more thorough. But again, the two are very similar, so I think the main decision here isn't whether to choose jemalloc or tcmalloc but whether to switch to a custom memory allocator that has support for profiling heap memory usage. On Mon, Aug 21, 2017 at 4:26 PM, Alexander Rojas wrote: Hi Benno, This does sound
RE: [Proposal] Use jemalloc as default memory allocator for Mesos
The fact that Firefox works with jemalloc isn't necessarily indicative. I, for one, would like to avoid dependencies on Cygwin for Mesos. We don't need it today, and we're building an awful lot. (Interesting that you brought up SASL-based auth - that's currently in the process of being ported - natively - to Windows.) There are many options for memory allocators that run both on Linux and Windows. For example, I've used SmartHeap in the past, and that works well on UNIX, Linux, Windows, and more. (That's commercial; I'm not sure it's free for open source products or not.) I'm not necessarily suggesting SmartHeap, I'm just pointing out that there are native options for both Linux and Windows that are well ported and work everywhere. Before we decide on using jemalloc, I'd like to see someone look at memory allocators and see if there's one we can use (i.e. free) that's natively supported both on Linux and Windows without jumping through hoops (like Cygwin for builds). If there are native choices, we should look at those much more aggressively than an option that doesn't work well on Windows. /Jeff -Original Message- From: Till Toenshoff [mailto:toensh...@me.com] Sent: Wednesday, August 30, 2017 6:27 PM To: dev@mesos.apache.org Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos It appears that jemalloc does support Windows (64bit) See: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjemalloc%2Fjemalloc%2Fblob%2Fdev%2Fmsvc%2FReadMe.txt&data=02%7C01%7CJeff.Coffler%40microsoft.com%7C1cb6c56ba96f401a8d6108d4f00f660e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636397396315954606&sdata=e6RrlOXc%2B9BAY0FwBx3UMKElg3S5SCgZXKVYKGSAfQE%3D&reserved=0 <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjemalloc%2Fjemalloc%2Fblob%2Fdev%2Fmsvc%2FReadMe.txt&data=02%7C01%7CJeff.Coffler%40microsoft.com%7C1cb6c56ba96f401a8d6108d4f00f660e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636397396315954606&sdata=e6RrlOXc%2B9BAY0FwBx3UMKElg3S5SCgZXKVYKGSAfQE%3D&reserved=0> tcmalloc on the other hand appears to only offer a minimal variant on Windows. See: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgperftools%2Fgperftools&data=02%7C01%7CJeff.Coffler%40microsoft.com%7C1cb6c56ba96f401a8d6108d4f00f660e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636397396315954606&sdata=rz1OflD81bHoGkSpw7JinsVKLjpGK9Lso546GnfP5L8%3D&reserved=0 <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgperftools%2Fgperftools&data=02%7C01%7CJeff.Coffler%40microsoft.com%7C1cb6c56ba96f401a8d6108d4f00f660e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636397396315954606&sdata=rz1OflD81bHoGkSpw7JinsVKLjpGK9Lso546GnfP5L8%3D&reserved=0> - grep for "COMPILING ON NON-LINUX SYSTEMS” See: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgperftools%2Fgperftools%2Fblob%2Fmaster%2FINSTALL&data=02%7C01%7CJeff.Coffler%40microsoft.com%7C1cb6c56ba96f401a8d6108d4f00f660e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636397396315954606&sdata=WcU72HYzVMao7yF7LJ7Ks%2Bdv0P7zhjE%2BM6cKIOfa488%3D&reserved=0 <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgperftools%2Fgperftools%2Fblob%2Fmaster%2FINSTALL&data=02%7C01%7CJeff.Coffler%40microsoft.com%7C1cb6c56ba96f401a8d6108d4f00f660e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636397396315954606&sdata=WcU72HYzVMao7yF7LJ7Ks%2Bdv0P7zhjE%2BM6cKIOfa488%3D&reserved=0> - grep for “Windows (MSVC, Cygwin, and MinGW)" So both options rely on Cygwin or MinGW for building - a requirement that the Mesos build itself does not have and proves your point of stuff not really “just working” at least when it comes to the build step of those packages. Seems that without trying it, we won’t find out if jemalloc works as hoped on Windows for us - the Firefox project results however are encouraging. On the other hand, if it doesn’t work, we could simply decide to disable it on Windows just like some other Mesos features will remain disabled on that platform unless someone decides to port them - e.g. SASL based authn. > On Aug 25, 2017, at 3:28 PM, Benno Evers wrote: > > Hi Jeff, > > from looking around on the internet, it seems like Firefox builds with > jemalloc on all platforms, and I've also seen reports of people > successfully using tcmalloc heap profiling on windows. I'm afraid I > don't currently have a Windows machine with development environment > set up, so I can't provide direct user experience. > > In the worst case, we would have to disable jemalloc-support on > windows, but I hope it won't be necessary. > > Since you probably have more experience with
Re: [Proposal] Use jemalloc as default memory allocator for Mesos
Ow, I forgot my actual vote :) +1 for using jemalloc +1 for making it non bundled +1 for using it by default if the configuration phase locates it > On Aug 25, 2017, at 3:28 PM, Benno Evers wrote: > > Hi Jeff, > > from looking around on the internet, it seems like Firefox builds with > jemalloc on all platforms, and I've also seen reports of people > successfully using tcmalloc heap profiling on windows. I'm afraid I don't > currently have a Windows machine with development environment set up, so I > can't provide direct user experience. > > In the worst case, we would have to disable jemalloc-support on windows, > but I hope it won't be necessary. > > Since you probably have more experience with memory management on windows, > is there a reason to suspect that it should or shouldn't work? > > Best regards, > Benno > > On Wed, Aug 23, 2017 at 6:16 PM, Jeff Coffler < > jeff.coff...@microsoft.com.invalid> wrote: > >> Hi Benno, >> >> What's the availability of both jemalloc and tcmalloc on the Windows >> platform? Do the products work there properly? >> >> There are solutions that I know work on Windows (from past work I've >> done). I'm unsure about either jemalloc and tcmalloc, however. >> >> Thanks, >> >> /Jeff >> >> -Original Message----- >> From: Benno Evers [mailto:bev...@mesosphere.com] >> Sent: Tuesday, August 22, 2017 3:16 AM >> To: dev@mesos.apache.org >> Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos >> >> Hi Alexander, >> >> in general, jemalloc and tcmalloc are very similar, and seem to be taking >> ideas from each other (in fact the jeprof executable started as a copy of >> pprof and there are still references the pprof documentation in some >> comments) >> >> From what I've seen, the main difference is that the profiling seems >> better-suited to multi-threaded programs, in particular the profile file >> format includes per-thread memory statistics and the profiling features can >> be turned on and off individually per thread. From an API perspective, all >> settings can be accessed by the mallctl() function, while it seems that >> tcmalloc requires some options to be set by environment variable ( >> https://na01.safelinks.protection.outlook.com/?url= >> https%3A%2F%2Fgperftools.github.io%2Fgperftools% >> 2Fheapprofile.html&data=02%7C01%7CJeff.Coffler%40microsoft.com% >> 7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011 >> db47%7C1%7C0%7C636389937852256730&sdata=IQeb2% >> 2BpcrWRQ8yvdTgOEHfyplgC36dy73nnXswdPamo%3D&reserved=0). Finally, I also >> found the documentation to be more thorough. >> >> But again, the two are very similar, so I think the main decision here >> isn't whether to choose jemalloc or tcmalloc but whether to switch to a >> custom memory allocator that has support for profiling heap memory usage. >> >> >> On Mon, Aug 21, 2017 at 4:26 PM, Alexander Rojas >> wrote: >> >>> Hi Benno, >>> >>> This does sound like a great addition to Mesos. Can you however >>> explain how jemalloc is better than tcmalloc? I think that for such >>> important change, we probably need some more information. >>> >>> Your comment in MESOS-7876 mentions that we already have tcmalloc >>> since it is part of gperftools, so I would like to have a whole >>> picture of the advantages and disadvantages of both options. >>> >>> Alexander Rojas >>> alexan...@mesosphere.io >>> >>> >>> >>> >>>> On 18. Aug 2017, at 12:49, Benno Evers wrote: >>>> >>>> Hi all, >>>> >>>> I would like to propose bundling jemalloc as a new dependency under >>>> `3rdparty/`, and to link Mesos against this new memory allocator by >>>> default. >>>> >>>> >>>> # Motivation >>>> >>>> The Mesos master and agent binaries are, ideally, very long-running >>>> processes. This makes them susceptible to memory issues, because >>>> even small leaks have a chance to build up over time to the point >>>> where they become problematic. >>>> >>>> We have seen several such issues on our internal Mesos >>>> installations, for example >>>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiss >>>> ues.apache.org%2Fjira%2Fbrowse%2FMESOS-7748&data=02%7C01%7CJeff.C
Re: [Proposal] Use jemalloc as default memory allocator for Mesos
It appears that jemalloc does support Windows (64bit) See: https://github.com/jemalloc/jemalloc/blob/dev/msvc/ReadMe.txt <https://github.com/jemalloc/jemalloc/blob/dev/msvc/ReadMe.txt> tcmalloc on the other hand appears to only offer a minimal variant on Windows. See: https://github.com/gperftools/gperftools <https://github.com/gperftools/gperftools> - grep for "COMPILING ON NON-LINUX SYSTEMS” See: https://github.com/gperftools/gperftools/blob/master/INSTALL <https://github.com/gperftools/gperftools/blob/master/INSTALL> - grep for “Windows (MSVC, Cygwin, and MinGW)" So both options rely on Cygwin or MinGW for building - a requirement that the Mesos build itself does not have and proves your point of stuff not really “just working” at least when it comes to the build step of those packages. Seems that without trying it, we won’t find out if jemalloc works as hoped on Windows for us - the Firefox project results however are encouraging. On the other hand, if it doesn’t work, we could simply decide to disable it on Windows just like some other Mesos features will remain disabled on that platform unless someone decides to port them - e.g. SASL based authn. > On Aug 25, 2017, at 3:28 PM, Benno Evers wrote: > > Hi Jeff, > > from looking around on the internet, it seems like Firefox builds with > jemalloc on all platforms, and I've also seen reports of people > successfully using tcmalloc heap profiling on windows. I'm afraid I don't > currently have a Windows machine with development environment set up, so I > can't provide direct user experience. > > In the worst case, we would have to disable jemalloc-support on windows, > but I hope it won't be necessary. > > Since you probably have more experience with memory management on windows, > is there a reason to suspect that it should or shouldn't work? > > Best regards, > Benno > > On Wed, Aug 23, 2017 at 6:16 PM, Jeff Coffler < > jeff.coff...@microsoft.com.invalid> wrote: > >> Hi Benno, >> >> What's the availability of both jemalloc and tcmalloc on the Windows >> platform? Do the products work there properly? >> >> There are solutions that I know work on Windows (from past work I've >> done). I'm unsure about either jemalloc and tcmalloc, however. >> >> Thanks, >> >> /Jeff >> >> -----Original Message- >> From: Benno Evers [mailto:bev...@mesosphere.com] >> Sent: Tuesday, August 22, 2017 3:16 AM >> To: dev@mesos.apache.org >> Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos >> >> Hi Alexander, >> >> in general, jemalloc and tcmalloc are very similar, and seem to be taking >> ideas from each other (in fact the jeprof executable started as a copy of >> pprof and there are still references the pprof documentation in some >> comments) >> >> From what I've seen, the main difference is that the profiling seems >> better-suited to multi-threaded programs, in particular the profile file >> format includes per-thread memory statistics and the profiling features can >> be turned on and off individually per thread. From an API perspective, all >> settings can be accessed by the mallctl() function, while it seems that >> tcmalloc requires some options to be set by environment variable ( >> https://na01.safelinks.protection.outlook.com/?url= >> https%3A%2F%2Fgperftools.github.io%2Fgperftools% >> 2Fheapprofile.html&data=02%7C01%7CJeff.Coffler%40microsoft.com% >> 7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011 >> db47%7C1%7C0%7C636389937852256730&sdata=IQeb2% >> 2BpcrWRQ8yvdTgOEHfyplgC36dy73nnXswdPamo%3D&reserved=0). Finally, I also >> found the documentation to be more thorough. >> >> But again, the two are very similar, so I think the main decision here >> isn't whether to choose jemalloc or tcmalloc but whether to switch to a >> custom memory allocator that has support for profiling heap memory usage. >> >> >> On Mon, Aug 21, 2017 at 4:26 PM, Alexander Rojas >> wrote: >> >>> Hi Benno, >>> >>> This does sound like a great addition to Mesos. Can you however >>> explain how jemalloc is better than tcmalloc? I think that for such >>> important change, we probably need some more information. >>> >>> Your comment in MESOS-7876 mentions that we already have tcmalloc >>> since it is part of gperftools, so I would like to have a whole >>> picture of the advantages and disadvantages of both options. >>> >>> Alexander Rojas >>> alexan...@mesosphere.io
RE: [Proposal] Use jemalloc as default memory allocator for Mesos
Hi Benno, These days, I never assume that a Linux package "just works" on Windows. I'm well aware (all too much) of just how different Windows is from Linux, and thus how difficult it can be to write software that works on both platforms. So unless the author of the memory management software (deployed on Linux) happened to consider Windows important enough to support, I'd assume that Windows is not supported (and would almost certainly not "just work" by chance). I really asked my question because Mesos is intended to be a portable memory package. Thus, if we pick up a new third party dependency that would affect the Mesos agent, we should pick third party dependencies that work on Windows, Mac, and other platforms. This is something we should be thinking about. /Jeff -Original Message- From: Benno Evers [mailto:bev...@mesosphere.com] Sent: Friday, August 25, 2017 6:28 AM To: dev@mesos.apache.org Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos Hi Jeff, from looking around on the internet, it seems like Firefox builds with jemalloc on all platforms, and I've also seen reports of people successfully using tcmalloc heap profiling on windows. I'm afraid I don't currently have a Windows machine with development environment set up, so I can't provide direct user experience. In the worst case, we would have to disable jemalloc-support on windows, but I hope it won't be necessary. Since you probably have more experience with memory management on windows, is there a reason to suspect that it should or shouldn't work? Best regards, Benno On Wed, Aug 23, 2017 at 6:16 PM, Jeff Coffler < jeff.coff...@microsoft.com.invalid> wrote: > Hi Benno, > > What's the availability of both jemalloc and tcmalloc on the Windows > platform? Do the products work there properly? > > There are solutions that I know work on Windows (from past work I've > done). I'm unsure about either jemalloc and tcmalloc, however. > > Thanks, > > /Jeff > > -Original Message- > From: Benno Evers [mailto:bev...@mesosphere.com] > Sent: Tuesday, August 22, 2017 3:16 AM > To: dev@mesos.apache.org > Subject: Re: [Proposal] Use jemalloc as default memory allocator for > Mesos > > Hi Alexander, > > in general, jemalloc and tcmalloc are very similar, and seem to be > taking ideas from each other (in fact the jeprof executable started as > a copy of pprof and there are still references the pprof documentation > in some > comments) > > From what I've seen, the main difference is that the profiling seems > better-suited to multi-threaded programs, in particular the profile > file format includes per-thread memory statistics and the profiling > features can be turned on and off individually per thread. From an API > perspective, all settings can be accessed by the mallctl() function, > while it seems that tcmalloc requires some options to be set by > environment variable ( > https://na01.safelinks.protection.outlook.com/?url= > https%3A%2F%2Fgperftools.github.io%2Fgperftools% > 2Fheapprofile.html&data=02%7C01%7CJeff.Coffler%40microsoft.com% > 7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011 > db47%7C1%7C0%7C636389937852256730&sdata=IQeb2% > 2BpcrWRQ8yvdTgOEHfyplgC36dy73nnXswdPamo%3D&reserved=0). Finally, I > also found the documentation to be more thorough. > > But again, the two are very similar, so I think the main decision here > isn't whether to choose jemalloc or tcmalloc but whether to switch to > a custom memory allocator that has support for profiling heap memory usage. > > > On Mon, Aug 21, 2017 at 4:26 PM, Alexander Rojas > > wrote: > > > Hi Benno, > > > > This does sound like a great addition to Mesos. Can you however > > explain how jemalloc is better than tcmalloc? I think that for such > > important change, we probably need some more information. > > > > Your comment in MESOS-7876 mentions that we already have tcmalloc > > since it is part of gperftools, so I would like to have a whole > > picture of the advantages and disadvantages of both options. > > > > Alexander Rojas > > alexan...@mesosphere.io > > > > > > > > > > > On 18. Aug 2017, at 12:49, Benno Evers wrote: > > > > > > Hi all, > > > > > > I would like to propose bundling jemalloc as a new dependency > > > under `3rdparty/`, and to link Mesos against this new memory > > > allocator by default. > > > > > > > > > # Motivation > > > > > > The Mesos master and agent binaries are, ideally, very > > > long-running
Re: [Proposal] Use jemalloc as default memory allocator for Mesos
Hi Jeff, from looking around on the internet, it seems like Firefox builds with jemalloc on all platforms, and I've also seen reports of people successfully using tcmalloc heap profiling on windows. I'm afraid I don't currently have a Windows machine with development environment set up, so I can't provide direct user experience. In the worst case, we would have to disable jemalloc-support on windows, but I hope it won't be necessary. Since you probably have more experience with memory management on windows, is there a reason to suspect that it should or shouldn't work? Best regards, Benno On Wed, Aug 23, 2017 at 6:16 PM, Jeff Coffler < jeff.coff...@microsoft.com.invalid> wrote: > Hi Benno, > > What's the availability of both jemalloc and tcmalloc on the Windows > platform? Do the products work there properly? > > There are solutions that I know work on Windows (from past work I've > done). I'm unsure about either jemalloc and tcmalloc, however. > > Thanks, > > /Jeff > > -Original Message- > From: Benno Evers [mailto:bev...@mesosphere.com] > Sent: Tuesday, August 22, 2017 3:16 AM > To: dev@mesos.apache.org > Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos > > Hi Alexander, > > in general, jemalloc and tcmalloc are very similar, and seem to be taking > ideas from each other (in fact the jeprof executable started as a copy of > pprof and there are still references the pprof documentation in some > comments) > > From what I've seen, the main difference is that the profiling seems > better-suited to multi-threaded programs, in particular the profile file > format includes per-thread memory statistics and the profiling features can > be turned on and off individually per thread. From an API perspective, all > settings can be accessed by the mallctl() function, while it seems that > tcmalloc requires some options to be set by environment variable ( > https://na01.safelinks.protection.outlook.com/?url= > https%3A%2F%2Fgperftools.github.io%2Fgperftools% > 2Fheapprofile.html&data=02%7C01%7CJeff.Coffler%40microsoft.com% > 7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011 > db47%7C1%7C0%7C636389937852256730&sdata=IQeb2% > 2BpcrWRQ8yvdTgOEHfyplgC36dy73nnXswdPamo%3D&reserved=0). Finally, I also > found the documentation to be more thorough. > > But again, the two are very similar, so I think the main decision here > isn't whether to choose jemalloc or tcmalloc but whether to switch to a > custom memory allocator that has support for profiling heap memory usage. > > > On Mon, Aug 21, 2017 at 4:26 PM, Alexander Rojas > wrote: > > > Hi Benno, > > > > This does sound like a great addition to Mesos. Can you however > > explain how jemalloc is better than tcmalloc? I think that for such > > important change, we probably need some more information. > > > > Your comment in MESOS-7876 mentions that we already have tcmalloc > > since it is part of gperftools, so I would like to have a whole > > picture of the advantages and disadvantages of both options. > > > > Alexander Rojas > > alexan...@mesosphere.io > > > > > > > > > > > On 18. Aug 2017, at 12:49, Benno Evers wrote: > > > > > > Hi all, > > > > > > I would like to propose bundling jemalloc as a new dependency under > > > `3rdparty/`, and to link Mesos against this new memory allocator by > > > default. > > > > > > > > > # Motivation > > > > > > The Mesos master and agent binaries are, ideally, very long-running > > > processes. This makes them susceptible to memory issues, because > > > even small leaks have a chance to build up over time to the point > > > where they become problematic. > > > > > > We have seen several such issues on our internal Mesos > > > installations, for example > > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiss > > > ues.apache.org%2Fjira%2Fbrowse%2FMESOS-7748&data=02%7C01%7CJeff.Coff > > > ler%40microsoft.com%7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f > > > 141af91ab2d7cd011db47%7C1%7C0%7C636389937852266742&sdata=L016YGyEkK5 > > > 0WtvhgSNS%2FT5ntkkd9qINorRI2Utp5lk%3D&reserved=0 > > > or https://na01.safelinks.protection.outlook.com/?url= > https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FMESOS- > 7800&data=02%7C01%7CJeff.Coffler%40microsoft.com% > 7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011 > db47%7C1%7C0%7C636389937852266742&sdata=IrzDO6o1VL9a8eGJIW3jKbWXk6U4fH > Fn3Xbn4po1r3c%3D&re
RE: [Proposal] Use jemalloc as default memory allocator for Mesos
Hi Benno, What's the availability of both jemalloc and tcmalloc on the Windows platform? Do the products work there properly? There are solutions that I know work on Windows (from past work I've done). I'm unsure about either jemalloc and tcmalloc, however. Thanks, /Jeff -Original Message- From: Benno Evers [mailto:bev...@mesosphere.com] Sent: Tuesday, August 22, 2017 3:16 AM To: dev@mesos.apache.org Subject: Re: [Proposal] Use jemalloc as default memory allocator for Mesos Hi Alexander, in general, jemalloc and tcmalloc are very similar, and seem to be taking ideas from each other (in fact the jeprof executable started as a copy of pprof and there are still references the pprof documentation in some comments) From what I've seen, the main difference is that the profiling seems better-suited to multi-threaded programs, in particular the profile file format includes per-thread memory statistics and the profiling features can be turned on and off individually per thread. From an API perspective, all settings can be accessed by the mallctl() function, while it seems that tcmalloc requires some options to be set by environment variable ( https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgperftools.github.io%2Fgperftools%2Fheapprofile.html&data=02%7C01%7CJeff.Coffler%40microsoft.com%7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636389937852256730&sdata=IQeb2%2BpcrWRQ8yvdTgOEHfyplgC36dy73nnXswdPamo%3D&reserved=0). Finally, I also found the documentation to be more thorough. But again, the two are very similar, so I think the main decision here isn't whether to choose jemalloc or tcmalloc but whether to switch to a custom memory allocator that has support for profiling heap memory usage. On Mon, Aug 21, 2017 at 4:26 PM, Alexander Rojas wrote: > Hi Benno, > > This does sound like a great addition to Mesos. Can you however > explain how jemalloc is better than tcmalloc? I think that for such > important change, we probably need some more information. > > Your comment in MESOS-7876 mentions that we already have tcmalloc > since it is part of gperftools, so I would like to have a whole > picture of the advantages and disadvantages of both options. > > Alexander Rojas > alexan...@mesosphere.io > > > > > > On 18. Aug 2017, at 12:49, Benno Evers wrote: > > > > Hi all, > > > > I would like to propose bundling jemalloc as a new dependency under > > `3rdparty/`, and to link Mesos against this new memory allocator by > > default. > > > > > > # Motivation > > > > The Mesos master and agent binaries are, ideally, very long-running > > processes. This makes them susceptible to memory issues, because > > even small leaks have a chance to build up over time to the point > > where they become problematic. > > > > We have seen several such issues on our internal Mesos > > installations, for example > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiss > > ues.apache.org%2Fjira%2Fbrowse%2FMESOS-7748&data=02%7C01%7CJeff.Coff > > ler%40microsoft.com%7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f > > 141af91ab2d7cd011db47%7C1%7C0%7C636389937852266742&sdata=L016YGyEkK5 > > 0WtvhgSNS%2FT5ntkkd9qINorRI2Utp5lk%3D&reserved=0 > > or > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FMESOS-7800&data=02%7C01%7CJeff.Coffler%40microsoft.com%7Ccb0bfb1eb3e242c0dd4108d4e946d709%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636389937852266742&sdata=IrzDO6o1VL9a8eGJIW3jKbWXk6U4fHFn3Xbn4po1r3c%3D&reserved=0. > > > > I imagine any organization running Mesos for an extended period of > > time has had its share of similar issues, so I expect this proposal > > to be useful for the whole community. > > > > > > # Why jemalloc? > > > > Given that memory issues tend to be most visible after a given > > process has been running for a long time, it would be great to have > > the option to enable heap tracking and profiling at runtime, without > > having to restart the process. (This ability could then be connected > > to a Mesos endpoint, similar to how we can adjust the log level at > > runtime) > > > > The two production-quality memory allocators that have this ability > > currently seem to be tcmalloc and jemalloc. Of these, jemalloc does > > produce in our experience better and more detailed statistics. > > > > > > # What is the impact on users who do not need this feature? > > > > Naturally, not every single user of Mesos will have a need for this > > feature. To ensur
Re: [Proposal] Use jemalloc as default memory allocator for Mesos
Hi Alexander, in general, jemalloc and tcmalloc are very similar, and seem to be taking ideas from each other (in fact the jeprof executable started as a copy of pprof and there are still references the pprof documentation in some comments) >From what I've seen, the main difference is that the profiling seems better-suited to multi-threaded programs, in particular the profile file format includes per-thread memory statistics and the profiling features can be turned on and off individually per thread. From an API perspective, all settings can be accessed by the mallctl() function, while it seems that tcmalloc requires some options to be set by environment variable ( https://gperftools.github.io/gperftools/heapprofile.html). Finally, I also found the documentation to be more thorough. But again, the two are very similar, so I think the main decision here isn't whether to choose jemalloc or tcmalloc but whether to switch to a custom memory allocator that has support for profiling heap memory usage. On Mon, Aug 21, 2017 at 4:26 PM, Alexander Rojas wrote: > Hi Benno, > > This does sound like a great addition to Mesos. Can you however explain > how jemalloc is better than tcmalloc? I think that for such important > change, we probably need some more information. > > Your comment in MESOS-7876 mentions that we already have tcmalloc since it > is part of gperftools, so I would like to have a whole picture of the > advantages and disadvantages of both options. > > Alexander Rojas > alexan...@mesosphere.io > > > > > > On 18. Aug 2017, at 12:49, Benno Evers wrote: > > > > Hi all, > > > > I would like to propose bundling jemalloc as a new dependency > > under `3rdparty/`, and to link Mesos against this new memory > > allocator by default. > > > > > > # Motivation > > > > The Mesos master and agent binaries are, ideally, very long-running > > processes. This makes them susceptible to memory issues, because > > even small leaks have a chance to build up over time to the point > > where they become problematic. > > > > We have seen several such issues on our internal Mesos installations, > > for example https://issues.apache.org/jira/browse/MESOS-7748 > > or https://issues.apache.org/jira/browse/MESOS-7800. > > > > I imagine any organization running Mesos for an extended period > > of time has had its share of similar issues, so I expect this > > proposal to be useful for the whole community. > > > > > > # Why jemalloc? > > > > Given that memory issues tend to be most visible after a given > > process has been running for a long time, it would be great to > > have the option to enable heap tracking and profiling at runtime, > > without having to restart the process. (This ability could then > > be connected to a Mesos endpoint, similar to how we can adjust > > the log level at runtime) > > > > The two production-quality memory allocators that have this > > ability currently seem to be tcmalloc and jemalloc. Of these, > > jemalloc does produce in our experience better and more > > detailed statistics. > > > > > > # What is the impact on users who do not need this feature? > > > > Naturally, not every single user of Mesos will have a need > > for this feature. To ensure these users would not experience serious > > performance regressions as a result of this change, we conducted > > a preliminary set of benchmarks whose results are collected > > under https://issues.apache.org/jira/browse/MESOS-7876 > > > > It turns out that we could probably even expect a small speedup (1% - 5%) > > as a nice side-effect of this change. > > > > Users who compile Mesos themselves would of course have the option > > to disable jemalloc at configuration time or replace it with their > > memory allocator of choice. > > > > > > > > I'm looking forward to hear any thoughts and comments. > > > > > > Thanks, > > -- > > Benno Evers > > Software Engineer, Mesosphere > > -- Benno Evers Software Engineer, Mesosphere
Re: [Proposal] Use jemalloc as default memory allocator for Mesos
Hi Benno, This does sound like a great addition to Mesos. Can you however explain how jemalloc is better than tcmalloc? I think that for such important change, we probably need some more information. Your comment in MESOS-7876 mentions that we already have tcmalloc since it is part of gperftools, so I would like to have a whole picture of the advantages and disadvantages of both options. Alexander Rojas alexan...@mesosphere.io > On 18. Aug 2017, at 12:49, Benno Evers wrote: > > Hi all, > > I would like to propose bundling jemalloc as a new dependency > under `3rdparty/`, and to link Mesos against this new memory > allocator by default. > > > # Motivation > > The Mesos master and agent binaries are, ideally, very long-running > processes. This makes them susceptible to memory issues, because > even small leaks have a chance to build up over time to the point > where they become problematic. > > We have seen several such issues on our internal Mesos installations, > for example https://issues.apache.org/jira/browse/MESOS-7748 > or https://issues.apache.org/jira/browse/MESOS-7800. > > I imagine any organization running Mesos for an extended period > of time has had its share of similar issues, so I expect this > proposal to be useful for the whole community. > > > # Why jemalloc? > > Given that memory issues tend to be most visible after a given > process has been running for a long time, it would be great to > have the option to enable heap tracking and profiling at runtime, > without having to restart the process. (This ability could then > be connected to a Mesos endpoint, similar to how we can adjust > the log level at runtime) > > The two production-quality memory allocators that have this > ability currently seem to be tcmalloc and jemalloc. Of these, > jemalloc does produce in our experience better and more > detailed statistics. > > > # What is the impact on users who do not need this feature? > > Naturally, not every single user of Mesos will have a need > for this feature. To ensure these users would not experience serious > performance regressions as a result of this change, we conducted > a preliminary set of benchmarks whose results are collected > under https://issues.apache.org/jira/browse/MESOS-7876 > > It turns out that we could probably even expect a small speedup (1% - 5%) > as a nice side-effect of this change. > > Users who compile Mesos themselves would of course have the option > to disable jemalloc at configuration time or replace it with their > memory allocator of choice. > > > > I'm looking forward to hear any thoughts and comments. > > > Thanks, > -- > Benno Evers > Software Engineer, Mesosphere
Re: [Proposal] Use jemalloc as default memory allocator for Mesos
I'm for making jemalloc default as well with adding an opt-out option to CMake and autotools build scripts. On Sat, Aug 19, 2017 at 3:23 AM, Benjamin Mahler wrote: > This will be a big win Benno, thanks for driving it! > > Nice to see that the heap profiling overhead is really low, I'd love to be > able to just hit an endpoint on the master or agent and get a memory > profile. > > I'm a +1 for making it the default, however, I seem to recall hearing that > there were some issues with JNI? > > Ben > > On Fri, Aug 18, 2017 at 3:49 AM, Benno Evers > wrote: > > > Hi all, > > > > I would like to propose bundling jemalloc as a new dependency > > under `3rdparty/`, and to link Mesos against this new memory > > allocator by default. > > > > > > # Motivation > > > > The Mesos master and agent binaries are, ideally, very long-running > > processes. This makes them susceptible to memory issues, because > > even small leaks have a chance to build up over time to the point > > where they become problematic. > > > > We have seen several such issues on our internal Mesos installations, > > for example https://issues.apache.org/jira/browse/MESOS-7748 > > or https://issues.apache.org/jira/browse/MESOS-7800. > > > > I imagine any organization running Mesos for an extended period > > of time has had its share of similar issues, so I expect this > > proposal to be useful for the whole community. > > > > > > # Why jemalloc? > > > > Given that memory issues tend to be most visible after a given > > process has been running for a long time, it would be great to > > have the option to enable heap tracking and profiling at runtime, > > without having to restart the process. (This ability could then > > be connected to a Mesos endpoint, similar to how we can adjust > > the log level at runtime) > > > > The two production-quality memory allocators that have this > > ability currently seem to be tcmalloc and jemalloc. Of these, > > jemalloc does produce in our experience better and more > > detailed statistics. > > > > > > # What is the impact on users who do not need this feature? > > > > Naturally, not every single user of Mesos will have a need > > for this feature. To ensure these users would not experience serious > > performance regressions as a result of this change, we conducted > > a preliminary set of benchmarks whose results are collected > > under https://issues.apache.org/jira/browse/MESOS-7876 > > > > It turns out that we could probably even expect a small speedup (1% - 5%) > > as a nice side-effect of this change. > > > > Users who compile Mesos themselves would of course have the option > > to disable jemalloc at configuration time or replace it with their > > memory allocator of choice. > > > > > > > > I'm looking forward to hear any thoughts and comments. > > > > > > Thanks, > > -- > > Benno Evers > > Software Engineer, Mesosphere > > >
Re: [Proposal] Use jemalloc as default memory allocator for Mesos
This will be a big win Benno, thanks for driving it! Nice to see that the heap profiling overhead is really low, I'd love to be able to just hit an endpoint on the master or agent and get a memory profile. I'm a +1 for making it the default, however, I seem to recall hearing that there were some issues with JNI? Ben On Fri, Aug 18, 2017 at 3:49 AM, Benno Evers wrote: > Hi all, > > I would like to propose bundling jemalloc as a new dependency > under `3rdparty/`, and to link Mesos against this new memory > allocator by default. > > > # Motivation > > The Mesos master and agent binaries are, ideally, very long-running > processes. This makes them susceptible to memory issues, because > even small leaks have a chance to build up over time to the point > where they become problematic. > > We have seen several such issues on our internal Mesos installations, > for example https://issues.apache.org/jira/browse/MESOS-7748 > or https://issues.apache.org/jira/browse/MESOS-7800. > > I imagine any organization running Mesos for an extended period > of time has had its share of similar issues, so I expect this > proposal to be useful for the whole community. > > > # Why jemalloc? > > Given that memory issues tend to be most visible after a given > process has been running for a long time, it would be great to > have the option to enable heap tracking and profiling at runtime, > without having to restart the process. (This ability could then > be connected to a Mesos endpoint, similar to how we can adjust > the log level at runtime) > > The two production-quality memory allocators that have this > ability currently seem to be tcmalloc and jemalloc. Of these, > jemalloc does produce in our experience better and more > detailed statistics. > > > # What is the impact on users who do not need this feature? > > Naturally, not every single user of Mesos will have a need > for this feature. To ensure these users would not experience serious > performance regressions as a result of this change, we conducted > a preliminary set of benchmarks whose results are collected > under https://issues.apache.org/jira/browse/MESOS-7876 > > It turns out that we could probably even expect a small speedup (1% - 5%) > as a nice side-effect of this change. > > Users who compile Mesos themselves would of course have the option > to disable jemalloc at configuration time or replace it with their > memory allocator of choice. > > > > I'm looking forward to hear any thoughts and comments. > > > Thanks, > -- > Benno Evers > Software Engineer, Mesosphere >
Re: [Proposal] Use jemalloc as default memory allocator for Mesos
Strongly in favor of this idea and willing to try it once it's available. On Fri, Aug 18, 2017 at 8:37 AM, James Peach wrote: > > > On Aug 18, 2017, at 3:49 AM, Benno Evers wrote: > > > > Hi all, > > > > I would like to propose bundling jemalloc as a new dependency > > under `3rdparty/`, and to link Mesos against this new memory > > allocator by default. > > I support doing this for all the Mesos executable programs. We have been > running under jemalloc for a couple of years with zero problems and > improved performance (we run a pretty old glibc). > > Note that we must not lib libmesos.so to jemalloc since that is used by > programs that may not be able to tolerate linking in a separate malloc. > > > # Motivation > > > > The Mesos master and agent binaries are, ideally, very long-running > > processes. This makes them susceptible to memory issues, because > > even small leaks have a chance to build up over time to the point > > where they become problematic. > > > > We have seen several such issues on our internal Mesos installations, > > for example https://issues.apache.org/jira/browse/MESOS-7748 > > or https://issues.apache.org/jira/browse/MESOS-7800. > > > > I imagine any organization running Mesos for an extended period > > of time has had its share of similar issues, so I expect this > > proposal to be useful for the whole community. > > > > > > # Why jemalloc? > > > > Given that memory issues tend to be most visible after a given > > process has been running for a long time, it would be great to > > have the option to enable heap tracking and profiling at runtime, > > without having to restart the process. (This ability could then > > be connected to a Mesos endpoint, similar to how we can adjust > > the log level at runtime) > > > > The two production-quality memory allocators that have this > > ability currently seem to be tcmalloc and jemalloc. Of these, > > jemalloc does produce in our experience better and more > > detailed statistics. > > > > > > # What is the impact on users who do not need this feature? > > > > Naturally, not every single user of Mesos will have a need > > for this feature. To ensure these users would not experience serious > > performance regressions as a result of this change, we conducted > > a preliminary set of benchmarks whose results are collected > > under https://issues.apache.org/jira/browse/MESOS-7876 > > > > It turns out that we could probably even expect a small speedup (1% - 5%) > > as a nice side-effect of this change. > > > > Users who compile Mesos themselves would of course have the option > > to disable jemalloc at configuration time or replace it with their > > memory allocator of choice. > > > > > > > > I'm looking forward to hear any thoughts and comments. > > > > > > Thanks, > > -- > > Benno Evers > > Software Engineer, Mesosphere > > -- Cheers, Zhitao Li
Re: [Proposal] Use jemalloc as default memory allocator for Mesos
> On Aug 18, 2017, at 3:49 AM, Benno Evers wrote: > > Hi all, > > I would like to propose bundling jemalloc as a new dependency > under `3rdparty/`, and to link Mesos against this new memory > allocator by default. I support doing this for all the Mesos executable programs. We have been running under jemalloc for a couple of years with zero problems and improved performance (we run a pretty old glibc). Note that we must not lib libmesos.so to jemalloc since that is used by programs that may not be able to tolerate linking in a separate malloc. > # Motivation > > The Mesos master and agent binaries are, ideally, very long-running > processes. This makes them susceptible to memory issues, because > even small leaks have a chance to build up over time to the point > where they become problematic. > > We have seen several such issues on our internal Mesos installations, > for example https://issues.apache.org/jira/browse/MESOS-7748 > or https://issues.apache.org/jira/browse/MESOS-7800. > > I imagine any organization running Mesos for an extended period > of time has had its share of similar issues, so I expect this > proposal to be useful for the whole community. > > > # Why jemalloc? > > Given that memory issues tend to be most visible after a given > process has been running for a long time, it would be great to > have the option to enable heap tracking and profiling at runtime, > without having to restart the process. (This ability could then > be connected to a Mesos endpoint, similar to how we can adjust > the log level at runtime) > > The two production-quality memory allocators that have this > ability currently seem to be tcmalloc and jemalloc. Of these, > jemalloc does produce in our experience better and more > detailed statistics. > > > # What is the impact on users who do not need this feature? > > Naturally, not every single user of Mesos will have a need > for this feature. To ensure these users would not experience serious > performance regressions as a result of this change, we conducted > a preliminary set of benchmarks whose results are collected > under https://issues.apache.org/jira/browse/MESOS-7876 > > It turns out that we could probably even expect a small speedup (1% - 5%) > as a nice side-effect of this change. > > Users who compile Mesos themselves would of course have the option > to disable jemalloc at configuration time or replace it with their > memory allocator of choice. > > > > I'm looking forward to hear any thoughts and comments. > > > Thanks, > -- > Benno Evers > Software Engineer, Mesosphere
[Proposal] Use jemalloc as default memory allocator for Mesos
Hi all, I would like to propose bundling jemalloc as a new dependency under `3rdparty/`, and to link Mesos against this new memory allocator by default. # Motivation The Mesos master and agent binaries are, ideally, very long-running processes. This makes them susceptible to memory issues, because even small leaks have a chance to build up over time to the point where they become problematic. We have seen several such issues on our internal Mesos installations, for example https://issues.apache.org/jira/browse/MESOS-7748 or https://issues.apache.org/jira/browse/MESOS-7800. I imagine any organization running Mesos for an extended period of time has had its share of similar issues, so I expect this proposal to be useful for the whole community. # Why jemalloc? Given that memory issues tend to be most visible after a given process has been running for a long time, it would be great to have the option to enable heap tracking and profiling at runtime, without having to restart the process. (This ability could then be connected to a Mesos endpoint, similar to how we can adjust the log level at runtime) The two production-quality memory allocators that have this ability currently seem to be tcmalloc and jemalloc. Of these, jemalloc does produce in our experience better and more detailed statistics. # What is the impact on users who do not need this feature? Naturally, not every single user of Mesos will have a need for this feature. To ensure these users would not experience serious performance regressions as a result of this change, we conducted a preliminary set of benchmarks whose results are collected under https://issues.apache.org/jira/browse/MESOS-7876 It turns out that we could probably even expect a small speedup (1% - 5%) as a nice side-effect of this change. Users who compile Mesos themselves would of course have the option to disable jemalloc at configuration time or replace it with their memory allocator of choice. I'm looking forward to hear any thoughts and comments. Thanks, -- Benno Evers Software Engineer, Mesosphere
