Re: [OMPI devel] MALLOC_MMAP_MAX (and MALLOC_MMAP_THRESHOLD)
On 10 Jan 2010, at 03:45, Barrett, Brian W wrote: > We should absolutely not change this. For simple applications, yes, things > work if large blocks are allocated on the heap. However, ptmalloc (and most > allocators, really), can't rationally cope with repeated allocations and > deallocations of large blocks. It would be *really bad* (as we've seen > before) to change the behavior of our version of ptmalloc from that which is > provided by Linux. Pain and suffering is all that path has ever lead to. I think there are two different issues here that are in danger of being confused. Having OMPI use it's own ptmalloc implantation as it has done at times in the past has been problematic and should be avoided in my opinion. Tuning the libc malloc implementation using the options they provide to do is is valid and provides real benefit to a lot of applications. For the record we used to disable mmap based allocations by default on Quadrics systems and I can't think of a single case of people who needed to re-enable it. Ashley.
Re: [OMPI devel] MALLOC_MMAP_MAX (and MALLOC_MMAP_THRESHOLD)
On 11 Jan 2010, at 16:52, Jeff Squyres wrote: > Arrgh -- if only the Linux kernel community had accepted ummunotify, this > would now be a moot point (i.e., the argument would be solely with the > OS/glibc, not the MPI!). Disabling MMAP based malloc is purely about performance, ummunotify is about correctness and is a much more complex issue. Ashley,
Re: [OMPI devel] MALLOC_MMAP_MAX (and MALLOC_MMAP_THRESHOLD)
On Jan 17, 2010, at 11:31 AM, Ashley Pittman wrote: > On 10 Jan 2010, at 03:45, Barrett, Brian W wrote: > >> We should absolutely not change this. For simple applications, yes, things >> work if large blocks are allocated on the heap. However, ptmalloc (and most >> allocators, really), can't rationally cope with repeated allocations and >> deallocations of large blocks. It would be *really bad* (as we've seen >> before) to change the behavior of our version of ptmalloc from that which is >> provided by Linux. Pain and suffering is all that path has ever lead to. > > I think there are two different issues here that are in danger of being > confused. Having OMPI use it's own ptmalloc implantation as it has done at > times in the past has been problematic and should be avoided in my opinion. I agree, and I also don't believe I'm confusing them. The shipping of ptmalloc2 is unfortunate and yet required until a better solution appears. The proposal/request which started this e-mail chain was to change a hard-coded constant in the ptmalloc2 shipped with Open MPI. While the fact that we can do this is an artifact of Open MPI providing it's own ptmalloc implementation, it none the less fundamentally changes how the memory allocator works when applications are linked against Open MPI, away from the default behavior found on Linux systems. > Tuning the libc malloc implementation using the options they provide to do is > is valid and provides real benefit to a lot of applications. For the record > we used to disable mmap based allocations by default on Quadrics systems and > I can't think of a single case of people who needed to re-enable it. I'm glad that you've been so fortunate. Unfortunately, I have a large application base in which that is not the case, and we have had to use mmap based allocation, otherwise memory usage would essentially grow without bound. So we go back to the age-old debate of correctness vs. performance. Brian
Re: [OMPI devel] MALLOC_MMAP_MAX (and MALLOC_MMAP_THRESHOLD)
On 17 Jan 2010, at 16:50, Barrett, Brian W wrote: > On Jan 17, 2010, at 11:31 AM, Ashley Pittman wrote: > >> On 10 Jan 2010, at 03:45, Barrett, Brian W wrote: >> >>> We should absolutely not change this. For simple applications, yes, things >>> work if large blocks are allocated on the heap. However, ptmalloc (and >>> most allocators, really), can't rationally cope with repeated allocations >>> and deallocations of large blocks. It would be *really bad* (as we've seen >>> before) to change the behavior of our version of ptmalloc from that which >>> is provided by Linux. Pain and suffering is all that path has ever lead to. >> >> I think there are two different issues here that are in danger of being >> confused. Having OMPI use it's own ptmalloc implantation as it has done at >> times in the past has been problematic and should be avoided in my opinion. > > I agree, and I also don't believe I'm confusing them. The shipping of > ptmalloc2 is unfortunate and yet required until a better solution appears. > The proposal/request which started this e-mail chain was to change a > hard-coded constant in the ptmalloc2 shipped with Open MPI. I was under the impression that OMPI had moved away from shipping it's own allocator by default although I don't follow that closely. Ultimately whatever the default there needs to be a way for the end-user to change the setting. > While the fact that we can do this is an artifact of Open MPI providing it's > own ptmalloc implementation. No it isn't, it's also standard libc option. It was this that we set by default at Quadrics but the mallopt() function. > it none the less fundamentally changes how the memory allocator works when > applications are linked against Open MPI, away from the default behavior > found on Linux systems. This isn't necessarily a bad thing, default behaviour for a distro has to be something that works well for all environments that use that distro, HPC is something very specialised so there is no reason to believe the defaults are good choices for HPC apps. >> Tuning the libc malloc implementation using the options they provide to do >> is is valid and provides real benefit to a lot of applications. For the >> record we used to disable mmap based allocations by default on Quadrics >> systems and I can't think of a single case of people who needed to re-enable >> it. > > I'm glad that you've been so fortunate. Unfortunately, I have a large > application base in which that is not the case, and we have had to use mmap > based allocation, otherwise memory usage would essentially grow without > bound. So we go back to the age-old debate of correctness vs. performance. It is something bear in mind when analysing application performance and diagnosing memory problems with applications, a FAQ entry should be considered the minimum IMHO. Ashley.
