On Tue, Dec 06, 2005 at 11:07:44AM -0500, Brian Barrett wrote:
> On Dec 6, 2005, at 10:53 AM, Gleb Natapov wrote:
>
> > On Tue, Dec 06, 2005 at 08:33:32AM -0700, Tim S. Woodall wrote:
> >>> Also memfree hooks decrease cache efficiency, the better solution
> >>> would
> >>> be to catch brk() system calls and remove memory from cache only
> >>> then,
> >>> but there is no way to do it for now.
> >>
> >> We are look at other options, including catching brk/munmap system
> >> calls, and
> >> will be experimenting w/ these on the trunk.
> >>
> > This will be really interesting. How are you going to catch brk/munmap
> > without kernel help? Last time I checked preload tricks don't work if
> > syscall is done from inside libc itself.
>
> All of the tricks we are looking at assume that nothing in libc calls
> munmap.
glibc does call mmap/munmap internally for big allocations as strace of
this program shows:
int main ()
{
void *p = malloc (1024*1024);
free (p);
}
> We can successfully catch free() calls from inside libc
> without any problems. The LAM/MPI team and Myricom (with MPICH-gm)
> have been doing this for many years without any problems. On the
> small percentage of MPI applications that require some linker tricks
> (some of the commercial apps are this way), we won't be able to
> intercept any free/munmap calls, so we're going to fall back to our
> RDMA pipeline algorithm.
>
Yes, but catching free is not good enough. This way we sometimes evict
cache entries that may safely remains in the cache. Ideally we should be
able to catch events that return memory to OS (munmap/brk) and remove the
memory from cache only then.
--
Gleb.
