On Tue, Apr 28, 2015 at 5:26 AM, Barry Smith <bsm...@mcs.anl.gov> wrote:
> > > On Apr 27, 2015, at 1:51 PM, Richard Mills <r...@utk.edu> wrote: > > > > All, > > > > I'd like to add support for the allocators provided by the 'memkind' > library (https://github.com/memkind/memkind). I've discussed memkind a > little bit with some of you off-list. Briefly, memkind provides a "user > extensible heap manager built on top of jemalloc which enables control of > memory characteristics and a partitioning of the heap between kinds of > memory". The immediate motivation is to support placement of critical data > structures into the high bandwidth on-package memory that will be available > with Intel's "Knights Landing" generation of Xeon Phi processor (like on > the upcoming NERSC Cori machine), but what the library provides is more > general, and it can also be used for placing data in memory such as > nonvolatile RAM (NVRAM), which will be appearing in more systems. > > > > I'm with Jed in thinking that, ideally, PETSc (or its users) shouldn't > have to make decisions about the optimal way to solve the "packing problem" > of what should go into high-bandwidth memory. (In fact, I think this is a > really interesting research problem that relates to some work on > memory-adaptation in scientific applications that I did back when I was > doing my Ph.D. research, e.g., > http://www.climatemodeling.org/~rmills/pubs/JGC_mmlib_2007.pdf.) > However, right now I'd like to take the baby step of coming up with a > mechanism to simply tag PETSc objects with a kind of memory that is > preferred, and then having associated allocations reflect that preference > (or requirement, if the user wants allocations to fail if such memory is > not available). Later we can worry about how to move data structures in > and out of a kind of memory. > > > > It might make sense to add an option for certain PETSc classes--Mat and > Vec are the most obvious here--to prefer allocations in a certain kind of > memory. Or, would it make more sense to add such an option at the > PetscObject level? > > > > I think it is possible to add the memkind support without breaking all > of the interfaces used throughout PETSc for PetscMalloc(), etc. > > I don't think having this as a goal is useful at all! Just break the > current interface; add an abstract "memkind" argument to all PetscMalloc() > and Free() calls that indicates any additional information about the memory > requested. By making it "abstract" it will always just be there and on > systems without any special memory options it just doesn't do anything. > Since Malloc() is so pervasive, I think it would be useful to have a 2 level interface here. The standard Malloc() would call you advanced PlacedMalloc(), and anyone could call that function, but I think its just cruel to make everyone allocating memory give arguments they do not understand or need. Matt > Barry > > Note: it is not clear to me how this could be helpful on its own > because I agree with Jed how is the user when creating the object supposed > to know the optimal place to put the object? For more complex objects it > may be that different parts of the object would be stored in different > types of memory etc etc. > > > > I recently sat with Chris Cantalupo, the main memkind developer, and > walked him through PETSc's allocation routines, and we came up with the > following: The imalloc() function pointer could have an implementation > something like > > > > PetcErrorCode PetscMemkindMalloc(size_t size, const char *func, const > char *file, void **result) > > > > { > > > > struct memkind *kind; > > > > int err; > > > > > > if (*result == NULL) { > > > > kind = MEMKIND_DEFAULT; > > > > } > > > > else { > > > > kind = (struct memkind *)(*result); > > > > } > > > > > > err = memkind_posix_memalign(kind, result, 16, size); > > > > return PosixErrToPetscErr(err); > > > > } > > > > > > and ifree will look something like: > > > > PetcErrorCode PetscMemkindFree(void *ptr, int a, const char *func, const > char *file) > > > > { > > > > memkind_free(0, ptr); > > > > return 0; > > > > } > > > > > > This gives us (1) a method of passing the kind of memory without > modifying the petsc allocation routine calling sequence, and (2) support a > fall back code path legacy applications which will not set the pointer to > NULL. Or am I missing something? > > > > Thoughts? I'd like to hash out something soon and start writing some > code. > > > > --Richard > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener