On Thu, Apr 30, 2009 at 02:24:47PM -0400, Tom Talpey wrote: > At 06:11 PM 4/29/2009, Barrett, Brian W wrote: > >On 4/29/09 15:55 , "Jason Gunthorpe" <[email protected]> > >wrote: > > > >>> The problem is that MPI needs to be aware of the application doing > >>> the free() and unregister or flush its MR cache for that virtual > >>> address range. Of course it would be difficult for OpenMPI to have > >>> callbacks or hooks into every way memory could be allocated/freed > >>> that an application might use. > >> > >> There are only three calls that affect the way VM memory maps to > >> physical and thus would invalidate the mr cache: mmap, munmap and brk. > > > >There's also System V shared memory, which at least one scientific code out > >there uses. > > Don't forget fork, vfork, clone and exec, and also don't forget any > copy-on-write mappings that result. Oh, and those pesky stack pages.
verbs just plain can't inherit QP and MR across fork and all the related. There is no way to split a QP and a MR across to processes and have things still make sense. One process gets it, so there isn't really a problem. Stack pages are mostly fixed in VM address space, particularly if you give up MAP_GROWSDOWN which I think most threading libraries do these days? Anyway, it doesn't matter, if the kernel has a syscall to enable registering all VM in a process then the kernel is perfectly able to capture all the wierd cases and fix them up. The API to userspace is very simple and sane. > I think the point is that making any guarantees that memory remains fixed > and present will inevitably lead to nontransparent API requirements on the > applications. Been there, done that, got plenty of t-shirts. It's a hard road, > because APIs are forever. That's why we are here, the MPI spec says all memory in a process is valid to use with a network operation - that is utterly incompatible with verb's notion of memory registration. Jason _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
