On Thu, Apr 30, 2009 at 09:52:32AM -0400, Jeff Squyres wrote: > I think Jason is the only one who is remaining at least somewhat on-topic > here.
Thanks, but I have no stake in this, it is just interesting :) After reading all the postings, I think my idea to fix the verbs API to not, essentially, corrupt an existing registration when the virtual address space changes is the best bet. This slightly changes the semantics of the verbs MR to refer to virtual address space within the process, not the underlying object(s) that happen to be mapped there when the registration is made. The data corruption problem would be immediately solved by doing this and no changes at all would be needed in any MPIs. Surely a good thing? MPIs can choose to continue to hook malloc/free/etc or not, it doesn't really matter from a correctness perspective. Minimizating the amount of VM that is registered is perhaps important, perhaps not, I suspect, depending on the job.. Certainly there are fairly fast ways to do periodic garbage collection on the registration cache (ie by inspecting proc/self/maps, or the glibc free block list, or whatever.). Notifiers are going to be very troublesome, every time any sort of synchronous to user space notifier has been proposed or implemented in the kernel it has been a disaster. I would not hold out much hope for this.. > While MPI is currently the biggest victim, this broken memory management > model is also an enormous roadblock for any other application or ULP to > write to verbs. I'm not sure this is true.. The purpose built verbs apps I've worked on don't have a problem like MPI, and managing the memory registration was not hard. My main complaint would be the lack of FMR and FRMR in userspace - mostly because of the performance of the registration functions.. Jason _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
