On Jun 7, 2013, at 4:57 PM, Jason Gunthorpe <jguntho...@obsidianresearch.com> wrote:
>> We talked about this at the MPI Forum this week; it doesn't seem >> like ODP fixes any MPI problems. > > ODP without 'register all address space' changes the nature of the > problem, and fixes only one problem. I agree that pushing all registration issues out of the application and (somewhere) into the verbs stack would be a nice solution. > You do need to cache registrations, and all the tuning parameters (how > much do I cache, how long do I hold it for, etc, etc) all still apply. > > What goes away (is fixed) is the need for intercepts and the need to > purge address space from the cache because the backing registration > has become non-coherent/invalid. Registrations are always > coherent/valid with ODP. > This cache, and the associated optimization problem, can never go > away. With a 'register all of memory' semantic the cache can move into > the kernel, but the performance implication and overheads are all > still present, just migrated. Good summary; and you corrected some of my mistakes -- thanks. That being said, everyone I've talked to about ODP finds it very, very strange that the kernel would keep memory registrations around for memory that is no longer part of a process. Not only does it lead to the "new memory is magically already registered" semantic that I find weird, it's just plain *odd* for the kernel to maintain state for something that doesn't exist any more. It feels dirty. Sidenote: I was just informed today that the current way MPI implementations implement registration cache coherence (glibc malloc hooks) has been deprecated and will be removed from glibc (http://sourceware.org/ml/libc-alpha/2011-05/msg00103.html). This really puts on the pressure to find a new / proper solution. >> What MPI wants is: >> >> 1. verbs for ummunotify-like functionality >> 2. non-blocking memory registration verbs; poll the cq to know when it has >> completed > > To me, ODP with an additional 'register all address space' semantic, plus > an asynchronous prefetch does both of these for you. > > 1. ummunotify functionality and caching is now in the kernel, under > ODP. RDMA access to an 'all of memory' registration always does the > right thing. "Register all address space" is the moral equivalent of not having userspace registration, so let's talk about it in those terms. Specifically, there's a subtle difference between: a) telling verbs to register (0...2^64) --> Which is weird because it tells verbs to register memory that isn't in my address space b) telling verbs that the app doesn't want to handle registration --> How that gets implemented is not important (from userspace's point of view) -- if the kernel chooses to implement that by registering non-existent memory, that's the kernel's problem I guess I'm arguing that registering non-existent memory is not the Right Thing. Regardless of what solution is devised for registered memory management (ummunotify, ODP, or something else), a non-blocking verb for registering memory would still be a Very Useful Thing. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
