On Jun 5, 2013, at 12:05 PM, Jason Gunthorpe <jguntho...@obsidianresearch.com> wrote:
>> It does seem quite odd, abstractly speaking, that a registration >> would survive a free/re-malloc (which is arguably a "different" >> buffer). > > Not at all: the purpose of the registration is to allow access via > RDMA to a portion of the process's address space. The address space > doesn't change, but what it is mapped to can vary. I still think it's really weird. When I do this: a = malloc(N); ibv_reg_mr(..., a, N, ...); free(a); b = malloc(M); If b just happens to be partially or wholly registered by some quirk of the malloc() system (i.e., some/all of the virtual address space in b happens to have been covered by a prior malloc/ibv_reg_mr)... that's just weird. >> If ibv_reg_mr(..., >> 0, 2^64, ...) was supported, that would obviate the entire need for >> registration caches. That would be wonderful. > > Yes, except that this shifts around where the registration overhead > ends up. Basically the HCA driver now has the registration cache you > had in MPI, and all the same overheads still exist. There's fewer verbs drivers than applications, right? > Haggai: A verb to resize a registration would probably be a helpful > step. MPI could maintain one registration that covers the sbrk > region and one registration that covers the heap, much easier than > searching tables and things. If we still have to register buffers piecemeal, a non-blocking registration verb would be quite helpful. > Also bear in mind that all RDMA access protections will be disabled if > you register the entire process VM, the remote(s) can scribble/read > everything.. No problem for MPI/HPC... :-) -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
