On Jun 5, 2013, at 11:18 AM, Jason Gunthorpe <jguntho...@obsidianresearch.com> wrote:
>> Are you saying that the 2nd malloc will magically be registered >> (with the new physical address)? > > Yes, that is the whole point. Interesting. > ODP fundamentally fixes the *bug* where the HCA's view of process > memory can become inconsistent with the kernel's view. Hum. I was under the impression that with today's code (i.e., not ODP), if you a = malloc(N); ibv_reg_mr(..., a, N, ...); free(a); (assuming that the memory actually left the process at free) Then the relevant kernel verbs driver was notified, and would unregister that device. ...but I'm an MPI guy, not a kernel guy -- it seems like you're saying that my impression was wrong (which doesn't currently matter because we intercept free/sbrk and unregister such memory, anyway). > 'magically be registered' is the wrong way to think about it - the > registration of VA=0x100 is simply kept, and any change to the > underlying physical mapping of the VA is synchronized with the HCA. What happens if you: a = malloc(N * page_size); ibv_reg_mr(..., a, N * page_size, ...); free(a); // incoming RDMA arrives targeted at buffer a Or if you: a = malloc(N * page_size); ibv_reg_mr(..., a, N * page_size, ...); free(a); a = malloc(N / 2 * page_size); // incoming RDMA arrives targeted at buffer a that is of length (N*page_size) It does seem quite odd, abstractly speaking, that a registration would survive a free/re-malloc (which is arguably a "different" buffer). That being said, it still seems like MPI needs a registration cache. It is several good steps forward if we don't need to intercept free/sbrk/whatever, but when MPI_Send(buf, ...) is invoked, we still have to check that the entire buf is registered. If ibv_reg_mr(..., 0, 2^64, ...) was supported, that would obviate the entire need for registration caches. That would be wonderful. > Right, this was discussed at the Enterprise Summit a few weeks > ago. I'm sure Roland would welcome patches... That's why I asked at the beginning of this thread. He didn't provide any details about what still needs to be done, though. :-) -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
