Re: RDMA read does not update local memory
Hello, I found out that some code does madvise(MADV_DONTNEED) on the allocated memory. I think that this unmaps the memory from the user space process. If the process touches the memory again, a COW mapping or even a new, zeroed out page is mapped in the process. Either way, the relation between the physical page, where the data from the RDMA read ends up and the virtual page is broken. I realize that calling madvise(MADV_DONTNEED) doesn’t have any meaning for RDMA-registered memory. I feel however that my observed behavior is a bug and one of the following should happen: . madvise() on pinned memory/I/O memory should fail, i.e. return EINVAL, EIO or ENOMEM (I didn’t test what happens when you call madvise(MADV_DONTNEED) on I/O mem.) . RDMA actions on madvise()’d memory should fail. Best regards, Hannes Weisbach-- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RDMA read does not update local memory
Am 30.01.2014 um 22:36 schrieb Anuj Kalia anujkaliai...@gmail.com: Hannes, Have you tried marking the memory that is being read as volatile“? Thanks for the suggestion, but I don’t think that this solves the problem, because when I do: memset(…); rdma_post_read(…); /* wait */ memcmp(…); The compiler may not optimize across rdma_post_read() (or rather ibv_post_send() and finally ioctl()), because these function may have side-effects (which they do). Anyway, I implemented a little helper memcmpv(volatile unsigned char * s1, volatile unsiged char * s2, size_t size) to compare the memory as volatile, but the memory still has the preset 0x55 instead of the contents of the remote memory. May that memset() be still in the cache, so that the memory accesses of the CPU hit the cache instead of main memory? I can’t imagine the cache isn’t invalidated on reception of remote data. Best regards, Hannes-- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RDMA read does not update local memory
Are you sure that the compiler knows that post_send() has side effects? On Fri, Jan 31, 2014 at 8:28 AM, Hannes Weisbach hannes.weisb...@tu-dresden.de wrote: Am 30.01.2014 um 22:36 schrieb Anuj Kalia anujkaliai...@gmail.com: Hannes, Have you tried marking the memory that is being read as volatile? Thanks for the suggestion, but I don't think that this solves the problem, because when I do: memset(...); rdma_post_read(...); /* wait */ memcmp(...); The compiler may not optimize across rdma_post_read() (or rather ibv_post_send() and finally ioctl()), because these function may have side-effects (which they do). Anyway, I implemented a little helper memcmpv(volatile unsigned char * s1, volatile unsiged char * s2, size_t size) to compare the memory as volatile, but the memory still has the preset 0x55 instead of the contents of the remote memory. May that memset() be still in the cache, so that the memory accesses of the CPU hit the cache instead of main memory? I can't imagine the cache isn't invalidated on reception of remote data. Best regards, Hannes -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RDMA read does not update local memory
Am 31.01.2014 um 15:26 schrieb Anuj Kalia anujkaliai...@gmail.com: Are you sure that the compiler knows that post_send() has side effects? No, thats why I tested with the mentioned memcmpv. I examined the emitted code with objdump and the compiler issued the mov instructions to read the bytes from memory - still didn’t work. I’ll leave the memcpyv in for the moment just to be sure. Best regards, Hannes-- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RDMA read does not update local memory
Hello all, I’m developing an (userspace) RDMA-application under Linux and came across a problem, I’m not able to solve. On machine A I had a chunk of memory registered with rdma_reg_read(), so others can read from that memory later. Machine B has memory which is also registered with rdma_reg_read(). Machine B tells machine A about that memory and the rkey. Machine A rdma_post_read()s then from that memory on machine B in it’s own memory. The problem now is, that - although no error is reported - the contents of machine A’s memory does not change. I memset() it to 0x55 before the rdma_post_read() for debugging purposes and instead of having the contents I expect (an ASCII string), the memory is still 0x55 after the work completion is signaled. Thus, I’ve allocated different chunks of memory using malloc() and posix_memalign() with different sizes and registered them with rdma_reg_read(), to see if that makes a difference. The rdma_post_read() works on all tested memory, except one chunk. I don’t know what is different about that memory, where my mistake might be or how to debug this. Please note, that I check all return values for errors (there are none) and also the work completion reports IBV_SUCCESS. I hope you can give me some hints on how to proceed. I apologize if my problem description was too brief; please ask for details if I left important information out. Best regards, Hannes Weisbach-- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RDMA read does not update local memory
Hannes, Have you tried marking the memory that is being read as volatile? --Anuj On Thu, Jan 30, 2014 at 1:45 PM, Hannes Weisbach hannes.weisb...@tu-dresden.de wrote: Hello all, I'm developing an (userspace) RDMA-application under Linux and came across a problem, I'm not able to solve. On machine A I had a chunk of memory registered with rdma_reg_read(), so others can read from that memory later. Machine B has memory which is also registered with rdma_reg_read(). Machine B tells machine A about that memory and the rkey. Machine A rdma_post_read()s then from that memory on machine B in it's own memory. The problem now is, that - although no error is reported - the contents of machine A's memory does not change. I memset() it to 0x55 before the rdma_post_read() for debugging purposes and instead of having the contents I expect (an ASCII string), the memory is still 0x55 after the work completion is signaled. Thus, I've allocated different chunks of memory using malloc() and posix_memalign() with different sizes and registered them with rdma_reg_read(), to see if that makes a difference. The rdma_post_read() works on all tested memory, except one chunk. I don't know what is different about that memory, where my mistake might be or how to debug this. Please note, that I check all return values for errors (there are none) and also the work completion reports IBV_SUCCESS. I hope you can give me some hints on how to proceed. I apologize if my problem description was too brief; please ask for details if I left important information out. Best regards, Hannes Weisbach-- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
