This is a pre-RFC of some changes I am hoping to bring into the trunk. (I call this a pre-RFC as I have no timeout and I am not done with the code yet.)
With some prior commits, I have added the ability to send GPU buffers directly. This support consists of forcing the use of only the send/receive protocol when a GPU buffer is detected and using special memcpy functions to move data into and out of these buffers. My next set of changes intends to improve the performance within a node when sending GPU buffers. Essentially, we can do a RDMA remote get operations between two GPU cards. This code adds support for that. On the sending side, we use a very simple mpool to get a memory handle. On the receiveing side, we use a mpool to register and cache the memory handles. It is patterned after the rdma mpool. I am interested in anyone's reaction to this code. Most of the changes are contained in within code that is only used for CUDA, but there are several changes that happen within the sm BTL and the ob1 PML. What was kind of strange is that with these changes, we have the sm BTL that supports RDMA operations for GPU buffers, but not for host buffers, and that made for interesting code. The current changes can be viewed at: https://bitbucket.org/rolfv/ompi-trunk-cuda-rdma-1 [rvandevaart@rvandevaart-dt ompi-trunk-interp-rdma]$ svn status M opal/datatype/opal_convertor.h M ompi/mca/btl/sm/btl_sm_frag.c M ompi/mca/btl/sm/btl_sm.c M ompi/mca/btl/sm/btl_sm_component.c M ompi/mca/btl/sm/btl_sm_frag.h M ompi/mca/btl/sm/btl_sm.h M ompi/mca/btl/sm/Makefile.am M ompi/mca/btl/btl.h A ompi/mca/mpool/cuda A ompi/mca/mpool/cuda/mpool_cuda_component.c A ompi/mca/mpool/cuda/mpool_cuda_module.c A ompi/mca/mpool/cuda/mpool_cuda.h A ompi/mca/mpool/cuda/Makefile.am A ompi/mca/mpool/rcuda A ompi/mca/mpool/rcuda/mpool_rcuda_component.c A ompi/mca/mpool/rcuda/Makefile.am A ompi/mca/mpool/rcuda/mpool_rcuda_module.c A ompi/mca/mpool/rcuda/mpool_rcuda.h M ompi/mca/pml/ob1/pml_ob1_rdma.h M ompi/mca/pml/ob1/pml_ob1_sendreq.c M ompi/mca/pml/ob1/pml_ob1_sendreq.h M ompi/mca/pml/ob1/pml_ob1_recvreq.c M ompi/mca/pml/ob1/pml_ob1_rdma.c M ompi/mca/common/cuda/common_cuda.c M ompi/mca/common/cuda/common_cuda.h ----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -----------------------------------------------------------------------------------
