Rolf, I'm still experimenting with cuda-rdma-2 on CUDA 4.1 ...
I'll build up cuda-rdma-3 and see what performance changes result. Ken Lloyd On Fri, 2011-12-09 at 11:45 -0800, Rolf vandeVaart wrote: > WHAT: Add new sm BTL, and supporting mpools, that can also support > CUDA RDMA. > > > > WHY: With CUDA 4.1, there is some GPU IPC support available that we > can take advantage of to move data efficiently between GPUs within a > node. > > > > WHERE: new--> ompi/mca/btl/smcuda, ompi/mca/mpool/cuda, > ompi/mca/mpool/rcuda Along with a few minor changes in ob1. These new > components are only built if explicitly asked for by configure. > Otherwise, new components are not built, and there are no changes > within normal code paths. > > (Jeff's rule: Do no harm) > > > > WHEN: Two weeks from now, December 23, 2011 > > > > DETAILS: There is the ability to improve that transfer of GPU memory > between GPUs within a node by making use of some IPC support that is > soon to be available with CUDA 4.1. These changes take advantage of > that to implement a RDMA GET protocol for GPU memory. > > > > To prevent any pollution within existing sm BTL, a new one has been > created that has the added RDMA GET support. In addition, two new > memory pools are needed as well which are being added. One of the > memory pools is very simple whereas the second one is patterned after > the rdma memory pool. > > > > Changes can be viewed at: > > https://bitbucket.org/rolfv/ompi-trunk-cuda-rdma-3/changeset/29f3255cd2b8 > > > > M ompi/mca/btl/btl.h > > A ompi/mca/btl/smcuda > > A ompi/mca/btl/smcuda/btl_smcuda_component.c > > A ompi/mca/btl/smcuda/configure.m4 > > A ompi/mca/btl/smcuda/btl_smcuda_frag.h > > A ompi/mca/btl/smcuda/help-mpi-btl-smcuda.txt > > A ompi/mca/btl/smcuda/btl_smcuda_endpoint.h > > A ompi/mca/btl/smcuda/btl_smcuda.h > > A ompi/mca/btl/smcuda/btl_smcuda_fifo.h > > A ompi/mca/btl/smcuda/Makefile.am > > A ompi/mca/btl/smcuda/btl_smcuda_frag.c > > A ompi/mca/btl/smcuda/btl_smcuda.c > > A ompi/mca/mpool/cuda > > A ompi/mca/mpool/cuda/configure.m4 > > A ompi/mca/mpool/cuda/mpool_cuda_component.c > > A ompi/mca/mpool/cuda/mpool_cuda_module.c > > A ompi/mca/mpool/cuda/mpool_cuda.h > > A ompi/mca/mpool/cuda/Makefile.am > > A ompi/mca/mpool/rcuda > > A ompi/mca/mpool/rcuda/configure.m4 > > A ompi/mca/mpool/rcuda/mpool_rcuda_component.c > > A ompi/mca/mpool/rcuda/Makefile.am > > A ompi/mca/mpool/rcuda/mpool_rcuda_module.c > > A ompi/mca/mpool/rcuda/mpool_rcuda.h > > M ompi/mca/common/cuda/configure.m4 > > M ompi/mca/common/cuda/common_cuda.c > > M ompi/mca/common/cuda/help-mpi-common-cuda.txt > > M ompi/mca/common/cuda/common_cuda.h > > M ompi/mca/pml/ob1/pml_ob1_sendreq.c > > M ompi/mca/pml/ob1/pml_ob1_sendreq.h > > M ompi/mca/pml/ob1/pml_ob1_recvreq.c > > A ompi/mca/pml/ob1/pml_ob1_cuda.c > > M ompi/mca/pml/ob1/Makefile.am > > > > Rolf > > > > rvandeva...@nvidia.com > > 781-275-5358 > > > > > > ______________________________________________________________________ > This email message is for the sole use of the intended recipient(s) > and may contain confidential information. Any unauthorized review, > use, disclosure or distribution is prohibited. If you are not the > intended recipient, please contact the sender by reply email and > destroy all copies of the original message. > > ______________________________________________________________________ > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel ============== Kenneth A. Lloyd, Jr. CEO - Director of Systems Science Watt Systems Technologies Inc. Albuquerque, NM US This e-mail is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521 and is intended only for the addressee named above. It may contain privileged or confidential information. If you are not the addressee you must not copy, distribute, disclose or use any of the information in it. If you have received it in error please delete it and immediately notify the sender.
