Hi George: In the current implementation, to send CUDA device memory, we move it through internal host buffers. In other words, we force the usage of the send protocols. If the host buffers are CUDA registered, then when we call cuMemcpy to move the data into those host buffers, the cuMemcpy performs better if the memory is registered, and cannot be paged out. This also may allow future optimizations, for example, asynchronous copies, that require the host memory we are copying into to be CUDA registered.
Rolf >-----Original Message----- >From: devel-boun...@open-mpi.org [mailto:devel-boun...@open-mpi.org] >On Behalf Of George Bosilca >Sent: Tuesday, August 02, 2011 10:50 AM >To: Open MPI Developers >Subject: Re: [OMPI devel] RFC: CUDA register sm and openib host memory > >Rolf, > >Can we have some more details on how this will improve performance of >sending GPU device memory? I fail to see how registering the backend shared >memory file with CUDA is supposed to do anything at all, as this memory is >internal to Open MPI and not supposed to be visible at any other level. > > Thanks, > george. > >On Jul 28, 2011, at 23:52 , Rolf vandeVaart wrote: > >> DETAILS: In order to improve performance of sending GPU device memory, >> we need to register the host memory with the CUDA framework. These >> changes allow that to happen. These changes are somewhat different >> from what I proposed a while ago and I think a lot cleaner. There is >> a new memory pool flag that indicates whether a piece of memory should >> be registered. This allows us to register the sm memory and the >> pre-posted openib memory. > >_______________________________________________ >devel mailing list >de...@open-mpi.org >http://www.open-mpi.org/mailman/listinfo.cgi/devel ----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -----------------------------------------------------------------------------------
