Good day All,
My hypothesis is that with a SmartNIC offloading the CPU, some benefits of
Infiniband can also be achieved with Ethernet and I am looking for
information regarding fully supporting GPUDirect on the NIC's side.
I was able to DMA between a SmartNIC and a V100 GPU through PCIe. However,
to make this useful and more general it should work transparently with
things like MPI (and NCCL). Most resources I've found explains CUDA-Aware
MPI from a user's point of view, but I couldn't as of yet find information
about what need to be implemented on the NIC's side.
I've seen that there are MCA BTL parameters to set tcp, sm, self, openib
etc. I believe some development needs to be done in order to enable MPI to
make use of the SmartNIC, perhaps adding another BTL option? AFAIU, the
data that needs to be sent (and destination rank?), should be copied to the
TX Queue of the NIC. The NIC can then encap the raw data with relevant
headers and forward over a network without any CPU involvement.
Can anyone please point me to documentation, code, or give advice on how to
approach the integration between MPI and NIC?
Regards
Jaco
--
*Jaco Joubert*
*Software Engineer*
*Netronome* | 1st Floor, Southdowns Ridge Office Park, Cnr John Vorster &
Nellmapius Street, Irene, Centurion 0157, South Africa
Phone: +27 (012) 665-4427 <(012)%665-4427> | Skype: jaco.joubert12 |
www.netronome.com
_______________________________________________
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel
