OFI uses libpsm2 underneath it when omnipath detected Sent from my iPhone
> On Mar 11, 2019, at 9:06 AM, Gilles Gouaillardet > <gilles.gouaillar...@gmail.com> wrote: > > Michael, > > You can > > mpirun --mca pml_base_verbose 10 --mca btl_base_verbose 10 --mca > mtl_base_verbose 10 ... > > It might show that pml/cm and mtl/psm2 are used. In that case, then yes, the > OmniPath library is used even for intra node communications. If this library > is optimized for intra node, then it will internally uses shared memory > instead of the NIC. > > > You can force > > mpirun --mca pml ob1 ... > > > And btl/vader (shared memory) will be used for intra node communications ... > unless MPI tasks are from different jobs (read MPI_Comm_spawn()) > > Cheers, > > Gilles > > Michael Di Domenico <mdidomeni...@gmail.com> wrote: >> i have a user that's claiming when two ranks on the same node want to >> talk with each other, they're using the NIC to talk rather then just >> talking directly. >> >> i've never had to test such a scenario. is there a way for me to >> prove one way or another whether two ranks are talking through say the >> kernel (or however it actually works) or using the nic? >> >> i didn't set any flags when i compiled openmpi to change this. >> >> i'm running ompi 3.1, pmix 2.2.1, and slurm 18.05 running atop omnipath >> _______________________________________________ >> users mailing list >> users@lists.open-mpi.org >> https://lists.open-mpi.org/mailman/listinfo/users > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
