Hi OMPI_Users && OMPI_Developers, Is there an equivalent to the MCA parameter btl_openib_include_if when using MXM over Infiniband (e.g. either (pml=cm mtl=mxm) or (pml=yalla)) ?
I ask this question because I'm working on a cluster where LXC containers are used on compute nodes (with SR-IOV I think) and multiple mlx4 interfaces are reported by lstopo (e.g. mlx4_0, mlx4_1, ..., mlx4_16) even if a single physical Mellanox Connect-X3 HCA is present per node. I found that when I use the plain openib btl (e.g. (pml=ob1 btl=openib)), it is much faster if I specify the MCA parameter btl_openib_include_if=mlx4_0 to force Open MPI to use a single interface. By doing that the latency is lower while the bandwidth higher. I guess it is because otherwise Open MPI mess by trying to use all "virtual" interfaces at once. However we all know that MXM is better than plain openib since it allows the HCAs to perform message matching, transfer message in the background and provide communication progress. So in this case is there a way to use only mlx4_0 ? I mean when using mxm mtl (pml=cm mtl=mxm) or preferably using it more directly by yalla pml (pml=yalla). Note I'm using Open MPI 1.10.3 I compiled myself for now but I could use instead Open MPI 2.0 if necessary . Thanks, Martin Audet
_______________________________________________ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
