In addition to what Howard said, there's actually two other metrics that are used, as well. Each BTL exports a priority and an exclusivity value.
IIRC (it's been a while since I've looked at this code), the PML gathers up all BTL modules that claim that they can communicate between a pair of peers, and uses all the modules that share the same, highest exclusivity rating. TCP has a lower exclusivity rating than OS-bypass BTLs (such as usnic and openib). Hence, even if TCP is available between a pair of peers, OS-bypass transports will be preferred over TCP. On Dec 23, 2014, at 4:04 PM, Howard Pritchard <hpprit...@gmail.com> wrote: > HI Gary, > > The decision occurs within the MPI processes themselves (during the call to > MPI_Init) - so > after the orte daemons have started on the nodes. > > The BTL's report their "latency" and "bandwidth" - up the stack to the > PML/BML layer which > then decides based on these metrics which BTL to use to send/recv messages > between > any given pair of MPI ranks. > > Hope this helps, > > Howard > > > > > 2014-12-23 13:29 GMT-07:00 Gary Jackson <ga...@cs.umd.edu>: > > I'm not having any trouble getting it to start, and it's definitely using the > openib btl. I was just wondering how it decided whether the openib btl was > appropriate before going down the btl list to tcp when all mpirun gets is a > hostname and no other information about connectivity on the remote end. For > instance, is this determined before or after orted runs on the remote end? > > On 12/23/14, 2:18 PM, Howard Pritchard wrote: > Hello Gary, > > It depends on how the Open MPI was built, and on mca parameters passed > to the job either via settings in an mca params conf > file or the mpirun command line or env. variables. If you have mxm > (MLNX) or psm (qlogic/intel) installed on the system > where your open mpi was built, you may actually be using one of those > via the MTL path. > > Try > > mpirun -np 2 -H hosta,hostb -mca btl self,vader,openib ./your_favorite_test > > That should force open mpi to try using openib between the pair of > hosts. Note you don't need "vader" on the command line > if you are running only one mpi rank/node. > > Howard > > > > > 2014-12-23 11:48 GMT-07:00 Gary Jackson <ga...@cs.umd.edu > <mailto:ga...@cs.umd.edu>>: > > > How does OpenMPI decide whether to use the IB BTL between a given > pair of hosts, assuming there is an IB interface available? > > -- > Gary > _________________________________________________ > users mailing list > us...@open-mpi.org <mailto:us...@open-mpi.org> > Subscription: http://www.open-mpi.org/__mailman/listinfo.cgi/users > <http://www.open-mpi.org/mailman/listinfo.cgi/users> > Link to this post: > http://www.open-mpi.org/__community/lists/users/2014/12/__26063.php > <http://www.open-mpi.org/community/lists/users/2014/12/26063.php> > > > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/12/26064.php > > > > -- > Gary > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/12/26065.php > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/12/26066.php -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
