Brock, Is your post related to ib0/eoib0 being used at all, or being used with load balancing ?
let me clarify this : --mca btl ^openib disables the openib btl aka *native* infiniband. This does not disable ib0 and eoib0 that are handled by the tcp btl. As you already figured out, btl_tcp_if_include (or btl_tcp_if_exclude) can be used for that purpose. Cheers, Gilles Ralph Castain <rhc.open...@gmail.com> wrote: >OMPI discovers all active interfaces and automatically considers them >available for its use unless instructed otherwise via the params. I’d have to >look at the TCP BTL code to see the loadbalancing algo - I thought we didn’t >have that “on” by default across BTLs, but I don’t know if the TCP one >automatically uses all available Ethernet interfaces by default. Sounds like >it must. > > >> On Nov 7, 2014, at 11:53 AM, Brock Palen <bro...@umich.edu> wrote: >> >> I was doing a test on our IB based cluster, where I was diabling IB >> >> --mca btl ^openib --mca mtl ^mxm >> >> I was sending very large messages >1GB and I was surppised by the speed. >> >> I noticed then that of all our ethernet interfaces >> >> eth0 (1gig-e) >> ib0 (ip over ib, for lustre configuration at vendor request) >> eoib0 (ethernet over IB interface for IB -> Ethernet gateway for some >> extrnal storage support at >1Gig speed >> >> I saw all three were getting traffic. >> >> We use torque for our Resource Manager and use TM support, the hostnames >> given by torque match the eth0 interfaces. >> >> How does OMPI figure out that it can also talk over the others? How does it >> chose to load balance? >> >> BTW that is fine, but we will use if_exclude on one of the IB ones as ib0 >> and eoib0 are the same physical device and may screw with load balancing if >> anyone ver falls back to TCP. >> >> Brock Palen >> www.umich.edu/~brockp >> CAEN Advanced Computing >> XSEDE Campus Champion >> bro...@umich.edu >> (734)936-1985 >> >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2014/11/25709.php > >_______________________________________________ >users mailing list >us...@open-mpi.org >Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >Link to this post: >http://www.open-mpi.org/community/lists/users/2014/11/25710.php
