Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

Since these ports are running in actual Ethernet mode (as opposed to IPoIB), I do not think the interface names will be of the ibN (ib0, ib1, etc) format. It is more likely that the interface names will be of the form ethN or enPApBsCfD. It would be best to check with your system administrator,

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

Thanks Gus. I'll try and post results. I am newbie in this and appreciate any advice very much. Cheers --Boris On Mon, Jul 17, 2017 at 10:09 AM, Gus Correa wrote: > On 07/17/2017 01:06 PM, Gus Correa wrote: > >> Hi Boris >> >> The nodes may have standard Gigabit

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

On 07/17/2017 01:06 PM, Gus Correa wrote: Hi Boris The nodes may have standard Gigabit Ethernet interfaces, besides the Infiniband (RoCE). You may want to direct OpenMPI to use the Infiniband interfaces, not Gigabit Ethernet, by adding something like this to "--mca btl self,vader,self": Oops!

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

Hi Boris The nodes may have standard Gigabit Ethernet interfaces, besides the Infiniband (RoCE). You may want to direct OpenMPI to use the Infiniband interfaces, not Gigabit Ethernet, by adding something like this to "--mca btl self,vader,self": "--mca btl_tcp_if_include ib0,ib1" (Where the

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

Gus, Gilles, Russell, John: Thanks very much for the replies and the help. I got confirmation from the "root" that it is indeed RoCE with 100G. I'll go over the info in the link Russell provided, but have a quick question: if I run the "*mpiexec*" with "*-mca btl tcp,self*" do I get the benefit

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

It looks like you have two dual-port Mellanox VPI cards in this machine. These cards can be set to run InfiniBand or Ethernet on a port-by-port basis, and all four of your ports are set to Ethernet mode. Two of your ports have active 100 gigabit Ethernet links, and the other two have no link up at

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

Boris, these logs seem a bit odd to me. as far as i remember, the state is POLLING when there is no subnet manager. and when there is one, the state is ACTIVE *but* both Base and SM lid are non zero btw, is IPoIB configured ? if yes, then can your hosts ping each other with this interface. i

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

Boris, do you have a Subnet Manager running on your fabric? I am sorry if there have bene other replies ot this over the weekend. On 14 July 2017 at 18:34, Boris M. Vulovic wrote: > Gus, Gilles and John, > > Thanks for the help. Let me first post (below) the output

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

Gus, Gilles and John, Thanks for the help. Let me first post (below) the output from checkouts of the IB network: ibdiagnet ibhosts ibstat (for login node, for now) What do you think? Thanks --Boris

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

ABoris, as Gilles says - first do som elower level checkouts of your Infiniband network. I suggest running: ibdiagnet ibhosts and then as Gilles says 'ibstat' on each node On 14 July 2017 at 03:58, Gilles Gouaillardet wrote: > Boris, > > > Open MPI should automatically

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

Boris, Open MPI should automatically detect the infiniband hardware, and use openib (and *not* tcp) for inter node communications and a shared memory optimized btl (e.g. sm or vader) for intra node communications. note if you "-mca btl openib,self", you tell Open MPI to use the openib

Re: [OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

Have you tried: -mca btl vader,openib,self or -mca btl sm,openib,self by chance? That adds a btl for intra-node communication (vader or sm). On 07/13/2017 05:43 PM, Boris M. Vulovic wrote: I would like to know how to invoke InfiniBand hardware on CentOS 6x cluster with OpenMPI (static

[OMPI users] Q: Basic invoking of InfiniBand with OpenMPI

I would like to know how to invoke InfiniBand hardware on CentOS 6x cluster with OpenMPI (static libs.) for running my C++ code. This is how I compile and run: /usr/local/open-mpi/1.10.7/bin/mpic++ -L/usr/local/open-mpi/1.10.7/lib -Bstatic main.cpp -o DoWork usr/local/open-mpi/1.10.7/bin/mpiexec