Hi, I hope this is the right forum for my questions. I am running into
a problem when scaling >512 cores on a infiniband cluster which has
14,336 cores. I am new to openmpi and trying to figure out the right
-mca options to pass to avoid the "mca_oob_tcp_peer_complete_connect:
connection failed:" on a cluster which has infiniband HCAs and OFED
v1.3GA release. Other MPI implementation like Intel MPI and mvapich
work fine using uDAPL or VERBs IB layers for MPI communications.
I find it difficult to understand which network interface or IB layer
being used. When I explicitly state not to use eth0,lo,ib1, or ib1:0
interfaces with the cmdline option "-mca oob_tcp_exclude" openmpi will
continue to probe these interfaces. For all MPI traffic openmpi should
use IB0 which is the 10.148 network. But with debugging enabled I see
references trying the 10.149 network which is IB1. Below is the
ifconfig network device output for a compute node.
Questions:
1. Is there away to determine which network device is being used and not
have openmpi fallback to another device? With Intel MPI or HP MPI you
can state not to use a fallback device. I thought "-mca
oob_tcp_exclude" would be the correct option to pass but I maybe wrong.
2. How can I determine infiniband openib device is actually being used?
When running a MPI app I continue to see counters for in/out packets at
a tcp level increasing when it should be using the IB RDMA device for
all MPI comms over the IB0 or mtcha0 device? OpenMPI was bundled with
OFED v1.3 so I am assuming the openib interface should work. Running
ompi_info shows btl_open_* references.
/usr/mpi/openmpi-1.2-2/intel/bin/mpiexec -mca
btl_openib_warn_default_gid_prefix 0 -mca oob_tcp_exclude
eth0,lo,ib1,ib1:0 -mca btl openib,sm,self -machinefile mpd.hosts.$$ -np
1024 ~/bin/test_ompi < input1
3. When trying to avoid the "mca_oob_tcp_peer_complete_connect:
connection failed:" message I tried using "-mca btl openib,sm,self" and
"-mca btl ^tcp" but I still get these error messages. In cases with
using the "-mca btl openib,sm,self" openmpi will retry to use the IB1
(10.149 net) fabric to establish a connection with a node. What are my
options to avoid these connection failed messages? I suspect openmpi is
overflowing the tcp buffer on the clients based on large core count of
this job since I see lots of tcp buffer errors based on netstat -s
output. I reviewed all of the online FAQs and I am not sure what options
to pass to get around this issue.
OBTW, I did check the
/usr/mpi/openmpi-1.2-2/intel/etc/openmpi-mca-params.conf file and no
defaults are being specified.
----
Ompi_info:
Open MPI: 1.2.2
Open MPI SVN revision: r14613
Open RTE: 1.2.2
Open RTE SVN revision: r14613
OPAL: 1.2.2
OPAL SVN revision: r14613
Prefix: /usr/mpi/openmpi-1.2-2/intel
Configured architecture: x86_64-suse-linux-gnu
------
Following is the cluster configuration:
1792 nodes with 8 cores per node = 14336 cores
Ofed Rel: OFED-1.3-rc1
IB Device(s): mthca0 FW=1.2.0 Rate=20 Gb/sec (4X DDR) mthca1 FW=1.2.0
Rate=20 Gb/sec (4X DDR)
Processors: 2 x 4 Cores Intel(R) Xeon(R) CPU X5365 @ 3.00GHz 8192KB
Cache FSB:1333MHz
Total Mem: 16342776 KB
OS Release: SUSE Linux Enterprise Server 10 (x86_64) VERSION = 10 SP1
Kernel Ver: 2.6.16.54-0.2.5-smp
------
Ifconfig output:
eth0 Link encap:Ethernet HWaddr 00:30:48:7B:A7:AC
inet addr:192.168.159.41 Bcast:192.168.159.255
Mask:255.255.255.0
inet6 addr: fe80::230:48ff:fe7b:a7ac/64 Scope:Link
UP BROADCAST NOTRAILERS RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1215826 errors:0 dropped:0 overruns:0 frame:0
TX packets:1342035 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:787514337 (751.0 Mb) TX bytes:170968505 (163.0 Mb)
Base address:0x2000 Memory:dfa00000-dfa20000
ib0 Link encap:UNSPEC HWaddr
80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:10.148.3.73 Bcast:10.148.255.255 Mask:255.255.0.0
inet6 addr: fe80::230:487b:a7ac:1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:20823896 errors:0 dropped:0 overruns:0 frame:0
TX packets:19276836 errors:0 dropped:42 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:176581223103 (168400.9 Mb) TX bytes:182691213682
(174227.9 Mb)
ib1 Link encap:UNSPEC HWaddr
80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:10.149.195.73 Bcast:10.149.255.255
Mask:255.255.192.0
inet6 addr: fe80::230:487b:a7ad:1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:175609 errors:0 dropped:0 overruns:0 frame:0
TX packets:31175 errors:0 dropped:6 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:139196236 (132.7 Mb) TX bytes:4515680 (4.3 Mb)
ib1:0 Link encap:UNSPEC HWaddr
80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:10.149.3.73 Bcast:10.149.63.255 Mask:255.255.192.0
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:30554 errors:0 dropped:0 overruns:0 frame:0
TX packets:30554 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:54170543 (51.6 Mb) TX bytes:54170543 (51.6 Mb)
--------
Ibstatus output:
Infiniband device 'mthca0' port 1 status:
default gid: fe80:0000:0000:0000:0030:487c:04b4:0001
base lid: 0x4fb
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 20 Gb/sec (4X DDR)
Infiniband device 'mthca1' port 1 status:
default gid: fe80:0000:0000:0000:0030:487c:04b5:0001
base lid: 0x50c
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 20 Gb/sec (4X DDR)
--------
Thanks in advance,
Scott
