On Apr 17, 2016, at 8:43 PM, dpchoudh . <dpcho...@gmail.com
<mailto:dpcho...@gmail.com>> wrote:
Hello Gilles and all
I am sorry to be bugging the developers, but this issue seems to
be nagging me, and I am surprised it does not seem to affect
anybody else. But then again, I am using the master branch, and
most users are probably using a released version.
This time I am using a totally different cluster. This has NO
verbs capable interface; just 2 Ethernet (1 of which has no IP
address and hence is unusable) plus 1 proprietary interface that
currently supports only IP traffic. The two IP interfaces
(Ethernet and proprietary) are on different IP subnets.
My test program is as follows:
#include <stdio.h>
#include <string.h>
#include "mpi.h"
int main(int argc, char *argv[])
{
char host[128];
int n;
MPI_Init(&argc, &argv);
MPI_Get_processor_name(host, &n);
printf("Hello from %s\n", host);
MPI_Comm_size(MPI_COMM_WORLD, &n);
printf("The world has %d nodes\n", n);
MPI_Comm_rank(MPI_COMM_WORLD, &n);
printf("My rank is %d\n",n);
//#if 0
if (n == 0)
{
strcpy(host, "ha!");
MPI_Send(host, strlen(host) + 1, MPI_CHAR, 1, 1, MPI_COMM_WORLD);
printf("sent %s\n", host);
}
else
{
//int len = strlen(host) + 1;
bzero(host, 128);
MPI_Recv(host, 4, MPI_CHAR, 0, 1, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);
printf("Received %s from rank 0\n", host);
}
//#endif
MPI_Finalize();
return 0;
}
This program, when run between two nodes, hangs. The command was:
[durga@b-1 ~]$ mpirun -np 2 -hostfile ~/hostfile -mca btl
self,tcp -mca pml ob1 -mca btl_tcp_if_include eno1 ./mpitest
And the hang is with the following output: (eno1 is one of the
gigEth interfaces, that takes OOB traffic as well)
Hello from b-1
The world has 2 nodes
My rank is 0
Hello from b-2
The world has 2 nodes
My rank is 1
Note that if I uncomment the #if 0 - #endif (i.e. comment out the
MPI_Send()/MPI_Recv() part, the program runs to completion. Also
note that the printfs following MPI_Send()/MPI_Recv() do not show
up on console.
Upon attaching gdb, the stack trace from the master node is as
follows:
Missing separate debuginfos, use: debuginfo-install
glibc-2.17-78.el7.x86_64 libpciaccess-0.13.4-2.el7.x86_64
(gdb) bt
#0 0x00007f72a533eb7d in poll () from /lib64/libc.so.6
#1 0x00007f72a4cb7146 in poll_dispatch (base=0xee33d0,
tv=0x7fff81057b70)
at poll.c:165
#2 0x00007f72a4caede0 in opal_libevent2022_event_base_loop
(base=0xee33d0,
flags=2) at event.c:1630
#3 0x00007f72a4c4e692 in opal_progress () at
runtime/opal_progress.c:171
#4 0x00007f72a0d07ac1 in opal_condition_wait (
c=0x7f72a5bb1e00 <ompi_request_cond>, m=0x7f72a5bb1d80
<ompi_request_lock>)
at ../../../../opal/threads/condition.h:76
#5 0x00007f72a0d07ca2 in ompi_request_wait_completion
(req=0x113eb80)
at ../../../../ompi/request/request.h:383
#6 0x00007f72a0d09cd5 in mca_pml_ob1_send (buf=0x7fff81057db0,
count=4,
datatype=0x601080 <ompi_mpi_char>, dst=1, tag=1,
sendmode=MCA_PML_BASE_SEND_STANDARD, comm=0x601280
<ompi_mpi_comm_world>)
at pml_ob1_isend.c:251
#7 0x00007f72a58d6be3 in PMPI_Send (buf=0x7fff81057db0, count=4,
type=0x601080 <ompi_mpi_char>, dest=1, tag=1,
comm=0x601280 <ompi_mpi_comm_world>) at psend.c:78
#8 0x0000000000400afa in main (argc=1, argv=0x7fff81057f18) at
mpitest.c:19
(gdb)
And the backtrace on the non-master node is:
(gdb) bt
#0 0x00007ff3b377e48d in nanosleep () from /lib64/libc.so.6
#1 0x00007ff3b37af014 in usleep () from /lib64/libc.so.6
#2 0x00007ff3b0c922de in OPAL_PMIX_PMIX120_PMIx_Fence
(procs=0x0, nprocs=0,
info=0x0, ninfo=0) at src/client/pmix_client_fence.c:100
#3 0x00007ff3b0c5f1a6 in pmix120_fence (procs=0x0, collect_data=0)
at pmix120_client.c:258
#4 0x00007ff3b3cf8f4b in ompi_mpi_finalize ()
at runtime/ompi_mpi_finalize.c:242
#5 0x00007ff3b3d23295 in PMPI_Finalize () at pfinalize.c:47
#6 0x0000000000400958 in main (argc=1, argv=0x7fff785e8788) at
mpitest.c:30
(gdb)
The hostfile is as follows:
[durga@b-1 ~]$ cat hostfile
10.4.70.10 slots=1
10.4.70.11 slots=1
#10.4.70.12 slots=1
And the ifconfig output from the master node is as follows (the
other node is similar; all the IP interfaces are in their
respective subnets) :
[durga@b-1 ~]$ ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.4.70.10 netmask 255.255.255.0 broadcast 10.4.70.255
inet6 fe80::21e:c9ff:fefe:13df prefixlen 64 scopeid
0x20<link>
ether 00:1e:c9:fe:13:df txqueuelen 1000 (Ethernet)
RX packets 48215 bytes 27842846 (26.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 52746 bytes 7817568 (7.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16
eno2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 00:1e:c9:fe:13:e0 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 17
lf0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2016
inet 192.168.1.2 netmask 255.255.255.0 broadcast
192.168.1.255
inet6 fe80::3002:ff:fe33:3333 prefixlen 64 scopeid
0x20<link>
ether 32:02:00:33:33:33 txqueuelen 1000 (Ethernet)
RX packets 10 bytes 512 (512.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 22 bytes 1536 (1.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 26 bytes 1378 (1.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 26 bytes 1378 (1.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Please help me with this. I am stuck with the TCP transport,
which is the most basic of all transports.
Thanks in advance
Durga
1% of the executables have 99% of CPU privilege!
Userspace code! Unite!! Occupy the kernel!!!
On Tue, Apr 12, 2016 at 9:32 PM, Gilles Gouaillardet
<gil...@rist.or.jp <mailto:gil...@rist.or.jp>> wrote:
This is quite unlikely, and fwiw, your test program works for me.
i suggest you check your 3 TCP networks are usable, for example
$ mpirun -np 2 -hostfile ~/hostfile -mca btl self,tcp -mca
pml ob1 --mca btl_tcp_if_include xxx ./mpitest
in which xxx is a [list of] interface name :
eth0
eth1
ib0
eth0,eth1
eth0,ib0
...
eth0,eth1,ib0
and see where problem start occuring.
btw, are your 3 interfaces in 3 different subnet ? is routing
required between two interfaces of the same type ?
Cheers,
Gilles
On 4/13/2016 7:15 AM, dpchoudh . wrote:
Hi all
I have reported this issue before, but then had brushed it
off as something that was caused by my modifications to the
source tree. It looks like that is not the case.
Just now, I did the following:
1. Cloned a fresh copy from master.
2. Configured with the following flags, built and installed
it in my two-node "cluster".
--enable-debug --enable-debug-symbols --disable-dlopen
3. Compiled the following program, mpitest.c with these
flags: -g3 -Wall -Wextra
4. Ran it like this:
[durga@smallMPI ~]$ mpirun -np 2 -hostfile ~/hostfile -mca
btl self,tcp -mca pml ob1 ./mpitest
With this, the code hangs at MPI_Barrier() on both nodes,
after generating the following output:
Hello world from processor smallMPI, rank 0 out of 2 processors
Hello world from processor bigMPI, rank 1 out of 2 processors
smallMPI sent haha!
bigMPI received haha!
<Hangs until killed by ^C>
Attaching to the hung process at one node gives the
following backtrace:
(gdb) bt
#0 0x00007f55b0f41c3d in poll () from /lib64/libc.so.6
#1 0x00007f55b03ccde6 in poll_dispatch (base=0x70e7b0,
tv=0x7ffd1bb551c0) at poll.c:165
#2 0x00007f55b03c4a90 in opal_libevent2022_event_base_loop
(base=0x70e7b0, flags=2) at event.c:1630
#3 0x00007f55b02f0144 in opal_progress () at
runtime/opal_progress.c:171
#4 0x00007f55b14b4d8b in opal_condition_wait
(c=0x7f55b19fec40 <ompi_request_cond>, m=0x7f55b19febc0
<ompi_request_lock>) at ../opal/threads/condition.h:76
#5 0x00007f55b14b531b in ompi_request_default_wait_all
(count=2, requests=0x7ffd1bb55370, statuses=0x7ffd1bb55340)
at request/req_wait.c:287
#6 0x00007f55b157a225 in ompi_coll_base_sendrecv_zero
(dest=1, stag=-16, source=1, rtag=-16, comm=0x601280
<ompi_mpi_comm_world>)
at base/coll_base_barrier.c:63
#7 0x00007f55b157a92a in
ompi_coll_base_barrier_intra_two_procs (comm=0x601280
<ompi_mpi_comm_world>, module=0x7c2630) at
base/coll_base_barrier.c:308
#8 0x00007f55b15aafec in
ompi_coll_tuned_barrier_intra_dec_fixed (comm=0x601280
<ompi_mpi_comm_world>, module=0x7c2630) at
coll_tuned_decision_fixed.c:196
#9 0x00007f55b14d36fd in PMPI_Barrier (comm=0x601280
<ompi_mpi_comm_world>) at pbarrier.c:63
#10 0x0000000000400b0b in main (argc=1, argv=0x7ffd1bb55658)
at mpitest.c:26
(gdb)
Thinking that this might be a bug in tuned collectives,
since that is what the stack shows, I ran the program like
this (basically adding the ^tuned part)
[durga@smallMPI ~]$ mpirun -np 2 -hostfile ~/hostfile -mca
btl self,tcp -mca pml ob1 -mca coll ^tuned ./mpitest
It still hangs, but now with a different stack trace:
(gdb) bt
#0 0x00007f910d38ac3d in poll () from /lib64/libc.so.6
#1 0x00007f910c815de6 in poll_dispatch (base=0x1a317b0,
tv=0x7fff43ee3610) at poll.c:165
#2 0x00007f910c80da90 in opal_libevent2022_event_base_loop
(base=0x1a317b0, flags=2) at event.c:1630
#3 0x00007f910c739144 in opal_progress () at
runtime/opal_progress.c:171
#4 0x00007f910db130f7 in opal_condition_wait
(c=0x7f910de47c40 <ompi_request_cond>, m=0x7f910de47bc0
<ompi_request_lock>)
at ../../../../opal/threads/condition.h:76
#5 0x00007f910db132d8 in ompi_request_wait_completion
(req=0x1b07680) at ../../../../ompi/request/request.h:383
#6 0x00007f910db1533b in mca_pml_ob1_send (buf=0x0,
count=0, datatype=0x7f910de1e340 <ompi_mpi_byte>, dst=1,
tag=-16, sendmode=MCA_PML_BASE_SEND_STANDARD,
comm=0x601280 <ompi_mpi_comm_world>) at pml_ob1_isend.c:259
#7 0x00007f910d9c3b38 in
ompi_coll_base_barrier_intra_basic_linear (comm=0x601280
<ompi_mpi_comm_world>, module=0x1b092c0) at
base/coll_base_barrier.c:368
#8 0x00007f910d91c6fd in PMPI_Barrier (comm=0x601280
<ompi_mpi_comm_world>) at pbarrier.c:63
#9 0x0000000000400b0b in main (argc=1, argv=0x7fff43ee3a58)
at mpitest.c:26
(gdb)
The mpitest.c program is as follows:
#include <mpi.h>
#include <stdio.h>
#include <string.h>
int main(int argc, char** argv)
{
int world_size, world_rank, name_len;
char hostname[MPI_MAX_PROCESSOR_NAME], buf[8];
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
MPI_Get_processor_name(hostname, &name_len);
printf("Hello world from processor %s, rank %d out of %d
processors\n", hostname, world_rank, world_size);
if (world_rank == 1)
{
MPI_Recv(buf, 6, MPI_CHAR, 0, 99, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);
printf("%s received %s\n", hostname, buf);
}
else
{
strcpy(buf, "haha!");
MPI_Send(buf, 6, MPI_CHAR, 1, 99, MPI_COMM_WORLD);
printf("%s sent %s\n", hostname, buf);
}
MPI_Barrier(MPI_COMM_WORLD);
MPI_Finalize();
return 0;
}
The hostfile is as follows:
10.10.10.10 slots=1
10.10.10.11 slots=1
The two nodes are connected by three physical and 3 logical
networks:
Physical: Gigabit Ethernet, 10G iWARP, 20G Infiniband
Logical: IP (all 3), PSM (Qlogic Infiniband), Verbs (iWARP
and Infiniband)
Please note again that this is a fresh, brand new clone.
Is this a bug (perhaps a side effect of --disable-dlopen) or
something I am doing wrong?
Thanks
Durga
We learn from history that we never learn from history.
_______________________________________________
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription:http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this
post:http://www.open-mpi.org/community/lists/users/2016/04/28930.php
_______________________________________________
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2016/04/28932.php
_______________________________________________
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2016/04/28942.php
