Re: [OMPI devel] Routed 'unity' broken on trunk

[Josh Hursey]

At the moment I only use unity with C/R. Mostly because I have not  
verified that the other components work properly under the C/R  
conditions. I can verify others, but that doesn't solve the problem  
with the unity component. :/

It is not critical that these jobs launch quickly, but that they  
launch correctly for the moment. When you say 'slow the launch' are  
you talking severely as in seconds/minutes for small nps? I guess a  
followup question is why did this component break in the first place?  
or worded differently, what changed in ORTE such that the unity  
component will suddenly deadlock when it didn't before?

Thanks for looking into this,
Josh

On Mar 31, 2008, at 11:10 AM, Ralph H Castain wrote:

I figured out the issue - there is a simple and a hard way to fix  
this. So
before I do, let me see what makes sense.

The simple solution involves updating the daemons with contact info  
for the
procs so that they can send their collected modex info to the rank=0  
proc.
This will measurably slow the launch when using unity.

The hard solution is to do a hybrid routed approach whereby the  
daemons
would route any daemon-to-proc communication while the procs  
continue to do
direct proc-to-proc messaging.

Is there some reason to be using the "unity" component? Do you care  
if jobs
using unity launch slower?

Thanks
Ralph



On 3/31/08 7:57 AM, "Josh Hursey" <jjhur...@open-mpi.org> wrote:

Ralph,

I've just noticed that it seems that the 'unity' routed component
seems to be broken when using more than one machine. I'm using Odin
and r18028 of the trunk, and have confirmed that this problem occurs
with SLURM and rsh. I think this break came in on Friday as that is
when some of my MTT tests started to hang and fail, but I cannot  
point
to a specific revision at this point. The backtraces (enclosed) of  
the
processes point to the grpcomm allgather routine.

The 'noop' program calls MPI_Init, sleeps, then calls MPI_Finalize.

RSH example from odin023 - so no SLURM variables:
These work:
 shell$ mpirun -np 2 -host odin023  noop -v 1
 shell$ mpirun -np 2 -host odin023,odin024  noop -v 1
 shell$ mpirun -np 2 -mca routed unity -host odin023  noop -v 1

This hangs:
 shell$ mpirun -np 2 -mca routed unity -host odin023,odin024  noop - 
v 1


If I attach to the 'noop' process on odin023 I get the following
backtrace:
------------------------------------------------
(gdb) bt
#0  0x0000002a96226b39 in syscall () from /lib64/tls/libc.so.6
#1  0x0000002a95a1e485 in epoll_wait (epfd=3, events=0x50b330,
maxevents=1023, timeout=1000) at epoll_sub.c:61
#2  0x0000002a95a1e7f7 in epoll_dispatch (base=0x506c30,  
arg=0x506910,
tv=0x7fbfffe840) at epoll.c:210
#3  0x0000002a95a1c057 in opal_event_base_loop (base=0x506c30,
flags=5) at event.c:779
#4  0x0000002a95a1be8f in opal_event_loop (flags=5) at event.c:702
#5  0x0000002a95a0bef8 in opal_progress () at runtime/ 
opal_progress.c:
169
#6  0x0000002a958b9e48 in orte_grpcomm_base_allgather
(sbuf=0x7fbfffeae0, rbuf=0x7fbfffea80) at base/
grpcomm_base_allgather.c:238
#7  0x0000002a958bd37c in orte_grpcomm_base_modex (procs=0x0) at  
base/
grpcomm_base_modex.c:413
#8  0x0000002a956b8416 in ompi_mpi_init (argc=3, argv=0x7fbfffed58,
requested=0, provided=0x7fbfffec38) at runtime/ompi_mpi_init.c:510
#9  0x0000002a956f2109 in PMPI_Init (argc=0x7fbfffec7c,
argv=0x7fbfffec70) at pinit.c:88
#10 0x0000000000400bf4 in main (argc=3, argv=0x7fbfffed58) at  
noop.c:39
------------------------------------------------

The 'noop' process on odin024 has a similar backtrace:
------------------------------------------------
(gdb) bt
#0  0x0000002a96226b39 in syscall () from /lib64/tls/libc.so.6
#1  0x0000002a95a1e485 in epoll_wait (epfd=3, events=0x50b390,
maxevents=1023, timeout=1000) at epoll_sub.c:61
#2  0x0000002a95a1e7f7 in epoll_dispatch (base=0x506cc0,  
arg=0x506c20,
tv=0x7fbfffe9d0) at epoll.c:210
#3  0x0000002a95a1c057 in opal_event_base_loop (base=0x506cc0,
flags=5) at event.c:779
#4  0x0000002a95a1be8f in opal_event_loop (flags=5) at event.c:702
#5  0x0000002a95a0bef8 in opal_progress () at runtime/ 
opal_progress.c:
169
#6  0x0000002a958b97c5 in orte_grpcomm_base_allgather
(sbuf=0x7fbfffec70, rbuf=0x7fbfffec10) at base/
grpcomm_base_allgather.c:163
#7  0x0000002a958bd37c in orte_grpcomm_base_modex (procs=0x0) at  
base/
grpcomm_base_modex.c:413
#8  0x0000002a956b8416 in ompi_mpi_init (argc=3, argv=0x7fbfffeee8,
requested=0, provided=0x7fbfffedc8) at runtime/ompi_mpi_init.c:510
#9  0x0000002a956f2109 in PMPI_Init (argc=0x7fbfffee0c,
argv=0x7fbfffee00) at pinit.c:88
#10 0x0000000000400bf4 in main (argc=3, argv=0x7fbfffeee8) at  
noop.c:39
------------------------------------------------


Cheers,
Josh