I figured out the issue - there is a simple and a hard way to fix this. So before I do, let me see what makes sense.
The simple solution involves updating the daemons with contact info for the procs so that they can send their collected modex info to the rank=0 proc. This will measurably slow the launch when using unity. The hard solution is to do a hybrid routed approach whereby the daemons would route any daemon-to-proc communication while the procs continue to do direct proc-to-proc messaging. Is there some reason to be using the "unity" component? Do you care if jobs using unity launch slower? Thanks Ralph On 3/31/08 7:57 AM, "Josh Hursey" <jjhur...@open-mpi.org> wrote: > Ralph, > > I've just noticed that it seems that the 'unity' routed component > seems to be broken when using more than one machine. I'm using Odin > and r18028 of the trunk, and have confirmed that this problem occurs > with SLURM and rsh. I think this break came in on Friday as that is > when some of my MTT tests started to hang and fail, but I cannot point > to a specific revision at this point. The backtraces (enclosed) of the > processes point to the grpcomm allgather routine. > > The 'noop' program calls MPI_Init, sleeps, then calls MPI_Finalize. > > RSH example from odin023 - so no SLURM variables: > These work: > shell$ mpirun -np 2 -host odin023 noop -v 1 > shell$ mpirun -np 2 -host odin023,odin024 noop -v 1 > shell$ mpirun -np 2 -mca routed unity -host odin023 noop -v 1 > > This hangs: > shell$ mpirun -np 2 -mca routed unity -host odin023,odin024 noop -v 1 > > > If I attach to the 'noop' process on odin023 I get the following > backtrace: > ------------------------------------------------ > (gdb) bt > #0 0x0000002a96226b39 in syscall () from /lib64/tls/libc.so.6 > #1 0x0000002a95a1e485 in epoll_wait (epfd=3, events=0x50b330, > maxevents=1023, timeout=1000) at epoll_sub.c:61 > #2 0x0000002a95a1e7f7 in epoll_dispatch (base=0x506c30, arg=0x506910, > tv=0x7fbfffe840) at epoll.c:210 > #3 0x0000002a95a1c057 in opal_event_base_loop (base=0x506c30, > flags=5) at event.c:779 > #4 0x0000002a95a1be8f in opal_event_loop (flags=5) at event.c:702 > #5 0x0000002a95a0bef8 in opal_progress () at runtime/opal_progress.c: > 169 > #6 0x0000002a958b9e48 in orte_grpcomm_base_allgather > (sbuf=0x7fbfffeae0, rbuf=0x7fbfffea80) at base/ > grpcomm_base_allgather.c:238 > #7 0x0000002a958bd37c in orte_grpcomm_base_modex (procs=0x0) at base/ > grpcomm_base_modex.c:413 > #8 0x0000002a956b8416 in ompi_mpi_init (argc=3, argv=0x7fbfffed58, > requested=0, provided=0x7fbfffec38) at runtime/ompi_mpi_init.c:510 > #9 0x0000002a956f2109 in PMPI_Init (argc=0x7fbfffec7c, > argv=0x7fbfffec70) at pinit.c:88 > #10 0x0000000000400bf4 in main (argc=3, argv=0x7fbfffed58) at noop.c:39 > ------------------------------------------------ > > The 'noop' process on odin024 has a similar backtrace: > ------------------------------------------------ > (gdb) bt > #0 0x0000002a96226b39 in syscall () from /lib64/tls/libc.so.6 > #1 0x0000002a95a1e485 in epoll_wait (epfd=3, events=0x50b390, > maxevents=1023, timeout=1000) at epoll_sub.c:61 > #2 0x0000002a95a1e7f7 in epoll_dispatch (base=0x506cc0, arg=0x506c20, > tv=0x7fbfffe9d0) at epoll.c:210 > #3 0x0000002a95a1c057 in opal_event_base_loop (base=0x506cc0, > flags=5) at event.c:779 > #4 0x0000002a95a1be8f in opal_event_loop (flags=5) at event.c:702 > #5 0x0000002a95a0bef8 in opal_progress () at runtime/opal_progress.c: > 169 > #6 0x0000002a958b97c5 in orte_grpcomm_base_allgather > (sbuf=0x7fbfffec70, rbuf=0x7fbfffec10) at base/ > grpcomm_base_allgather.c:163 > #7 0x0000002a958bd37c in orte_grpcomm_base_modex (procs=0x0) at base/ > grpcomm_base_modex.c:413 > #8 0x0000002a956b8416 in ompi_mpi_init (argc=3, argv=0x7fbfffeee8, > requested=0, provided=0x7fbfffedc8) at runtime/ompi_mpi_init.c:510 > #9 0x0000002a956f2109 in PMPI_Init (argc=0x7fbfffee0c, > argv=0x7fbfffee00) at pinit.c:88 > #10 0x0000000000400bf4 in main (argc=3, argv=0x7fbfffeee8) at noop.c:39 > ------------------------------------------------ > > > Cheers, > Josh
