Thanks! I'll give it a try. My tests are all conducted with fast launches (just running slurm on large clusters) and using an mpi hello world that calls mpi_init at first instruction. I'll see if adding the delay causes it to misbehave.
On Nov 20, 2009, at 6:55 AM, Sylvain Jeaugey wrote: > Hi Ralph, > > Thanks for your efforts. I will look at our configuration and see how it may > differ from ours. > > Here is a patch which helps reproducing the bug even with a small number of > nodes. > > diff -r b622b9e8f1ac orte/orted/orted_comm.c > --- a/orte/orted/orted_comm.c Wed Nov 18 09:27:55 2009 +0100 > +++ b/orte/orted/orted_comm.c Fri Nov 20 14:47:39 2009 +0100 > @@ -126,6 +126,13 @@ > ORTE_ERROR_LOG(ret); > goto CLEANUP; > } > + { /* Add delay to reproduce bug */ > + char * str = getenv("ORTE_RELAY_DELAY"); > + int sec = str ? atoi(str) : 0; > + if (sec) { > + sleep(sec); > + } > + } > } > > CLEANUP: > > Just set ORTE_RELAY_DELAY to 1 (second) and you should reproduce the bug. > > During our experiments, the bug disappeared when we added a delay before > calling MPI_Init. So, configurations where processes are launched slowly or > take some time before MPI_Init should be immune to this bug. > > We usually reproduce the bug with one ppn (faster to spawn). > > Sylvain > > On Thu, 19 Nov 2009, Ralph Castain wrote: > >> Hi Sylvain >> >> I've spent several hours trying to replicate the behavior you described on >> clusters up to a couple of hundred nodes (all running slurm), without >> success. I'm becoming increasingly convinced that this is a configuration >> issue as opposed to a code issue. >> >> I have enclosed the platform file I use below. Could you compare it to your >> configuration? I'm wondering if there is something critical about the config >> that may be causing the problem (perhaps we have a problem in our default >> configuration). >> >> Also, is there anything else you can tell us about your configuration? How >> many ppn triggers it, or do you always get the behavior every time you >> launch over a certain number of nodes? >> >> Meantime, I will look into this further. I am going to introduce a "slow >> down" param that will force the situation you encountered - i.e., will >> ensure that the relay is still being sent when the daemon receives the first >> collective input. We can then use that to try and force replication of the >> behavior you are encountering. >> >> Thanks >> Ralph >> >> enable_dlopen=no >> enable_pty_support=no >> with_blcr=no >> with_openib=yes >> with_memory_manager=no >> enable_mem_debug=yes >> enable_mem_profile=no >> enable_debug_symbols=yes >> enable_binaries=yes >> with_devel_headers=yes >> enable_heterogeneous=no >> enable_picky=yes >> enable_debug=yes >> enable_shared=yes >> enable_static=yes >> with_slurm=yes >> enable_contrib_no_build=libnbc,vt >> enable_visibility=yes >> enable_memchecker=no >> enable_ipv6=no >> enable_mpi_f77=no >> enable_mpi_f90=no >> enable_mpi_cxx=no >> enable_mpi_cxx_seek=no >> enable_mca_no_build=pml-dr,pml-crcp2,crcp >> enable_io_romio=no >> >> On Nov 19, 2009, at 8:08 AM, Ralph Castain wrote: >> >>> >>> On Nov 19, 2009, at 7:52 AM, Sylvain Jeaugey wrote: >>> >>>> Thank you Ralph for this precious help. >>>> >>>> I setup a quick-and-dirty patch basically postponing process_msg (hence >>>> daemon_collective) until the launch is done. In process_msg, I therefore >>>> requeue a process_msg handler and return. >>> >>> That is basically the idea I proposed, just done in a slightly different >>> place >>> >>>> >>>> In this "all-must-be-non-blocking-and-done-through-opal_progress" >>>> algorithm, I don't think that blocking calls like the one in >>>> daemon_collective should be allowed. This also applies to the blocking one >>>> in send_relay. [Well, actually, one is okay, 2 may lead to interlocking.] >>> >>> Well, that would be problematic - you will find "progressed_wait" used >>> repeatedly in the code. Removing them all would take a -lot- of effort and >>> a major rewrite. I'm not yet convinced it is required. There may be >>> something strange in how you are setup, or your cluster - like I said, this >>> is the first report of a problem we have had, and people with much bigger >>> slurm clusters have been running this code every day for over a year. >>> >>>> >>>> If you have time doing a nicer patch, it would be great and I would be >>>> happy to test it. Otherwise, I will try to implement your idea properly >>>> next week (with my limited knowledge of orted). >>> >>> Either way is fine - I'll see if I can get to it. >>> >>> Thanks >>> Ralph >>> >>>> >>>> For the record, here is the patch I'm currently testing at large scale : >>>> >>>> diff -r ec68298b3169 -r b622b9e8f1ac >>>> orte/mca/grpcomm/bad/grpcomm_bad_module.c >>>> --- a/orte/mca/grpcomm/bad/grpcomm_bad_module.c Mon Nov 09 13:29:16 2009 >>>> +0100 >>>> +++ b/orte/mca/grpcomm/bad/grpcomm_bad_module.c Wed Nov 18 09:27:55 2009 >>>> +0100 >>>> @@ -687,14 +687,6 @@ >>>> opal_list_append(&orte_local_jobdata, &jobdat->super); >>>> } >>>> >>>> - /* it may be possible to get here prior to having actually finished >>>> processing our >>>> - * local launch msg due to the race condition between different nodes >>>> and when >>>> - * they start their individual procs. Hence, we have to first ensure >>>> that we >>>> - * -have- finished processing the launch msg, or else we won't know >>>> whether >>>> - * or not to wait before sending this on >>>> - */ >>>> - ORTE_PROGRESSED_WAIT(jobdat->launch_msg_processed, 0, 1); >>>> - >>>> /* unpack the collective type */ >>>> n = 1; >>>> if (ORTE_SUCCESS != (rc = opal_dss.unpack(data, >>>> &jobdat->collective_type, &n, ORTE_GRPCOMM_COLL_T))) { >>>> @@ -894,6 +886,28 @@ >>>> >>>> proc = &mev->sender; >>>> buf = mev->buffer; >>>> + >>>> + jobdat = NULL; >>>> + for (item = opal_list_get_first(&orte_local_jobdata); >>>> + item != opal_list_get_end(&orte_local_jobdata); >>>> + item = opal_list_get_next(item)) { >>>> + jobdat = (orte_odls_job_t*)item; >>>> + >>>> + /* is this the specified job? */ >>>> + if (jobdat->jobid == proc->jobid) { >>>> + break; >>>> + } >>>> + } >>>> + if (NULL == jobdat || jobdat->launch_msg_processed != 1) { >>>> + /* it may be possible to get here prior to having actually >>>> finished processing our >>>> + * local launch msg due to the race condition between different >>>> nodes and when >>>> + * they start their individual procs. Hence, we have to first >>>> ensure that we >>>> + * -have- finished processing the launch msg. Requeue this event >>>> until it is done. >>>> + */ >>>> + int tag = &mev->tag; >>>> + ORTE_MESSAGE_EVENT(proc, buf, tag, process_msg); >>>> + return; >>>> + } >>>> >>>> /* is the sender a local proc, or a daemon relaying the collective? */ >>>> if (ORTE_PROC_MY_NAME->jobid == proc->jobid) { >>>> >>>> Sylvain >>>> >>>> On Thu, 19 Nov 2009, Ralph Castain wrote: >>>> >>>>> Very strange. As I said, we routinely launch jobs spanning several >>>>> hundred nodes without problem. You can see the platform files for that >>>>> setup in contrib/platform/lanl/tlcc >>>>> >>>>> That said, it is always possible you are hitting some kind of race >>>>> condition we don't hit. In looking at the code, one possibility would be >>>>> to make all the communications flow through the daemon cmd processor in >>>>> orte/orted_comm.c. This is the way it used to work until I reorganized >>>>> the code a year ago for other reasons that never materialized. >>>>> >>>>> Unfortunately, the daemon collective has to wait until the local launch >>>>> cmd has been completely processed so it can know whether or not to wait >>>>> for contributions from local procs before sending along the collective >>>>> message, so this kinda limits our options. >>>>> >>>>> About the only other thing you could do would be to not send the relay at >>>>> all until -after- processing the local launch cmd. You can then remove >>>>> the "wait" in the daemon collective as you will know how many local procs >>>>> are involved, if any. >>>>> >>>>> I used to do it that way and it guarantees it will work. The negative is >>>>> that we lose some launch speed as the next nodes in the tree don't get >>>>> the launch message until this node finishes launching all its procs. >>>>> >>>>> The way around that, of course, would be to: >>>>> >>>>> 1. process the launch message, thus extracting the number of any local >>>>> procs and setting up all data structures...but do -not- launch the procs >>>>> at this time (as this is what takes all the time) >>>>> >>>>> 2. send the relay - the daemon collective can now proceed without a >>>>> "wait" in it >>>>> >>>>> 3. now launch the local procs >>>>> >>>>> It would be a fairly simple reorganization of the code in the >>>>> orte/mca/odls area. I can do it this weekend if you like, or you can do >>>>> it - either way is fine, but if you do it, please contribute it back to >>>>> the trunk. >>>>> >>>>> Ralph >>>>> >>>>> >>>>> On Nov 19, 2009, at 1:39 AM, Sylvain Jeaugey wrote: >>>>> >>>>>> I would say I use the default settings, i.e. I don't set anything >>>>>> "special" at configure. >>>>>> >>>>>> I'm launching my processes with SLURM (salloc + mpirun). >>>>>> >>>>>> Sylvain >>>>>> >>>>>> On Wed, 18 Nov 2009, Ralph Castain wrote: >>>>>> >>>>>>> How did you configure OMPI? >>>>>>> >>>>>>> What launch mechanism are you using - ssh? >>>>>>> >>>>>>> On Nov 17, 2009, at 9:01 AM, Sylvain Jeaugey wrote: >>>>>>> >>>>>>>> I don't think so, and I'm not doing it explicitely at least. How do I >>>>>>>> know ? >>>>>>>> >>>>>>>> Sylvain >>>>>>>> >>>>>>>> On Tue, 17 Nov 2009, Ralph Castain wrote: >>>>>>>> >>>>>>>>> We routinely launch across thousands of nodes without a problem...I >>>>>>>>> have never seen it stick in this fashion. >>>>>>>>> >>>>>>>>> Did you build and/or are using ORTE threaded by any chance? If so, >>>>>>>>> that definitely won't work. >>>>>>>>> >>>>>>>>> On Nov 17, 2009, at 9:27 AM, Sylvain Jeaugey wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> We are currently experiencing problems at launch on the 1.5 branch >>>>>>>>>> on relatively large number of nodes (at least 80). Some processes >>>>>>>>>> are not spawned and orted processes are deadlocked. >>>>>>>>>> >>>>>>>>>> When MPI processes are calling MPI_Init before send_relay is >>>>>>>>>> complete, the send_relay function and the daemon_collective function >>>>>>>>>> are doing a nice interlock : >>>>>>>>>> >>>>>>>>>> Here is the scenario : >>>>>>>>>>> send_relay >>>>>>>>>> performs the send tree : >>>>>>>>>>> orte_rml_oob_send_buffer >>>>>>>>>>> orte_rml_oob_send >>>>>>>>>>> opal_wait_condition >>>>>>>>>> Waiting on completion from send thus calling opal_progress() >>>>>>>>>>> opal_progress() >>>>>>>>>> But since a collective request arrived from the network, entered : >>>>>>>>>>> daemon_collective >>>>>>>>>> However, daemon_collective is waiting for the job to be initialized >>>>>>>>>> (wait on jobdat->launch_msg_processed) before continuing, thus >>>>>>>>>> calling : >>>>>>>>>>> opal_progress() >>>>>>>>>> >>>>>>>>>> At this time, the send may complete, but since we will never go back >>>>>>>>>> to orte_rml_oob_send, we will never perform the launch (setting >>>>>>>>>> jobdat->launch_msg_processed to 1). >>>>>>>>>> >>>>>>>>>> I may try to solve the bug (this is quite a top priority problem for >>>>>>>>>> me), but maybe people who are more familiar with orted than I am may >>>>>>>>>> propose a nice and clean solution ... >>>>>>>>>> >>>>>>>>>> For those who like real (and complete) gdb stacks, here they are : >>>>>>>>>> #0 0x0000003b7fed4f38 in poll () from /lib64/libc.so.6 >>>>>>>>>> #1 0x00007fd0de5d861a in poll_dispatch (base=0x930230, >>>>>>>>>> arg=0x91a4b0, tv=0x7fff0d977880) at poll.c:167 >>>>>>>>>> #2 0x00007fd0de5d586f in opal_event_base_loop (base=0x930230, >>>>>>>>>> flags=1) at event.c:823 >>>>>>>>>> #3 0x00007fd0de5d556b in opal_event_loop (flags=1) at event.c:746 >>>>>>>>>> #4 0x00007fd0de5aeb6d in opal_progress () at >>>>>>>>>> runtime/opal_progress.c:189 >>>>>>>>>> #5 0x00007fd0dd340a02 in daemon_collective (sender=0x97af50, >>>>>>>>>> data=0x97b010) at grpcomm_bad_module.c:696 >>>>>>>>>> #6 0x00007fd0dd341809 in process_msg (fd=-1, opal_event=1, >>>>>>>>>> data=0x97af20) at grpcomm_bad_module.c:901 >>>>>>>>>> #7 0x00007fd0de5d5334 in event_process_active (base=0x930230) at >>>>>>>>>> event.c:667 >>>>>>>>>> #8 0x00007fd0de5d597a in opal_event_base_loop (base=0x930230, >>>>>>>>>> flags=1) at event.c:839 >>>>>>>>>> #9 0x00007fd0de5d556b in opal_event_loop (flags=1) at event.c:746 >>>>>>>>>> #10 0x00007fd0de5aeb6d in opal_progress () at >>>>>>>>>> runtime/opal_progress.c:189 >>>>>>>>>> #11 0x00007fd0dd340a02 in daemon_collective (sender=0x979700, >>>>>>>>>> data=0x9676e0) at grpcomm_bad_module.c:696 >>>>>>>>>> #12 0x00007fd0dd341809 in process_msg (fd=-1, opal_event=1, >>>>>>>>>> data=0x9796d0) at grpcomm_bad_module.c:901 >>>>>>>>>> #13 0x00007fd0de5d5334 in event_process_active (base=0x930230) at >>>>>>>>>> event.c:667 >>>>>>>>>> #14 0x00007fd0de5d597a in opal_event_base_loop (base=0x930230, >>>>>>>>>> flags=1) at event.c:839 >>>>>>>>>> #15 0x00007fd0de5d556b in opal_event_loop (flags=1) at event.c:746 >>>>>>>>>> #16 0x00007fd0de5aeb6d in opal_progress () at >>>>>>>>>> runtime/opal_progress.c:189 >>>>>>>>>> #17 0x00007fd0dd340a02 in daemon_collective (sender=0x97b420, >>>>>>>>>> data=0x97b4e0) at grpcomm_bad_module.c:696 >>>>>>>>>> #18 0x00007fd0dd341809 in process_msg (fd=-1, opal_event=1, >>>>>>>>>> data=0x97b3f0) at grpcomm_bad_module.c:901 >>>>>>>>>> #19 0x00007fd0de5d5334 in event_process_active (base=0x930230) at >>>>>>>>>> event.c:667 >>>>>>>>>> #20 0x00007fd0de5d597a in opal_event_base_loop (base=0x930230, >>>>>>>>>> flags=1) at event.c:839 >>>>>>>>>> #21 0x00007fd0de5d556b in opal_event_loop (flags=1) at event.c:746 >>>>>>>>>> #22 0x00007fd0de5aeb6d in opal_progress () at >>>>>>>>>> runtime/opal_progress.c:189 >>>>>>>>>> #23 0x00007fd0dd969a8a in opal_condition_wait (c=0x97b210, >>>>>>>>>> m=0x97b1a8) at ../../../../opal/threads/condition.h:99 >>>>>>>>>> #24 0x00007fd0dd96a4bf in orte_rml_oob_send (peer=0x7fff0d9785a0, >>>>>>>>>> iov=0x7fff0d978530, count=1, tag=1, flags=16) at rml_oob_send.c:153 >>>>>>>>>> #25 0x00007fd0dd96ac4d in orte_rml_oob_send_buffer >>>>>>>>>> (peer=0x7fff0d9785a0, buffer=0x7fff0d9786b0, tag=1, flags=0) at >>>>>>>>>> rml_oob_send.c:270 >>>>>>>>>> #26 0x00007fd0de86ed2a in send_relay (buf=0x7fff0d9786b0) at >>>>>>>>>> orted/orted_comm.c:127 >>>>>>>>>> #27 0x00007fd0de86f6de in orte_daemon_cmd_processor (fd=-1, >>>>>>>>>> opal_event=1, data=0x965fc0) at orted/orted_comm.c:308 >>>>>>>>>> #28 0x00007fd0de5d5334 in event_process_active (base=0x930230) at >>>>>>>>>> event.c:667 >>>>>>>>>> #29 0x00007fd0de5d597a in opal_event_base_loop (base=0x930230, >>>>>>>>>> flags=0) at event.c:839 >>>>>>>>>> #30 0x00007fd0de5d556b in opal_event_loop (flags=0) at event.c:746 >>>>>>>>>> #31 0x00007fd0de5d5418 in opal_event_dispatch () at event.c:682 >>>>>>>>>> #32 0x00007fd0de86e339 in orte_daemon (argc=19, argv=0x7fff0d979ca8) >>>>>>>>>> at orted/orted_main.c:769 >>>>>>>>>> #33 0x00000000004008e2 in main (argc=19, argv=0x7fff0d979ca8) at >>>>>>>>>> orted.c:62 >>>>>>>>>> >>>>>>>>>> Thanks in advance, >>>>>>>>>> Sylvain >>>>>>>>>> _______________________________________________ >>>>>>>>>> devel mailing list >>>>>>>>>> de...@open-mpi.org >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> devel mailing list >>>>>>>>> de...@open-mpi.org >>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>>>> >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> devel mailing list >>>>>>>> de...@open-mpi.org >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> devel mailing list >>>>>>> de...@open-mpi.org >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> devel mailing list >>>>>> de...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>> >>>>> >>>>> _______________________________________________ >>>>> devel mailing list >>>>> de...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>> >>>>> >>>> _______________________________________________ >>>> devel mailing list >>>> de...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>> >> >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> >> > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel