On Oct 10, 2011, at 11:29 AM, Ralph Castain wrote: > > On Oct 10, 2011, at 11:14 AM, George Bosilca wrote: > >> Ralph, >> >> If you don't mind I would like to understand this issue a little bit more. >> What exactly is broken in the termination detection? >> >>> From a network point of view, there is a slight issue with the commit >>> 25245. A direct call to exit will close all pending sockets, with a linger >>> of 60 seconds (quite bad if you use static ports as an example). There are >>> proper protocols to shutdown sockets in a reliable way, maybe it is time to >>> implement one of them.
I should have clarified something here. The change in r25245 doesn't cause the daemons to directly call exit. It causes them to call orte_quit, which runs thru orte_finalize on its way out. So if the oob properly shuts down its sockets, then there will be no issue with lingering sockets. The question has always been: does the oob properly shut down sockets? If someone has time to address that issue, it would indeed be helpful. > > Did I miss this message somewhere? I didn't receive it. > > The problem is that the daemons with children are not seeing the sockets > close when their child daemons terminate. So the lowest layer of daemons > terminate (as they have no children). The next layer does not terminate, > leaving mpirun hanging. > > What's interesting is that mpirun does see the sockets directly connected to > it close. As you probably recall, the binomial routed module has daemons > directly callback to mpirun, so that connection is created. The socket to > the next level of daemon gets created later - it is this later socket whose > closure isn't being detected. > > From what I could see, it appeared to be a progress issue. The socket may be > closed, but the event lib may not be progressing to detect it. As we intend > to fix that anyway with the async progress work, and static ports are not the > default behavior (and rarely used), I didn't feel it worth leaving the trunk > hanging while continuing to chase it. > > If someone has time, properly closing the sockets is something we've talked > about for quite awhile, but never gotten around to doing. I'm not sure it > will resolve this problem, but would be worth doing anyway. > > >> >> Thanks, >> george. >> >> On Oct 10, 2011, at 12:40 , Ralph Castain wrote: >> >>> It wasn't the launcher that was broken, but termination detection, and not >>> for all environments (e.g., worked fine for slurm). It is a >>> progress-related issue. >>> >>> Should be fixed in r25245. >>> >>> >>> On Oct 10, 2011, at 8:33 AM, Shamis, Pavel wrote: >>> >>>> + 1 , I see the same issue. >>>> >>>>> -----Original Message----- >>>>> From: devel-boun...@open-mpi.org [mailto:devel-boun...@open-mpi.org] >>>>> On Behalf Of Yevgeny Kliteynik >>>>> Sent: Monday, October 10, 2011 10:24 AM >>>>> To: OpenMPI Devel >>>>> Subject: [OMPI devel] Launcher in trunk is broken? >>>>> >>>>> It looks like the process launcher is broken in the OMPI trunk: >>>>> If you run any simple test (not necessarily including MPI calls) on 4 or >>>>> more nodes, the MPI processes won't be killed after the test finishes. >>>>> >>>>> $ mpirun -host host_1,host_2,host_3,host_4 -np 4 --mca btl sm,tcp,self >>>>> /bin/hostname >>>>> >>>>> Output: >>>>> host_1 >>>>> host_2 >>>>> host_3 >>>>> host_4 >>>>> >>>>> And test is hanging...... >>>>> >>>>> I have an older trunk (r25228), and everything is OK there. >>>>> Not sure if it means that something was broken after that, or the problem >>>>> existed before, but kicked in only now due to some other change. >>>>> >>>>> -- YK >>>>> _______________________________________________ >>>>> devel mailing list >>>>> de...@open-mpi.org >>>>> hxxp://www.open-mpi.org/mailman/listinfo.cgi/devel >>>> >>>> >>>> _______________________________________________ >>>> devel mailing list >>>> de...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>> >>> >>> _______________________________________________ >>> devel mailing list >>> de...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >
