That's how it works now. My concern is with the error message scenario. IIRC, Jeff's issue was that the error message only contains the hostname of the proc that generates it - it doesn't tell you the hostname of the remote proc. Hence, we included that info in the proc_t.
However, IIRC we also provided an option to *not* send that info due to scaling concerns way back when. I wonder if we can resolve this simply by having Nathan set that option in his platform .conf files, and then removing ompi_proc_get_hostname completely. Since the IP-based comm channels will call modex_recv anyway, we'll get the hostname at that time. Otherwise, the errors print "NULL" for proc->hostname. Yes, that means that users of direct-launched apps on Nathan's systems will get less informative error messages - but they can always override Nathan's default param if they want better info. After all, the vast majority of users aren't running such big jobs as to care about this optimization. Make sense? Ralph On Aug 19, 2013, at 4:36 PM, George Bosilca <bosi...@icl.utk.edu> wrote: > Some networks need the name to resolve their own communication channel so > they will look it up (all based on IP). However, this is indeed not enough > for all cases. The general solution will be to set the proc hostname the > first time the peer information is looked up in the database > (ompi_modex_recv_* functions as they already have the proc pointer), and then > cached it for later usage in the "->proc_hostname". > > George. > > On Aug 20, 2013, at 01:23 , Ralph Castain <r...@open-mpi.org> wrote: > >> Hmmm...but then proc->hostname will *never* be filled in, because it is only >> ever accessed in an error message - i.e., in opal_output and its variants. >> >> If we are not going to retrieve it be default, then we need another solution >> *if* we want hostnames for error messages under direct launch. If we don't >> care, then we can ignore this issue and follow the proposal. >> >> I suppose one could ask why we are even bothering with hostname since the >> opal_output message includes the hostname in its prefix anyway. >> >> Jeff: this was your baby - what do you think? >> >> >> On Aug 19, 2013, at 3:43 PM, Nathan Hjelm <hje...@lanl.gov> wrote: >> >>> That solution is fine with me. >>> >>> -Nathan >>> >>> On Tue, Aug 20, 2013 at 12:41:49AM +0200, George Bosilca wrote: >>>> If your offer is between quadratic and non-deterministic, I'll take the >>>> former. >>>> >>>> I would advocate for a middle-ground solution. Clearly document in the >>>> header file that the ompi_proc_get_hostname is __not__ safe to be used in >>>> all contexts as it might exhibit recursive behavior due to communications. >>>> Then revert all its uses in the context of opal_output, >>>> opal_output_verbose and all variants back to using "->proc_hostname". We >>>> might get a (null) instead of the peer name, but this removes the >>>> potential loops. >>>> >>>> George. >>>> >>>> On Aug 19, 2013, at 23:52 , Nathan Hjelm <hje...@lanl.gov> wrote: >>>> >>>>> It would require a db read from every rank which is what we are trying >>>>> to avoid. This scales quadratic at best on Cray systems. >>>>> >>>>> -Nathan >>>>> >>>>> On Mon, Aug 19, 2013 at 02:48:18PM -0700, Ralph Castain wrote: >>>>>> Yeah, I have some concerns about it too...been trying to test it out >>>>>> some more. Would be good to see just how much that one change makes - >>>>>> maybe restoring just the hostname wouldn't have that big an impact. >>>>>> >>>>>> I'm leery of trying to ensure we strip all the opal_output loops if we >>>>>> don't find the hostname. >>>>>> >>>>>> On Aug 19, 2013, at 2:41 PM, George Bosilca <bosi...@icl.utk.edu> wrote: >>>>>> >>>>>>> As a result of this patch the first decode of a peer host name might >>>>>>> happen in the middle of a debug message (on the first call to >>>>>>> ompi_proc_get_hostname). Such a behavior might generate deadlocks based >>>>>>> on the level of output verbosity, and has significant potential to >>>>>>> reintroduce the recursive behavior the new state machine was supposed >>>>>>> to remove. >>>>>>> >>>>>>> George. >>>>>>> >>>>>>> >>>>>>> On Aug 17, 2013, at 02:49 , svn-commit-mai...@open-mpi.org wrote: >>>>>>> >>>>>>>> Author: rhc (Ralph Castain) >>>>>>>> Date: 2013-08-16 20:49:18 EDT (Fri, 16 Aug 2013) >>>>>>>> New Revision: 29040 >>>>>>>> URL: https://svn.open-mpi.org/trac/ompi/changeset/29040 >>>>>>>> >>>>>>>> Log: >>>>>>>> When we direct launch an application, we rely on PMI for wireup >>>>>>>> support. In doing so, we lose the de facto data compression we get >>>>>>>> from the ORTE modex since we no longer get all the wireup info from >>>>>>>> every proc in a single blob. Instead, we have to iterate over all the >>>>>>>> procs, calling PMI_KVS_get for every value we require. >>>>>>>> >>>>>>>> This creates a really bad scaling behavior. Users have found a nearly >>>>>>>> 20% launch time differential between mpirun and PMI, with PMI being >>>>>>>> the slower method. Some of the problem is attributable to poor >>>>>>>> exchange algorithms in RM's like Slurm and Alps, but we make things >>>>>>>> worse by calling "get" so many times. >>>>>>>> >>>>>>>> Nathan (with a tad advice from me) has attempted to alleviate this >>>>>>>> problem by reducing the number of "get" calls. This required the >>>>>>>> following changes: >>>>>>>> >>>>>>>> * upon first request for data, have the OPAL db pmi component fetch >>>>>>>> and decode *all* the info from a given remote proc. It turned out we >>>>>>>> weren't caching the info, so we would continually request it and only >>>>>>>> decode the piece we needed for the immediate request. We now decode >>>>>>>> all the info and push it into the db hash component for local storage >>>>>>>> - and then all subsequent retrievals are fulfilled locally >>>>>>>> >>>>>>>> * reduced the amount of data by eliminating the exchange of the >>>>>>>> OMPI_ARCH value if heterogeneity is not enabled. This was used solely >>>>>>>> as a check so we would error out if the system wasn't actually >>>>>>>> homogeneous, which was fine when we thought there was no cost in doing >>>>>>>> the check. Unfortunately, at large scale and with direct launch, there >>>>>>>> is a non-zero cost of making this test. We are open to finding a >>>>>>>> compromise (perhaps turning the test off if requested?), if people >>>>>>>> feel strongly about performing the test >>>>>>>> >>>>>>>> * reduced the amount of RTE data being automatically fetched, and >>>>>>>> fetched the rest only upon request. In particular, we no longer >>>>>>>> immediately fetch the hostname (which is only used for error >>>>>>>> reporting), but instead get it when needed. Likewise for the RML uri >>>>>>>> as that info is only required for some (not all) environments. In >>>>>>>> addition, we no longer fetch the locality unless required, relying >>>>>>>> instead on the PMI clique info to tell us who is on our local node (if >>>>>>>> additional info is required, the fetch is performed when a modex_recv >>>>>>>> is issued). >>>>>>>> >>>>>>>> Again, all this only impacts direct launch - all the info is provided >>>>>>>> when launched via mpirun as there is no added cost to getting it >>>>>>>> >>>>>>>> Barring objections, we may move this (plus any required other pieces) >>>>>>>> to the 1.7 branch once it soaks for an appropriate time. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> devel mailing list >>>>>>> de...@open-mpi.org >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>>> >>>>>> _______________________________________________ >>>>>> devel mailing list >>>>>> de...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>>> _______________________________________________ >>>>> devel mailing list >>>>> de...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>>> >>>> _______________________________________________ >>>> devel mailing list >>>> de...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >>> _______________________________________________ >>> devel mailing list >>> de...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
