Thanks for clarifying, I now understand what your objection/suggestion was. We all misconfigured OMPI at least once, but that allowed us to learn how to do it right.
Instead of adding extra protections for corner-cases, maybe we should fix our exclusivity flag so that the scenario you describe would not happen. George. PS: "btl_tcp_if_exclude = ^ib0" qualifies as a honest mistake. I wouldn't dare proposing a new MCA param to prevent this ... On Wed, Sep 21, 2016 at 10:54 PM, Gilles Gouaillardet < gilles.gouaillar...@gmail.com> wrote: > ok, i was not clear > > by "let's consider the case where "lo" is *not* excluded via the > btl_tcp_if_exclude MCA param" i really meant > "let's consider the case where the value of the btl_tcp_if_exclude MCA > param has been forced to a list of network/interfaces that do not > contain any reference (e.g. name nor subnet) to the loopback > interface" > /* in a previous example, i did mpirun --mca btl_tcp_if_exclude ^ib0 */ > > my concern is that openmpi-mca-params.conf contains > btl_tcp_if_exclude = ^ib0 > > then hiccups will start when Open MPI is updated, and i expect some > complains. > of course we can reply, doc should have been read and advices > followed, so one cannot complain just because he has been lucky so > far. > or we can do things a bit differently so we do not run into this case > > /* if btl/self is excluded, the app will not start and it is trivial > to append to the error message a note asking to ensure btl/self was > not excluded. > in this case, i do not think we have a mechanism to issue a warning > message (e.g. "ensure lo is excluded") when hiccups occur. */ > > Cheers, > > Gilles > > On Thu, Sep 22, 2016 at 9:54 AM, George Bosilca <bosi...@icl.utk.edu> > wrote: > > On Wednesday, September 21, 2016, Gilles Gouaillardet > > <gilles.gouaillar...@gmail.com> wrote: > >> > >> George, > >> > >> let's consider the case where "lo" is *not* excluded via the > >> btl_tcp_if_exclude MCA param > >> (if i understand correctly, the following is also true if "lo" is > >> included via the btl_tcp_if_include MCA param) > >> > >> currently, and because of/thanks to the test that is done "deep inside" > >> 1) on a disconnected laptop, mpirun --mca btl tcp,self ... fails with > >> 2 tasks or more because tasks cannot reach each other > >> 2) on a (connected) cluster, "lo" is never used and mpirun --mca btl > >> tcp,self ... does not hang when tasks are running on two nodes or more > >> > >> with your proposal : > >> 3) on a disconnected laptop, mpirun --mca btl tcp,self ... works with > >> any number of taks, because "lo" is used by btl/tcp > >> 4) on a (connected) cluster, "lo" is used and mpirun --mca btl > >> tcp,self ... will very likely hang when tasks are running on two nodes > >> or more > >> > >> am i right so far ? > > > > > > No, you are missing the fact that thanks to our if_exclude (which > contains > > by default 127.0.0.0/24) we will never use lo (not even with my patch). > > Thus, local interfaces will remain out of reach for most users, with the > > exception of those that manually force the inclusion of lo via > if_include. > > > > On a cluster where a user explicitly enable lo, there will be some > hiccups > > during startup. However, as Paul states we explicitly discourage people > of > > doing that in the README. Second, the connection over lo will eventually > > timeout, and lo it will be dropped and all pending communications will be > > redirected through another TCP interface. > > > > Cheers, > > George. > > > > > >> > >> my concern is 4) > >> as Paul pointed out, we can consider this is not an issue since this > >> is a user/admin mistake, and we do not care whether this is an honest > >> one or not. that being said, this is not very friendly since something > >> that is working fine today will (likely) start hanging when your patch > >> is merged. > >> > >> my suggestion differs since it is basically 2) and 3), which can be > >> seen as the best of both worlds > >> > >> makes sense ? > >> > >> as a side note, there were some discussions about automatically adding > >> the self btl, > >> and even offering a user friendly alternative to --mca btl xxx > >> (for example --networks shm,infiniband. today Open MPI does not > >> provide any alternative to btl/self. also infiniband can be used via > >> btl/openib, mtl/mxm or libfabric, which makes it painful to > >> blacklist). i cannot remember the outcome of the discussion (if any). > >> > >> Cheers, > >> > >> Gilles > >> > >> On Thu, Sep 22, 2016 at 4:57 AM, George Bosilca <bosi...@icl.utk.edu> > >> wrote: > >> > Gilles, > >> > > >> > I don't understand how your proposal is any different than what we > have > >> > today. I quote "If [locality flag is set], then we could keep a hard > >> > coded > >> > test so 127.x.y.z address (and IPv6 equivalent) are never used (even > if > >> > included or not excluded) for inter node communication". We already > have > >> > a > >> > hardcoded test to prevent 127.x.y.z addresses from being used. In fact > >> > we > >> > have 2 tests, one because this address range is part of our default > >> > if_exclude, and then a second test (that only does something useful in > >> > case > >> > you manually added lo* to if_include) deep inside the IP matching > logic. > >> > > >> > George. > >> > > >> > > >> > On Wed, Sep 21, 2016 at 12:36 PM, Gilles Gouaillardet > >> > <gilles.gouaillar...@gmail.com> wrote: > >> >> > >> >> George, > >> >> > >> >> i got that, and i consider my suggestion as an improvement to your > >> >> proposal. > >> >> > >> >> if i want to exclude ib0, i might want to > >> >> mpirun --mca btl_tcp_if_exclude ib0 ... > >> >> > >> >> to me, this is an honest mistake, but with your proposal, i would be > >> >> screwed when > >> >> running on more than one node because i should have > >> >> mpirun --mca btl_tcp_if_exclude ib0,lo ... > >> >> > >> >> and if this parameter is set by the admin in the system-wide config, > >> >> then this configuration must be adapted by the admin, and that could > >> >> generate some confusion. > >> >> > >> >> my suggestion simply adds a "safety net" to your proposal > >> >> > >> >> for the sake of completion, i do not really care whether there should > >> >> be a safety net or not if localhost is explicitly included via the > the > >> >> btl_tcp_if_include MCA parameter > >> >> > >> >> a different and safe/friendly proposal is to add a new > >> >> btl_tcp_if_exclude_localhost MCA param, which is true by default, so > >> >> you would simply force it to false if you want to MPI_Comm_spawn or > >> >> use the tcp btl on your disconnected laptop. > >> >> > >> >> as a side note, this reminds me that the openib/btl is used by > default > >> >> for intra node communication between two tasks from different jobs > (sm > >> >> nor vader cannot be used yet, and btl/openib has a higher exclusivity > >> >> than btl/tcp). my first impression is that i am not so comfortable > >> >> with that, and we could add yet an other MCA parameter so btl/openib > >> >> disqualifies itself for intra node communications. > >> >> > >> >> > >> >> Cheers, > >> >> > >> >> Gilles > >> >> > >> >> On Thu, Sep 22, 2016 at 12:56 AM, George Bosilca < > bosi...@icl.utk.edu> > >> >> wrote: > >> >> > My proposal is not about adding new ways of deciding what is local > >> >> > and > >> >> > what > >> >> > not. I proposed to use the corresponding MCA parameters to allow > the > >> >> > user to > >> >> > decide. More specifically, I want to be able to change the exclude > >> >> > and > >> >> > include MCA to enable TCP over local addresses. > >> >> > > >> >> > George > >> >> > > >> >> > > >> >> > On Sep 21, 2016 4:32 PM, "Gilles Gouaillardet" > >> >> > <gilles.gouaillar...@gmail.com> wrote: > >> >> >> > >> >> >> George, > >> >> >> > >> >> >> Is proc locality already set at that time ? > >> >> >> > >> >> >> If yes, then we could keep a hard coded test so 127.x.y.z address > >> >> >> (and > >> >> >> IPv6 equivalent) are never used (even if included or not excluded) > >> >> >> for > >> >> >> inter > >> >> >> node communication > >> >> >> > >> >> >> Cheers, > >> >> >> > >> >> >> Gilles > >> >> >> > >> >> >> "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> wrote: > >> >> >> >On Sep 21, 2016, at 10:56 AM, George Bosilca < > bosi...@icl.utk.edu> > >> >> >> > wrote: > >> >> >> >> > >> >> >> >> No, because 127.x.x.x is by default part of the exclude, so it > >> >> >> >> will > >> >> >> >> never get into the modex. The problem today, is that even if > you > >> >> >> >> manually > >> >> >> >> remove it from the exclude and add it to the include, it will > not > >> >> >> >> work, > >> >> >> >> because of the hardcoded checks. Once we remove those checks, > >> >> >> >> things > >> >> >> >> will > >> >> >> >> work the way we expect, interfaces are removed because they > don't > >> >> >> >> match the > >> >> >> >> provided addresses. > >> >> >> > > >> >> >> >Gotcha. > >> >> >> > > >> >> >> >> I would have agreed with you if the current code was doing a > >> >> >> >> better > >> >> >> >> decision of what is local and what not. But it is not, it > simply > >> >> >> >> remove all > >> >> >> >> 127.x.x.x interfaces (opal/util/net.c:222). Thus, the only > thing > >> >> >> >> the > >> >> >> >> current > >> >> >> >> code does, is preventing a power-user from using the loopback > >> >> >> >> (despite being > >> >> >> >> explicitly enabled via the corresponding MCA parameters). > >> >> >> > > >> >> >> >Fair enough. > >> >> >> > > >> >> >> >Should we have a keyword that can be used in the > >> >> >> > btl_tcp_if_include/exclude (e.g., "local") that removes all > >> >> >> > local-only > >> >> >> > interfaces? I.E., all 127.x.x.x/8 interfaces *and* all > local-only > >> >> >> > interfaces (e.g., bridging interfaces to local VMs and the > like)? > >> >> >> > > >> >> >> >We could then replace the default "127.0.0.0/8" value in > >> >> >> > btl_tcp_if_exclude with this token, and therefore actually > exclude > >> >> >> > the > >> >> >> > VM-only interfaces (which have caused some users problems in the > >> >> >> > past). > >> >> >> > > >> >> >> >-- > >> >> >> >Jeff Squyres > >> >> >> >jsquy...@cisco.com > >> >> >> >For corporate legal information go to: > >> >> >> > http://www.cisco.com/web/about/doing_business/legal/cri/ > >> >> >> > > >> >> >> >_______________________________________________ > >> >> >> >devel mailing list > >> >> >> >devel@lists.open-mpi.org > >> >> >> >https://rfd.newmexicoconsortium.org/mailman/listinfo/devel > >> >> >> _______________________________________________ > >> >> >> devel mailing list > >> >> >> devel@lists.open-mpi.org > >> >> >> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel > >> >> > > >> >> > > >> >> > _______________________________________________ > >> >> > devel mailing list > >> >> > devel@lists.open-mpi.org > >> >> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel > >> >> _______________________________________________ > >> >> devel mailing list > >> >> devel@lists.open-mpi.org > >> >> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel > >> > > >> > > >> > > >> > _______________________________________________ > >> > devel mailing list > >> > devel@lists.open-mpi.org > >> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel > >> _______________________________________________ > >> devel mailing list > >> devel@lists.open-mpi.org > >> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel > > > > > > _______________________________________________ > > devel mailing list > > devel@lists.open-mpi.org > > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel > _______________________________________________ > devel mailing list > devel@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >
_______________________________________________ devel mailing list devel@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
