I don't think what the openib BTL is doing is that bad. It is returning an
error because something really went bad in IB. So yes, it could blank the
bitmask and return success, but would you really want IB to fail and
fallback on TCP once in a while without any notice ? I wouldn't.
So, as it seems that all "normal" problems can be handled through the
reachable bitmask, it seems a good idea to me that BTLs returning errors
make the application stop.
Sylvain
On Wed, 26 May 2010, Barrett, Brian W wrote:
George -
I'm not sure I agree - the return code should indicate a failure beyond
"something prohibited me from talking to the remote side" - something
occurred that resulted in it being highly unlikely the app can
successfully run to completion (such as malloc failing). On the other
hand, I also think that the OpenIB BTL is probably doing the wrong thing
- I can't imagine that the error returned reaches that state of badness,
and it should probably zero out the bitmask and quietly return rather
than try to cause the app to abort.
Just my $0.02.
Brian
On May 25, 2010, at 12:27 PM, George Bosilca wrote:
The BTLs are allowed to fail adding procs without major consequences in
the short term. As you noticed each BTL returns a bit mask array
containing all procs reachable through this particular instance of the
BTL. Later (in the same file line 395) we check for the complete
coverage for all procs, and only complain if one of the peers is
unreachable.
If you replace the continue statement by a return, we will never give a
chance to the other BTLs and we will complain about lack of
connectivity as soon as one BTL fails (for some reasons). Without
talking about the fact that all the eager, send and rmda endpoint
arrays will not be built.
george.
On May 25, 2010, at 05:10 , Sylvain Jeaugey wrote:
Hi,
I'm currently trying to have Open MPI exit more gracefully when a BTL returns an error
during the "add procs" phase.
The current bml/r2 code silently ignores btl->add_procs() error codes with the
following comment :
---- ompi/mca/bml/r2/bml_r2.c:208 ----
/* This BTL has troubles adding the nodes. Let's continue maybe some other BTL
* can take care of this task. */
continue;
--------------------------------------
This seems wrong to me : either a proc is reached (the "reachable" bit field is
therefore updated), either it is not (and nothing is done). Any error code should denote
a fatal error needing a clean abort.
In the current openib btl code, the "reachable" bit is set but an error is
returned - then ignored by r2. The next call to the openib BTL results in a segmentation
fault.
So, maybe this simple fix would do the trick :
========================================================================
diff -r 96e0793d7885 ompi/mca/bml/r2/bml_r2.c
--- a/ompi/mca/bml/r2/bml_r2.c Wed May 19 14:35:27 2010 +0200
+++ b/ompi/mca/bml/r2/bml_r2.c Tue May 25 10:54:19 2010 +0200
@@ -210,7 +210,7 @@
/* This BTL has troubles adding the nodes. Let's continue maybe some
other BTL
* can take care of this task.
*/
- continue;
+ return rc;
}
/* for each proc that is reachable */
========================================================================
Does anyone see a case (with a specific btl) where add_procs returns an error
but we still want to continue ?
Sylvain
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
Brian W. Barrett
Dept. 1423: Scalable System Software
Sandia National Laboratories
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
