[OMPI devel] Another Open MPI <-> PSM question (MPI_Isend()/MPI_Cancel())
Doing MPI_Isend() followed by a MPI_Cancel() fails on my PSM based system with 1.8.4 like this: n040108:0.1.Cannot cancel send requests (req=0x2b6279787f80) n040108:0.0.Cannot cancel send requests (req=0x2b3a3dc92f80) --- Primary job terminated normally, but 1 process returned a non-zero exit code.. Per user-direction, the job has been aborted. --- -- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[58364,1],1] Exit code:255 -- Is this something PSM actually cannot do or an Open MPI error? Adrian
Re: [OMPI devel] Another Open MPI <-> PSM question (MPI_Isend()/MPI_Cancel())
>From the MPI standard perspective MPI_Cancel doesn't have to succeed, it can also gracefully fail. However, the PSM MTL diverges from the MPI standard and if a request cannot be canceled an error is returned. Here is a patch to fix this issue. diff --git a/ompi/mca/mtl/psm/mtl_psm_cancel.c b/ompi/mca/mtl/psm/mtl_psm_cancel index 6da3386..277c761 100644 --- a/ompi/mca/mtl/psm/mtl_psm_cancel.c +++ b/ompi/mca/mtl/psm/mtl_psm_cancel.c @@ -37,10 +37,8 @@ int ompi_mtl_psm_cancel(struct mca_mtl_base_module_t* mtl, if(PSM_OK == err) { mtl_request->ompi_req->req_status._cancelled = true; mtl_psm_request->super.completion_callback(&mtl_psm_request->super); - return OMPI_SUCCESS; -} else { - return OMPI_ERROR; } +return OMPI_SUCCESS; } else if(PSM_MQ_INCOMPLETE == err) { return OMPI_SUCCESS; } George. On Thu, Jan 15, 2015 at 1:30 PM, Adrian Reber wrote: > Doing > > MPI_Isend() > > followed by a > > MPI_Cancel() > > fails on my PSM based system with 1.8.4 like this: > > n040108:0.1.Cannot cancel send requests (req=0x2b6279787f80) > n040108:0.0.Cannot cancel send requests (req=0x2b3a3dc92f80) > --- > Primary job terminated normally, but 1 process returned > a non-zero exit code.. Per user-direction, the job has been aborted. > --- > -- > mpirun detected that one or more processes exited with non-zero status, > thus causing > the job to be terminated. The first process to do so was: > > Process name: [[58364,1],1] > Exit code:255 > -- > > Is this something PSM actually cannot do or an Open MPI error? > > Adrian > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/01/16783.php >
Re: [OMPI devel] Another Open MPI <-> PSM question (MPI_Isend()/MPI_Cancel())
thanks George! 2015-01-15 11:43 GMT-07:00 George Bosilca : > From the MPI standard perspective MPI_Cancel doesn't have to succeed, it > can also gracefully fail. However, the PSM MTL diverges from the MPI > standard and if a request cannot be canceled an error is returned. Here is > a patch to fix this issue. > > diff --git a/ompi/mca/mtl/psm/mtl_psm_cancel.c > b/ompi/mca/mtl/psm/mtl_psm_cancel > index 6da3386..277c761 100644 > --- a/ompi/mca/mtl/psm/mtl_psm_cancel.c > +++ b/ompi/mca/mtl/psm/mtl_psm_cancel.c > @@ -37,10 +37,8 @@ int ompi_mtl_psm_cancel(struct mca_mtl_base_module_t* > mtl, > if(PSM_OK == err) { >mtl_request->ompi_req->req_status._cancelled = true; >mtl_psm_request->super.completion_callback(&mtl_psm_request->super); > - return OMPI_SUCCESS; > -} else { > - return OMPI_ERROR; > } > +return OMPI_SUCCESS; >} else if(PSM_MQ_INCOMPLETE == err) { > return OMPI_SUCCESS; >} > > George. > > > On Thu, Jan 15, 2015 at 1:30 PM, Adrian Reber wrote: > >> Doing >> >> MPI_Isend() >> >> followed by a >> >> MPI_Cancel() >> >> fails on my PSM based system with 1.8.4 like this: >> >> n040108:0.1.Cannot cancel send requests (req=0x2b6279787f80) >> n040108:0.0.Cannot cancel send requests (req=0x2b3a3dc92f80) >> --- >> Primary job terminated normally, but 1 process returned >> a non-zero exit code.. Per user-direction, the job has been aborted. >> --- >> -- >> mpirun detected that one or more processes exited with non-zero status, >> thus causing >> the job to be terminated. The first process to do so was: >> >> Process name: [[58364,1],1] >> Exit code:255 >> -- >> >> Is this something PSM actually cannot do or an Open MPI error? >> >> Adrian >> ___ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2015/01/16783.php >> > > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/01/16784.php >
Re: [OMPI devel] Another Open MPI <-> PSM question (MPI_Isend()/MPI_Cancel())
As PSM on master is still broken I applied it on 1.8.4. Unfortunately it does not work. The error is the same as before. Looking at your patch I would also expect that this is the correct fix and I even tried to change ompi_mtl_psm_cancel() to always return OMPI_SUCCESS. MPI_Cancel() still fails. Looking at the PSM code it seems it can directly call exit(-1) and thus terminating and never returning to Open MPI. I do not see any debug output from Open MPI after "Cannot cancel send requests" from PSM. Adrian On Thu, Jan 15, 2015 at 01:43:11PM -0500, George Bosilca wrote: > >From the MPI standard perspective MPI_Cancel doesn't have to succeed, it > can also gracefully fail. However, the PSM MTL diverges from the MPI > standard and if a request cannot be canceled an error is returned. Here is > a patch to fix this issue. > > diff --git a/ompi/mca/mtl/psm/mtl_psm_cancel.c > b/ompi/mca/mtl/psm/mtl_psm_cancel > index 6da3386..277c761 100644 > --- a/ompi/mca/mtl/psm/mtl_psm_cancel.c > +++ b/ompi/mca/mtl/psm/mtl_psm_cancel.c > @@ -37,10 +37,8 @@ int ompi_mtl_psm_cancel(struct mca_mtl_base_module_t* > mtl, > if(PSM_OK == err) { >mtl_request->ompi_req->req_status._cancelled = true; >mtl_psm_request->super.completion_callback(&mtl_psm_request->super); > - return OMPI_SUCCESS; > -} else { > - return OMPI_ERROR; > } > +return OMPI_SUCCESS; >} else if(PSM_MQ_INCOMPLETE == err) { > return OMPI_SUCCESS; >} > > George. > > > On Thu, Jan 15, 2015 at 1:30 PM, Adrian Reber wrote: > > > Doing > > > > MPI_Isend() > > > > followed by a > > > > MPI_Cancel() > > > > fails on my PSM based system with 1.8.4 like this: > > > > n040108:0.1.Cannot cancel send requests (req=0x2b6279787f80) > > n040108:0.0.Cannot cancel send requests (req=0x2b3a3dc92f80) > > --- > > Primary job terminated normally, but 1 process returned > > a non-zero exit code.. Per user-direction, the job has been aborted. > > --- > > -- > > mpirun detected that one or more processes exited with non-zero status, > > thus causing > > the job to be terminated. The first process to do so was: > > > > Process name: [[58364,1],1] > > Exit code:255 > > -- > > > > Is this something PSM actually cannot do or an Open MPI error? > > > > Adrian > > ___ > > devel mailing list > > de...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > > http://www.open-mpi.org/community/lists/devel/2015/01/16783.php > > > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/01/16784.php
Re: [OMPI devel] Another Open MPI <-> PSM question (MPI_Isend()/MPI_Cancel())
It even says so in the code: ompi/mca/mtl/psm/mtl_psm.c: /* Default error handling is enabled, errors will not be returned to * user. PSM prints the error and the offending endpoint's hostname * and exits with -1 */ Disabling the default PSM error handler makes MPI_Cancel() fail gracefully. But then no error is handled anymore. Adrian On Thu, Jan 15, 2015 at 10:21:05PM +0100, Adrian Reber wrote: > As PSM on master is still broken I applied it on 1.8.4. Unfortunately it > does not work. The error is the same as before. > > Looking at your patch I would also expect that this is the correct fix > and I even tried to change ompi_mtl_psm_cancel() to always return > OMPI_SUCCESS. MPI_Cancel() still fails. > > Looking at the PSM code it seems it can directly call exit(-1) and thus > terminating and never returning to Open MPI. I do not see any debug > output from Open MPI after "Cannot cancel send requests" from PSM. > > Adrian > > On Thu, Jan 15, 2015 at 01:43:11PM -0500, George Bosilca wrote: > > >From the MPI standard perspective MPI_Cancel doesn't have to succeed, it > > can also gracefully fail. However, the PSM MTL diverges from the MPI > > standard and if a request cannot be canceled an error is returned. Here is > > a patch to fix this issue. > > > > diff --git a/ompi/mca/mtl/psm/mtl_psm_cancel.c > > b/ompi/mca/mtl/psm/mtl_psm_cancel > > index 6da3386..277c761 100644 > > --- a/ompi/mca/mtl/psm/mtl_psm_cancel.c > > +++ b/ompi/mca/mtl/psm/mtl_psm_cancel.c > > @@ -37,10 +37,8 @@ int ompi_mtl_psm_cancel(struct mca_mtl_base_module_t* > > mtl, > > if(PSM_OK == err) { > >mtl_request->ompi_req->req_status._cancelled = true; > >mtl_psm_request->super.completion_callback(&mtl_psm_request->super); > > - return OMPI_SUCCESS; > > -} else { > > - return OMPI_ERROR; > > } > > +return OMPI_SUCCESS; > >} else if(PSM_MQ_INCOMPLETE == err) { > > return OMPI_SUCCESS; > >} > > > > George. > > > > > > On Thu, Jan 15, 2015 at 1:30 PM, Adrian Reber wrote: > > > > > Doing > > > > > > MPI_Isend() > > > > > > followed by a > > > > > > MPI_Cancel() > > > > > > fails on my PSM based system with 1.8.4 like this: > > > > > > n040108:0.1.Cannot cancel send requests (req=0x2b6279787f80) > > > n040108:0.0.Cannot cancel send requests (req=0x2b3a3dc92f80) > > > --- > > > Primary job terminated normally, but 1 process returned > > > a non-zero exit code.. Per user-direction, the job has been aborted. > > > --- > > > -- > > > mpirun detected that one or more processes exited with non-zero status, > > > thus causing > > > the job to be terminated. The first process to do so was: > > > > > > Process name: [[58364,1],1] > > > Exit code:255 > > > -- > > > > > > Is this something PSM actually cannot do or an Open MPI error? > > > > > > Adrian > > > ___ > > > devel mailing list > > > de...@open-mpi.org > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > Link to this post: > > > http://www.open-mpi.org/community/lists/devel/2015/01/16783.php > > > > > > ___ > > devel mailing list > > de...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > > http://www.open-mpi.org/community/lists/devel/2015/01/16784.php > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/01/16786.php Adrian -- Adrian Reber http://lisas.de/~adrian/ C-3PO: Don't call me a mindless philosopher, you overweight glob of grease!
Re: [OMPI devel] Another Open MPI <-> PSM question (MPI_Isend()/MPI_Cancel())
Skimming through the PSM code shows that the return values of the PSM functions are handled in most cases. Thus, removing the default error handler might not be such a bad idea. Did you experience any trouble running with the version without the default error handler registered? George. On Thu, Jan 15, 2015 at 4:40 PM, Adrian Reber wrote: > It even says so in the code: > > ompi/mca/mtl/psm/mtl_psm.c: > >/* Default error handling is enabled, errors will not be returned to > * user. PSM prints the error and the offending endpoint's > hostname > * and exits with -1 */ > > Disabling the default PSM error handler makes MPI_Cancel() fail > gracefully. But then no error is handled anymore. > > Adrian > > On Thu, Jan 15, 2015 at 10:21:05PM +0100, Adrian Reber wrote: > > As PSM on master is still broken I applied it on 1.8.4. Unfortunately it > > does not work. The error is the same as before. > > > > Looking at your patch I would also expect that this is the correct fix > > and I even tried to change ompi_mtl_psm_cancel() to always return > > OMPI_SUCCESS. MPI_Cancel() still fails. > > > > Looking at the PSM code it seems it can directly call exit(-1) and thus > > terminating and never returning to Open MPI. I do not see any debug > > output from Open MPI after "Cannot cancel send requests" from PSM. > > > > Adrian > > > > On Thu, Jan 15, 2015 at 01:43:11PM -0500, George Bosilca wrote: > > > >From the MPI standard perspective MPI_Cancel doesn't have to succeed, > it > > > can also gracefully fail. However, the PSM MTL diverges from the MPI > > > standard and if a request cannot be canceled an error is returned. > Here is > > > a patch to fix this issue. > > > > > > diff --git a/ompi/mca/mtl/psm/mtl_psm_cancel.c > > > b/ompi/mca/mtl/psm/mtl_psm_cancel > > > index 6da3386..277c761 100644 > > > --- a/ompi/mca/mtl/psm/mtl_psm_cancel.c > > > +++ b/ompi/mca/mtl/psm/mtl_psm_cancel.c > > > @@ -37,10 +37,8 @@ int ompi_mtl_psm_cancel(struct > mca_mtl_base_module_t* > > > mtl, > > > if(PSM_OK == err) { > > >mtl_request->ompi_req->req_status._cancelled = true; > > > > mtl_psm_request->super.completion_callback(&mtl_psm_request->super); > > > - return OMPI_SUCCESS; > > > -} else { > > > - return OMPI_ERROR; > > > } > > > +return OMPI_SUCCESS; > > >} else if(PSM_MQ_INCOMPLETE == err) { > > > return OMPI_SUCCESS; > > >} > > > > > > George. > > > > > > > > > On Thu, Jan 15, 2015 at 1:30 PM, Adrian Reber wrote: > > > > > > > Doing > > > > > > > > MPI_Isend() > > > > > > > > followed by a > > > > > > > > MPI_Cancel() > > > > > > > > fails on my PSM based system with 1.8.4 like this: > > > > > > > > n040108:0.1.Cannot cancel send requests (req=0x2b6279787f80) > > > > n040108:0.0.Cannot cancel send requests (req=0x2b3a3dc92f80) > > > > --- > > > > Primary job terminated normally, but 1 process returned > > > > a non-zero exit code.. Per user-direction, the job has been aborted. > > > > --- > > > > > -- > > > > mpirun detected that one or more processes exited with non-zero > status, > > > > thus causing > > > > the job to be terminated. The first process to do so was: > > > > > > > > Process name: [[58364,1],1] > > > > Exit code:255 > > > > > -- > > > > > > > > Is this something PSM actually cannot do or an Open MPI error? > > > > > > > > Adrian > > > > ___ > > > > devel mailing list > > > > de...@open-mpi.org > > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > > Link to this post: > > > > http://www.open-mpi.org/community/lists/devel/2015/01/16783.php > > > > > > > > > ___ > > > devel mailing list > > > de...@open-mpi.org > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/01/16784.php > > ___ > > devel mailing list > > de...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/01/16786.php > > Adrian > > -- > Adrian Reber http://lisas.de/~adrian/ > C-3PO: > Don't call me a mindless philosopher, you overweight > glob of grease! > ___ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/01/16787.php >
Re: [OMPI devel] Another Open MPI <-> PSM question (MPI_Isend()/MPI_Cancel())
See my comment on https://github.com/open-mpi/ompi/issues/347 On Thu, Jan 15, 2015 at 05:01:00PM -0500, George Bosilca wrote: > Skimming through the PSM code shows that the return values of the PSM > functions are handled in most cases. Thus, removing the default error > handler might not be such a bad idea. > > Did you experience any trouble running with the version without the default > error handler registered? > > George. > > > On Thu, Jan 15, 2015 at 4:40 PM, Adrian Reber wrote: > > > It even says so in the code: > > > > ompi/mca/mtl/psm/mtl_psm.c: > > > >/* Default error handling is enabled, errors will not be returned to > > * user. PSM prints the error and the offending endpoint's > > hostname > > * and exits with -1 */ > > > > Disabling the default PSM error handler makes MPI_Cancel() fail > > gracefully. But then no error is handled anymore. > > > > Adrian > > > > On Thu, Jan 15, 2015 at 10:21:05PM +0100, Adrian Reber wrote: > > > As PSM on master is still broken I applied it on 1.8.4. Unfortunately it > > > does not work. The error is the same as before. > > > > > > Looking at your patch I would also expect that this is the correct fix > > > and I even tried to change ompi_mtl_psm_cancel() to always return > > > OMPI_SUCCESS. MPI_Cancel() still fails. > > > > > > Looking at the PSM code it seems it can directly call exit(-1) and thus > > > terminating and never returning to Open MPI. I do not see any debug > > > output from Open MPI after "Cannot cancel send requests" from PSM. > > > > > > Adrian > > > > > > On Thu, Jan 15, 2015 at 01:43:11PM -0500, George Bosilca wrote: > > > > >From the MPI standard perspective MPI_Cancel doesn't have to succeed, > > it > > > > can also gracefully fail. However, the PSM MTL diverges from the MPI > > > > standard and if a request cannot be canceled an error is returned. > > Here is > > > > a patch to fix this issue. > > > > > > > > diff --git a/ompi/mca/mtl/psm/mtl_psm_cancel.c > > > > b/ompi/mca/mtl/psm/mtl_psm_cancel > > > > index 6da3386..277c761 100644 > > > > --- a/ompi/mca/mtl/psm/mtl_psm_cancel.c > > > > +++ b/ompi/mca/mtl/psm/mtl_psm_cancel.c > > > > @@ -37,10 +37,8 @@ int ompi_mtl_psm_cancel(struct > > mca_mtl_base_module_t* > > > > mtl, > > > > if(PSM_OK == err) { > > > >mtl_request->ompi_req->req_status._cancelled = true; > > > > > > mtl_psm_request->super.completion_callback(&mtl_psm_request->super); > > > > - return OMPI_SUCCESS; > > > > -} else { > > > > - return OMPI_ERROR; > > > > } > > > > +return OMPI_SUCCESS; > > > >} else if(PSM_MQ_INCOMPLETE == err) { > > > > return OMPI_SUCCESS; > > > >} > > > > > > > > George. > > > > > > > > > > > > On Thu, Jan 15, 2015 at 1:30 PM, Adrian Reber wrote: > > > > > > > > > Doing > > > > > > > > > > MPI_Isend() > > > > > > > > > > followed by a > > > > > > > > > > MPI_Cancel() > > > > > > > > > > fails on my PSM based system with 1.8.4 like this: > > > > > > > > > > n040108:0.1.Cannot cancel send requests (req=0x2b6279787f80) > > > > > n040108:0.0.Cannot cancel send requests (req=0x2b3a3dc92f80) > > > > > --- > > > > > Primary job terminated normally, but 1 process returned > > > > > a non-zero exit code.. Per user-direction, the job has been aborted. > > > > > --- > > > > > > > -- > > > > > mpirun detected that one or more processes exited with non-zero > > status, > > > > > thus causing > > > > > the job to be terminated. The first process to do so was: > > > > > > > > > > Process name: [[58364,1],1] > > > > > Exit code:255 > > > > > > > -- > > > > > > > > > > Is this something PSM actually cannot do or an Open MPI error? > > > > > > > > > > Adrian > > > > > ___ > > > > > devel mailing list > > > > > de...@open-mpi.org > > > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > > > Link to this post: > > > > > http://www.open-mpi.org/community/lists/devel/2015/01/16783.php > > > > > > > > > > > > ___ > > > > devel mailing list > > > > de...@open-mpi.org > > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > > Link to this post: > > http://www.open-mpi.org/community/lists/devel/2015/01/16784.php > > > ___ > > > devel mailing list > > > de...@open-mpi.org > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > Link to this post: > > http://www.open-mpi.org/community/lists/devel/2015/01/16786.php