[OMPI devel] MPI_Abort does not make mpirun return with the right exit code
Folks, let's look at the following trivial test program : #include #include int main (int argc, char * argv[]) { int rank, size; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_WORLD, &rank); printf ("I am %d/%d and i abort\n", rank, size); MPI_Abort(MPI_COMM_WORLD, 2); printf ("%d/%d aborted !\n", rank, size); return 3; } and let's run mpirun (trunk) on node0 and ask the mpi task to run on task 1 : with two tasks or more : node0 $ mpirun --mca btl tcp,self -host node1 -np 2 ./abort -- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 2. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -- I am 1/2 and i abort I am 0/2 and i abort [node0:00740] 1 more process has sent help message help-mpi-api.txt / mpi-abort [node0:00740] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages node0 $ echo $? 0 the exit status of mpirun is zero /* this is why the MPI_Errhandler_fatal_c test fails in mtt */ now if we run only one task : node0 $ mpirun --mca btl tcp,self -host node1 -np 1 ./abort I am 0/1 and i abort -- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 2. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -- -- mpirun has exited due to process rank 0 with PID 15884 on node node1 exiting improperly. There are three reasons this could occur: 1. this process did not call "init" before exiting, but others in the job did. This can cause a job to hang indefinitely while it waits for all processes to call "init". By rule, if one process calls "init", then ALL processes must call "init" prior to termination. 2. this process called "init", but exited without calling "finalize". By rule, all processes that call "init" MUST call "finalize" prior to exiting or it will be considered an "abnormal termination" 3. this process called "MPI_Abort" or "orte_abort" and the mca parameter orte_create_session_dirs is set to false. In this case, the run-time cannot detect that the abort call was an abnormal termination. Hence, the only error message you will receive is this one. This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). You can avoid this message by specifying -quiet on the mpirun command line. -- node0 $ echo $? 1 the program displayed a misleading error message and mpirun exited with error code 1 /* i would have expected 2, or 3 in the worst case scenario */ i digged it a bit and found a kind of race condition in orted (running on node 1) basically, when the process dies, it writes stuff in the openmpi session directory and exits. exiting send a SIGCHLD to orted and close the socket/pipe connected to orted. on orted, the loss of connection is generally processed before the SIGCHLD by libevent, and as a consequence, the exit code is not correctly set (e.g. it is left to zero). i did not see any kind of communication between the mpi task and orted (except writing a file in the openmpi session directory) as i would have expected /* but this was just my initial guess, the truth is i do not know what is supposed to happen */ i wrote the attached abort.patch patch to basically get it working. i highly suspect this is not the right thing to do so i did not commit it. it works fine with two tasks or more. with only one task, mpirun display a misleading error message but the exit status is ok. could someone (Ralph ?) have a look at this ? Cheers, Gilles node0 $ mpirun --mca btl tcp,self -host node1 -np 2 ./abort I am 1/2 and i abort I am 0/2 and i abort -- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 2. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -- [node0:00920] 1 more process has sent help message help-mpi-api.txt / mpi-abort [node0:00920] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages node0 $ echo $? 2 node0 $ mpirun --mca btl tcp,self -host node1 -np 1 ./abort I am 0/1 and i abort --
Re: [OMPI devel] MPI_Abort does not make mpirun return with the right exit code
I'm aware of the problem, but it will be fixed when the PMIx branch is merged later this week. On Aug 19, 2014, at 10:00 PM, Gilles Gouaillardet wrote: > Folks, > > let's look at the following trivial test program : > > #include > #include > > int main (int argc, char * argv[]) { >int rank, size; >MPI_Init(&argc, &argv); >MPI_Comm_size(MPI_COMM_WORLD, &size); >MPI_Comm_rank(MPI_COMM_WORLD, &rank); >printf ("I am %d/%d and i abort\n", rank, size); >MPI_Abort(MPI_COMM_WORLD, 2); >printf ("%d/%d aborted !\n", rank, size); >return 3; > } > > and let's run mpirun (trunk) on node0 and ask the mpi task to run on > task 1 : > with two tasks or more : > > node0 $ mpirun --mca btl tcp,self -host node1 -np 2 ./abort > -- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 2. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -- > I am 1/2 and i abort > I am 0/2 and i abort > [node0:00740] 1 more process has sent help message help-mpi-api.txt / > mpi-abort > [node0:00740] Set MCA parameter "orte_base_help_aggregate" to 0 to see > all help / error messages > > node0 $ echo $? > 0 > > the exit status of mpirun is zero > /* this is why the MPI_Errhandler_fatal_c test fails in mtt */ > > now if we run only one task : > > node0 $ mpirun --mca btl tcp,self -host node1 -np 1 ./abort > I am 0/1 and i abort > -- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 2. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -- > -- > mpirun has exited due to process rank 0 with PID 15884 on > node node1 exiting improperly. There are three reasons this could occur: > > 1. this process did not call "init" before exiting, but others in > the job did. This can cause a job to hang indefinitely while it waits > for all processes to call "init". By rule, if one process calls "init", > then ALL processes must call "init" prior to termination. > > 2. this process called "init", but exited without calling "finalize". > By rule, all processes that call "init" MUST call "finalize" prior to > exiting or it will be considered an "abnormal termination" > > 3. this process called "MPI_Abort" or "orte_abort" and the mca parameter > orte_create_session_dirs is set to false. In this case, the run-time cannot > detect that the abort call was an abnormal termination. Hence, the only > error message you will receive is this one. > > This may have caused other processes in the application to be > terminated by signals sent by mpirun (as reported here). > > You can avoid this message by specifying -quiet on the mpirun command line. > > -- > node0 $ echo $? > 1 > > the program displayed a misleading error message and mpirun exited with > error code 1 > /* i would have expected 2, or 3 in the worst case scenario */ > > > i digged it a bit and found a kind of race condition in orted (running > on node 1) > basically, when the process dies, it writes stuff in the openmpi session > directory and exits. > exiting send a SIGCHLD to orted and close the socket/pipe connected to > orted. > on orted, the loss of connection is generally processed before the > SIGCHLD by libevent, > and as a consequence, the exit code is not correctly set (e.g. it is > left to zero). > i did not see any kind of communication between the mpi task and orted > (except writing a file in the openmpi session directory) as i would have > expected > /* but this was just my initial guess, the truth is i do not know what > is supposed to happen */ > > i wrote the attached abort.patch patch to basically get it working. > i highly suspect this is not the right thing to do so i did not commit it. > > it works fine with two tasks or more. > with only one task, mpirun display a misleading error message but the > exit status is ok. > > could someone (Ralph ?) have a look at this ? > > Cheers, > > Gilles > > > node0 $ mpirun --mca btl tcp,self -host node1 -np 2 ./abort > I am 1/2 and i abort > I am 0/2 and i abort > -- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 2. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on
Re: [OMPI devel] MPI_Abort does not make mpirun return with the right exit code
Ralph, i tried again after the merge and found the same behaviour, though the internals are very different. i run without any batch manager from node0: mpirun -np 1 --mca btl tcp,self -host node1 ./abort exit with exit code zero :-( short story : i applied pmix.2.patch and that fixed my problem could you please review this ? long story : i initially applied pmix.1.patch and it solved my problem then i ran mpirun -np 1 --mca btl openib,self -host node1 ./abort and i came back to square one : exit code is zero so i used the debugger and was unable to reproduce the issue (one more race condition, yeah !) finally, i wrote pmix.2.patch, fixed my issue and realized that pmix.1.patch was no more needed. currently, and assuming pmix.2.patch is correct, i cannot tell wether pmix.1.patch is needed or not since this part of the code is no more executed. i also found one hang with the following trivial program within one node : int main (int argc, char *argv[]) { MPI_Init(&argc, &argv); MPI_Finalize(); return 3; } from node0 : $ mpirun -np 1 ./test --- Primary job terminated normally, but 1 process returned a non-zero exit code.. Per user-direction, the job has been aborted. --- AND THE PROGRAM HANGS *but* $ mpirun -np 1 -host node1 ./test --- Primary job terminated normally, but 1 process returned a non-zero exit code.. Per user-direction, the job has been aborted. --- -- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[22080,1],0] Exit code:3 -- return with exit code 3. then i found a strange behaviour with helloworld if only the self btl is used : $ mpirun -np 1 --mca btl self ./hw [helios91:23319] OPAL dss:unpack: got type 12 when expecting type 3 [helios91:23319] [[22303,0],0] ORTE_ERROR_LOG: Pack data mismatch in file ../../../src/ompi-trunk/orte/orted/pmix/pmix_server_sendrecv.c at line 722 the program returns with exit code zero, but display an error message. Cheers, Gilles On 2014/08/21 6:21, Ralph Castain wrote: > I'm aware of the problem, but it will be fixed when the PMIx branch is merged > later this week. > > On Aug 19, 2014, at 10:00 PM, Gilles Gouaillardet > wrote: > >> Folks, >> >> let's look at the following trivial test program : >> >> #include >> #include >> >> int main (int argc, char * argv[]) { >>int rank, size; >>MPI_Init(&argc, &argv); >>MPI_Comm_size(MPI_COMM_WORLD, &size); >>MPI_Comm_rank(MPI_COMM_WORLD, &rank); >>printf ("I am %d/%d and i abort\n", rank, size); >>MPI_Abort(MPI_COMM_WORLD, 2); >>printf ("%d/%d aborted !\n", rank, size); >>return 3; >> } >> >> and let's run mpirun (trunk) on node0 and ask the mpi task to run on >> task 1 : >> with two tasks or more : >> >> node0 $ mpirun --mca btl tcp,self -host node1 -np 2 ./abort >> -- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >> with errorcode 2. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> -- >> I am 1/2 and i abort >> I am 0/2 and i abort >> [node0:00740] 1 more process has sent help message help-mpi-api.txt / >> mpi-abort >> [node0:00740] Set MCA parameter "orte_base_help_aggregate" to 0 to see >> all help / error messages >> >> node0 $ echo $? >> 0 >> >> the exit status of mpirun is zero >> /* this is why the MPI_Errhandler_fatal_c test fails in mtt */ >> >> now if we run only one task : >> >> node0 $ mpirun --mca btl tcp,self -host node1 -np 1 ./abort >> I am 0/1 and i abort >> -- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >> with errorcode 2. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> -- >> -- >> mpirun has exited due to process rank 0 with PID 15884 on >> node node1 exiting improperly. There are three reasons this could occur: >> >> 1. this process did not call "init" before exiting, but others in >> the job did. This can cause a job to hang indefinitely while it waits >> for all processes to call "init". By rule, if one process call
Re: [OMPI devel] MPI_Abort does not make mpirun return with the right exit code
You might want to try again with current head of trunk as something seems off in what you are seeing - more below On Aug 22, 2014, at 3:12 AM, Gilles Gouaillardet wrote: > Ralph, > > i tried again after the merge and found the same behaviour, though the > internals are very different. > > i run without any batch manager > > from node0: > mpirun -np 1 --mca btl tcp,self -host node1 ./abort > > exit with exit code zero :-( Hmmm...it works fine for me, without your patch: 07:35:41 $ mpirun -n 1 -mca btl tcp,self -host bend002 ./abort Hello, World, I am 0 of 1 -- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 2. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -- -- mpirun noticed that process rank 0 with PID 24382 on node bend002 exited on signal 0 (Unknown signal 0). -- 07:35:56 $ showcode 130 > > short story : i applied pmix.2.patch and that fixed my problem > could you please review this ? > > long story : > i initially applied pmix.1.patch and it solved my problem > then i ran > mpirun -np 1 --mca btl openib,self -host node1 ./abort > and i came back to square one : exit code is zero > so i used the debugger and was unable to reproduce the issue > (one more race condition, yeah !) > finally, i wrote pmix.2.patch, fixed my issue and realized that > pmix.1.patch was no more needed. > currently, and assuming pmix.2.patch is correct, i cannot tell wether > pmix.1.patch is needed or not > since this part of the code is no more executed. > > i also found one hang with the following trivial program within one node : > > int main (int argc, char *argv[]) { > MPI_Init(&argc, &argv); >MPI_Finalize(); >return 3; > } > > from node0 : > $ mpirun -np 1 ./test > --- > Primary job terminated normally, but 1 process returned > a non-zero exit code.. Per user-direction, the job has been aborted. > --- > > AND THE PROGRAM HANGS This also works fine for me: 07:37:27 $ mpirun -n 1 ./mpi_no_op 07:37:36 $ cat mpi_no_op.c /* -*- C -*- * * $HEADER$ * * The most basic of MPI applications */ #include #include "mpi.h" int main(int argc, char* argv[]) { MPI_Init(&argc, &argv); MPI_Finalize(); return 0; } > > *but* > $ mpirun -np 1 -host node1 ./test > --- > Primary job terminated normally, but 1 process returned > a non-zero exit code.. Per user-direction, the job has been aborted. > --- > -- > mpirun detected that one or more processes exited with non-zero status, > thus causing > the job to be terminated. The first process to do so was: > > Process name: [[22080,1],0] > Exit code:3 > -- > > return with exit code 3. Likewise here - works just fine for me > > then i found a strange behaviour with helloworld if only the self btl is > used : > $ mpirun -np 1 --mca btl self ./hw > [helios91:23319] OPAL dss:unpack: got type 12 when expecting type 3 > [helios91:23319] [[22303,0],0] ORTE_ERROR_LOG: Pack data mismatch in > file ../../../src/ompi-trunk/orte/orted/pmix/pmix_server_sendrecv.c at > line 722 > > the program returns with exit code zero, but display an error message. > > Cheers, > > Gilles > > On 2014/08/21 6:21, Ralph Castain wrote: >> I'm aware of the problem, but it will be fixed when the PMIx branch is >> merged later this week. >> >> On Aug 19, 2014, at 10:00 PM, Gilles Gouaillardet >> wrote: >> >>> Folks, >>> >>> let's look at the following trivial test program : >>> >>> #include >>> #include >>> >>> int main (int argc, char * argv[]) { >>> int rank, size; >>> MPI_Init(&argc, &argv); >>> MPI_Comm_size(MPI_COMM_WORLD, &size); >>> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >>> printf ("I am %d/%d and i abort\n", rank, size); >>> MPI_Abort(MPI_COMM_WORLD, 2); >>> printf ("%d/%d aborted !\n", rank, size); >>> return 3; >>> } >>> >>> and let's run mpirun (trunk) on node0 and ask the mpi task to run on >>> task 1 : >>> with two tasks or more : >>> >>> node0 $ mpirun --mca btl tcp,self -host node1 -np 2 ./abort >>> -- >>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>> with errorcode 2. >>> >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>
Re: [OMPI devel] OMPI devel] MPI_Abort does not make mpirun return with the right exit code
Ralph, Will do on Monday About the first test, in my case echo $? returns 0 I noticed this confusing message in your output : mpirun noticed that process rank 0 with PID 24382 on node bend002 exited on signal 0 (Unknown signal 0). About the second test, please note my test program return 3; whereas your mpi_no_op.c return 0; Cheers, Gilles Ralph Castain wrote: >You might want to try again with current head of trunk as something seems off >in what you are seeing - more below > > > >On Aug 22, 2014, at 3:12 AM, Gilles Gouaillardet > wrote: > > >Ralph, > >i tried again after the merge and found the same behaviour, though the >internals are very different. > >i run without any batch manager > >from node0: >mpirun -np 1 --mca btl tcp,self -host node1 ./abort > >exit with exit code zero :-( > > >Hmmm...it works fine for me, without your patch: > > >07:35:41 $ mpirun -n 1 -mca btl tcp,self -host bend002 ./abort > >Hello, World, I am 0 of 1 > >-- > >MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > >with errorcode 2. > > >NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > >You may or may not see output from other processes, depending on > >exactly when Open MPI kills them. > >-- > >-- > >mpirun noticed that process rank 0 with PID 24382 on node bend002 exited on >signal 0 (Unknown signal 0). > >-- > >07:35:56 $ showcode > >130 > > > >short story : i applied pmix.2.patch and that fixed my problem >could you please review this ? > >long story : >i initially applied pmix.1.patch and it solved my problem >then i ran >mpirun -np 1 --mca btl openib,self -host node1 ./abort >and i came back to square one : exit code is zero >so i used the debugger and was unable to reproduce the issue >(one more race condition, yeah !) >finally, i wrote pmix.2.patch, fixed my issue and realized that >pmix.1.patch was no more needed. >currently, and assuming pmix.2.patch is correct, i cannot tell wether >pmix.1.patch is needed or not >since this part of the code is no more executed. > >i also found one hang with the following trivial program within one node : > >int main (int argc, char *argv[]) { >MPI_Init(&argc, &argv); > MPI_Finalize(); > return 3; >} > >from node0 : >$ mpirun -np 1 ./test >--- >Primary job terminated normally, but 1 process returned >a non-zero exit code.. Per user-direction, the job has been aborted. >--- > >AND THE PROGRAM HANGS > > >This also works fine for me: > > >07:37:27 $ mpirun -n 1 ./mpi_no_op > >07:37:36 $ cat mpi_no_op.c > >/* -*- C -*- > > * > > * $HEADER$ > > * > > * The most basic of MPI applications > > */ > > >#include > >#include "mpi.h" > > >int main(int argc, char* argv[]) > >{ > > MPI_Init(&argc, &argv); > > > MPI_Finalize(); > > return 0; > >} > > > > >*but* >$ mpirun -np 1 -host node1 ./test >--- >Primary job terminated normally, but 1 process returned >a non-zero exit code.. Per user-direction, the job has been aborted. >--- >-- >mpirun detected that one or more processes exited with non-zero status, >thus causing >the job to be terminated. The first process to do so was: > > Process name: [[22080,1],0] > Exit code: 3 >-- > >return with exit code 3. > > >Likewise here - works just fine for me > > > > >then i found a strange behaviour with helloworld if only the self btl is >used : >$ mpirun -np 1 --mca btl self ./hw >[helios91:23319] OPAL dss:unpack: got type 12 when expecting type 3 >[helios91:23319] [[22303,0],0] ORTE_ERROR_LOG: Pack data mismatch in >file ../../../src/ompi-trunk/orte/orted/pmix/pmix_server_sendrecv.c at >line 722 > >the program returns with exit code zero, but display an error message. > >Cheers, > >Gilles > >On 2014/08/21 6:21, Ralph Castain wrote: > >I'm aware of the problem, but it will be fixed when the PMIx branch is merged >later this week. > >On Aug 19, 2014, at 10:00 PM, Gilles Gouaillardet > wrote: > >Folks, > >let's look at the following trivial test program : > >#include >#include > >int main (int argc, char * argv[]) { > int rank, size; > MPI_Init(&argc, &argv); > MPI_Comm_size(MPI_COMM_WORLD, &size); > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > printf ("I am %d/%d and i abort\n", rank, size); > MPI_Abort(MPI_COMM_WORLD, 2); > printf ("%d/%d aborted !\n", rank, size); > return 3; >} > >and let's run mpirun (trunk) on node0 and ask the mpi task to run on >
Re: [OMPI devel] OMPI devel] MPI_Abort does not make mpirun return with the right exit code
On Aug 22, 2014, at 9:06 AM, Gilles Gouaillardet wrote: > Ralph, > > Will do on Monday > > About the first test, in my case echo $? returns 0 My "showcode" is just an alias for the echo > I noticed this confusing message in your output : > mpirun noticed that process rank 0 with PID 24382 on node bend002 exited on > signal 0 (Unknown signal 0). I'll take a look at why that happened > > About the second test, please note my test program return 3; > whereas your mpi_no_op.c return 0; I didn't see that little cuteness - sigh > > Cheers, > > Gilles > > Ralph Castain wrote: > You might want to try again with current head of trunk as something seems off > in what you are seeing - more below > > > On Aug 22, 2014, at 3:12 AM, Gilles Gouaillardet > wrote: > >> Ralph, >> >> i tried again after the merge and found the same behaviour, though the >> internals are very different. >> >> i run without any batch manager >> >> from node0: >> mpirun -np 1 --mca btl tcp,self -host node1 ./abort >> >> exit with exit code zero :-( > > Hmmm...it works fine for me, without your patch: > > 07:35:41 $ mpirun -n 1 -mca btl tcp,self -host bend002 ./abort > Hello, World, I am 0 of 1 > -- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 2. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -- > -- > mpirun noticed that process rank 0 with PID 24382 on node bend002 exited on > signal 0 (Unknown signal 0). > -- > 07:35:56 $ showcode > 130 > >> >> short story : i applied pmix.2.patch and that fixed my problem >> could you please review this ? >> >> long story : >> i initially applied pmix.1.patch and it solved my problem >> then i ran >> mpirun -np 1 --mca btl openib,self -host node1 ./abort >> and i came back to square one : exit code is zero >> so i used the debugger and was unable to reproduce the issue >> (one more race condition, yeah !) >> finally, i wrote pmix.2.patch, fixed my issue and realized that >> pmix.1.patch was no more needed. >> currently, and assuming pmix.2.patch is correct, i cannot tell wether >> pmix.1.patch is needed or not >> since this part of the code is no more executed. >> >> i also found one hang with the following trivial program within one node : >> >> int main (int argc, char *argv[]) { >> MPI_Init(&argc, &argv); >>MPI_Finalize(); >>return 3; >> } >> >> from node0 : >> $ mpirun -np 1 ./test >> --- >> Primary job terminated normally, but 1 process returned >> a non-zero exit code.. Per user-direction, the job has been aborted. >> --- >> >> AND THE PROGRAM HANGS > > This also works fine for me: > > 07:37:27 $ mpirun -n 1 ./mpi_no_op > 07:37:36 $ cat mpi_no_op.c > /* -*- C -*- > * > * $HEADER$ > * > * The most basic of MPI applications > */ > > #include > #include "mpi.h" > > int main(int argc, char* argv[]) > { > MPI_Init(&argc, &argv); > > MPI_Finalize(); > return 0; > } > > >> >> *but* >> $ mpirun -np 1 -host node1 ./test >> --- >> Primary job terminated normally, but 1 process returned >> a non-zero exit code.. Per user-direction, the job has been aborted. >> --- >> -- >> mpirun detected that one or more processes exited with non-zero status, >> thus causing >> the job to be terminated. The first process to do so was: >> >> Process name: [[22080,1],0] >> Exit code:3 >> -- >> >> return with exit code 3. > > Likewise here - works just fine for me > > >> >> then i found a strange behaviour with helloworld if only the self btl is >> used : >> $ mpirun -np 1 --mca btl self ./hw >> [helios91:23319] OPAL dss:unpack: got type 12 when expecting type 3 >> [helios91:23319] [[22303,0],0] ORTE_ERROR_LOG: Pack data mismatch in >> file ../../../src/ompi-trunk/orte/orted/pmix/pmix_server_sendrecv.c at >> line 722 >> >> the program returns with exit code zero, but display an error message. >> >> Cheers, >> >> Gilles >> >> On 2014/08/21 6:21, Ralph Castain wrote: >>> I'm aware of the problem, but it will be fixed when the PMIx branch is >>> merged later this week. >>> >>> On Aug 19, 2014, at 10:00 PM, Gilles Gouaillardet >>> wrote: >>> Folks, let's look at the following trivial test program : #include #
Re: [OMPI devel] OMPI devel] MPI_Abort does not make mpirun return with the right exit code
I think these are fixed now - at least, your test cases all pass for me On Aug 22, 2014, at 9:12 AM, Ralph Castain wrote: > > On Aug 22, 2014, at 9:06 AM, Gilles Gouaillardet > wrote: > >> Ralph, >> >> Will do on Monday >> >> About the first test, in my case echo $? returns 0 > > My "showcode" is just an alias for the echo > >> I noticed this confusing message in your output : >> mpirun noticed that process rank 0 with PID 24382 on node bend002 exited on >> signal 0 (Unknown signal 0). > > I'll take a look at why that happened > >> >> About the second test, please note my test program return 3; >> whereas your mpi_no_op.c return 0; > > I didn't see that little cuteness - sigh > >> >> Cheers, >> >> Gilles >> >> Ralph Castain wrote: >> You might want to try again with current head of trunk as something seems >> off in what you are seeing - more below >> >> >> On Aug 22, 2014, at 3:12 AM, Gilles Gouaillardet >> wrote: >> >>> Ralph, >>> >>> i tried again after the merge and found the same behaviour, though the >>> internals are very different. >>> >>> i run without any batch manager >>> >>> from node0: >>> mpirun -np 1 --mca btl tcp,self -host node1 ./abort >>> >>> exit with exit code zero :-( >> >> Hmmm...it works fine for me, without your patch: >> >> 07:35:41 $ mpirun -n 1 -mca btl tcp,self -host bend002 ./abort >> Hello, World, I am 0 of 1 >> -- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >> with errorcode 2. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> -- >> -- >> mpirun noticed that process rank 0 with PID 24382 on node bend002 exited on >> signal 0 (Unknown signal 0). >> -- >> 07:35:56 $ showcode >> 130 >> >>> >>> short story : i applied pmix.2.patch and that fixed my problem >>> could you please review this ? >>> >>> long story : >>> i initially applied pmix.1.patch and it solved my problem >>> then i ran >>> mpirun -np 1 --mca btl openib,self -host node1 ./abort >>> and i came back to square one : exit code is zero >>> so i used the debugger and was unable to reproduce the issue >>> (one more race condition, yeah !) >>> finally, i wrote pmix.2.patch, fixed my issue and realized that >>> pmix.1.patch was no more needed. >>> currently, and assuming pmix.2.patch is correct, i cannot tell wether >>> pmix.1.patch is needed or not >>> since this part of the code is no more executed. >>> >>> i also found one hang with the following trivial program within one node : >>> >>> int main (int argc, char *argv[]) { >>> MPI_Init(&argc, &argv); >>>MPI_Finalize(); >>>return 3; >>> } >>> >>> from node0 : >>> $ mpirun -np 1 ./test >>> --- >>> Primary job terminated normally, but 1 process returned >>> a non-zero exit code.. Per user-direction, the job has been aborted. >>> --- >>> >>> AND THE PROGRAM HANGS >> >> This also works fine for me: >> >> 07:37:27 $ mpirun -n 1 ./mpi_no_op >> 07:37:36 $ cat mpi_no_op.c >> /* -*- C -*- >> * >> * $HEADER$ >> * >> * The most basic of MPI applications >> */ >> >> #include >> #include "mpi.h" >> >> int main(int argc, char* argv[]) >> { >> MPI_Init(&argc, &argv); >> >> MPI_Finalize(); >> return 0; >> } >> >> >>> >>> *but* >>> $ mpirun -np 1 -host node1 ./test >>> --- >>> Primary job terminated normally, but 1 process returned >>> a non-zero exit code.. Per user-direction, the job has been aborted. >>> --- >>> -- >>> mpirun detected that one or more processes exited with non-zero status, >>> thus causing >>> the job to be terminated. The first process to do so was: >>> >>> Process name: [[22080,1],0] >>> Exit code:3 >>> -- >>> >>> return with exit code 3. >> >> Likewise here - works just fine for me >> >> >>> >>> then i found a strange behaviour with helloworld if only the self btl is >>> used : >>> $ mpirun -np 1 --mca btl self ./hw >>> [helios91:23319] OPAL dss:unpack: got type 12 when expecting type 3 >>> [helios91:23319] [[22303,0],0] ORTE_ERROR_LOG: Pack data mismatch in >>> file ../../../src/ompi-trunk/orte/orted/pmix/pmix_server_sendrecv.c at >>> line 722 >>> >>> the program returns with exit code zero, but display an error message. >>> >>> Cheers, >>> >>> Gilles >>> >>> On 2014/08/21 6:21, Ra
Re: [OMPI devel] OMPI devel] MPI_Abort does not make mpirun return with the right exit code
Thanks Ralph ! i confirm my all test cases pass now :-) FYI, i commited r32592 in order to fix a parsing bug on 32bits platform (hence the mtt failures on trunk on x86) Cheers, Gilles On 2014/08/23 4:59, Ralph Castain wrote: > I think these are fixed now - at least, your test cases all pass for me > > > On Aug 22, 2014, at 9:12 AM, Ralph Castain wrote: > >> On Aug 22, 2014, at 9:06 AM, Gilles Gouaillardet >> wrote: >> >>> Ralph, >>> >>> Will do on Monday >>> >>> About the first test, in my case echo $? returns 0 >> My "showcode" is just an alias for the echo >> >>> I noticed this confusing message in your output : >>> mpirun noticed that process rank 0 with PID 24382 on node bend002 exited on >>> signal 0 (Unknown signal 0). >> I'll take a look at why that happened >> >>> About the second test, please note my test program return 3; >>> whereas your mpi_no_op.c return 0; >> I didn't see that little cuteness - sigh >> >>> Cheers, >>> >>> Gilles >>> >>> Ralph Castain wrote: >>> You might want to try again with current head of trunk as something seems >>> off in what you are seeing - more below >>> >>> >>> On Aug 22, 2014, at 3:12 AM, Gilles Gouaillardet >>> wrote: >>> Ralph, i tried again after the merge and found the same behaviour, though the internals are very different. i run without any batch manager from node0: mpirun -np 1 --mca btl tcp,self -host node1 ./abort exit with exit code zero :-( >>> Hmmm...it works fine for me, without your patch: >>> >>> 07:35:41 $ mpirun -n 1 -mca btl tcp,self -host bend002 ./abort >>> Hello, World, I am 0 of 1 >>> -- >>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>> with errorcode 2. >>> >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>> You may or may not see output from other processes, depending on >>> exactly when Open MPI kills them. >>> -- >>> -- >>> mpirun noticed that process rank 0 with PID 24382 on node bend002 exited on >>> signal 0 (Unknown signal 0). >>> -- >>> 07:35:56 $ showcode >>> 130 >>> short story : i applied pmix.2.patch and that fixed my problem could you please review this ? long story : i initially applied pmix.1.patch and it solved my problem then i ran mpirun -np 1 --mca btl openib,self -host node1 ./abort and i came back to square one : exit code is zero so i used the debugger and was unable to reproduce the issue (one more race condition, yeah !) finally, i wrote pmix.2.patch, fixed my issue and realized that pmix.1.patch was no more needed. currently, and assuming pmix.2.patch is correct, i cannot tell wether pmix.1.patch is needed or not since this part of the code is no more executed. i also found one hang with the following trivial program within one node : int main (int argc, char *argv[]) { MPI_Init(&argc, &argv); MPI_Finalize(); return 3; } from node0 : $ mpirun -np 1 ./test --- Primary job terminated normally, but 1 process returned a non-zero exit code.. Per user-direction, the job has been aborted. --- AND THE PROGRAM HANGS >>> This also works fine for me: >>> >>> 07:37:27 $ mpirun -n 1 ./mpi_no_op >>> 07:37:36 $ cat mpi_no_op.c >>> /* -*- C -*- >>> * >>> * $HEADER$ >>> * >>> * The most basic of MPI applications >>> */ >>> >>> #include >>> #include "mpi.h" >>> >>> int main(int argc, char* argv[]) >>> { >>> MPI_Init(&argc, &argv); >>> >>> MPI_Finalize(); >>> return 0; >>> } >>> >>> *but* $ mpirun -np 1 -host node1 ./test --- Primary job terminated normally, but 1 process returned a non-zero exit code.. Per user-direction, the job has been aborted. --- -- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[22080,1],0] Exit code:3 -- return with exit code 3. >>> Likewise here - works just fine for me >>> >>> then i found a strange behaviour with helloworld if only the self btl is used : $ mpirun -np 1 --mca btl self ./hw [helios91:23319] OPAL dss:unpack: got type 12 when expecting type 3 [he