When you fork that process off, do you set its process group? Or is it in the same process group as the shell script?
> On Jun 19, 2017, at 10:19 AM, Ted Sussman <ted.suss...@adina.com> wrote: > > If I replace the sleep with an infinite loop, I get the same behavior. One > "aborttest" process > remains after all the signals are sent. > > On 19 Jun 2017 at 10:10, r...@open-mpi.org wrote: > >> >> That is typical behavior when you throw something into "sleep" - not much we >> can do about it, I >> think. >> >> On Jun 19, 2017, at 9:58 AM, Ted Sussman <ted.suss...@adina.com> wrote: >> >> Hello, >> >> I have rebuilt Open MPI 2.1.1 on the same computer, including >> --enable-debug. >> >> I have attached the abort test program aborttest10.tgz. This version >> sleeps for 5 sec before >> calling MPI_ABORT, so that I can check the pids using ps. >> >> This is what happens (see run2.sh.out). >> >> Open MPI invokes two instances of dum.sh. Each instance of dum.sh >> invokes aborttest.exe. >> >> Pid Process >> ------------------- >> 19565 dum.sh >> 19566 dum.sh >> 19567 aborttest10.exe >> 19568 aborttest10.exe >> >> When MPI_ABORT is called, Open MPI sends SIGCONT, SIGTERM and SIGKILL to >> both >> instances of dum.sh (pids 19565 and 19566). >> >> ps shows that both the shell processes vanish, and that one of the >> aborttest10.exe processes >> vanishes. But the other aborttest10.exe remains and continues until it >> is finished sleeping. >> >> Hope that this information is useful. >> >> Sincerely, >> >> Ted Sussman >> >> >> >> On 19 Jun 2017 at 23:06, gil...@rist.or.jp wrote: >> >> >> Ted, >> >> some traces are missing because you did not configure with --enable-debug >> i am afraid you have to do it (and you probably want to install that >> debug version in an >> other >> location since its performances are not good for production) in order to >> get all the logs. >> >> Cheers, >> >> Gilles >> >> ----- Original Message ----- >> Hello Gilles, >> >> I retried my example, with the same results as I observed before. The >> process with rank >> 1 >> does not get killed by MPI_ABORT. >> >> I have attached to this E-mail: >> >> config.log.bz2 >> ompi_info.bz2 (uses ompi_info -a) >> aborttest09.tgz >> >> This testing is done on a computer running Linux 3.10.0. This is a >> different computer >> than >> the computer that I previously used for testing. You can confirm that >> I am using Open >> MPI >> 2.1.1. >> >> tar xvzf aborttest09.tgz >> cd aborttest09 >> ./sh run2.sh >> >> run2.sh contains the command >> >> /opt/openmpi-2.1.1-GNU/bin/mpirun -np 2 -mca btl tcp,self --mca >> odls_base_verbose >> 10 >> ./dum.sh >> >> The output from this run is in aborttest09/run2.sh.out. >> >> The output shows that the the "default" component is selected by odls. >> >> The only messages from odls are: odls: launch spawning child ... (two >> messages). >> There >> are no messages from odls with "kill" and I see no SENDING SIGCONT / >> SIGKILL >> messages. >> >> I am not running from within any batch manager. >> >> Sincerely, >> >> Ted Sussman >> >> On 17 Jun 2017 at 16:02, gil...@rist.or.jp wrote: >> >> Ted, >> >> i do not observe the same behavior you describe with Open MPI 2.1.1 >> >> # mpirun -np 2 -mca btl tcp,self --mca odls_base_verbose 5 ./abort.sh >> >> abort.sh 31361 launching abort >> abort.sh 31362 launching abort >> I am rank 0 with pid 31363 >> I am rank 1 with pid 31364 >> ------------------------------------------------------------------------ >> -- >> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >> with errorcode 1. >> >> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >> You may or may not see output from other processes, depending on >> exactly when Open MPI kills them. >> ------------------------------------------------------------------------ >> -- >> [linux:31356] [[18199,0],0] odls:kill_local_proc working on WILDCARD >> [linux:31356] [[18199,0],0] odls:kill_local_proc checking child process >> [[18199,1],0] >> [linux:31356] [[18199,0],0] SENDING SIGCONT TO [[18199,1],0] >> [linux:31356] [[18199,0],0] odls:default:SENT KILL 18 TO PID 31361 >> SUCCESS >> [linux:31356] [[18199,0],0] odls:kill_local_proc checking child process >> [[18199,1],1] >> [linux:31356] [[18199,0],0] SENDING SIGCONT TO [[18199,1],1] >> [linux:31356] [[18199,0],0] odls:default:SENT KILL 18 TO PID 31362 >> SUCCESS >> [linux:31356] [[18199,0],0] SENDING SIGTERM TO [[18199,1],0] >> [linux:31356] [[18199,0],0] odls:default:SENT KILL 15 TO PID 31361 >> SUCCESS >> [linux:31356] [[18199,0],0] SENDING SIGTERM TO [[18199,1],1] >> [linux:31356] [[18199,0],0] odls:default:SENT KILL 15 TO PID 31362 >> SUCCESS >> [linux:31356] [[18199,0],0] SENDING SIGKILL TO [[18199,1],0] >> [linux:31356] [[18199,0],0] odls:default:SENT KILL 9 TO PID 31361 >> SUCCESS >> [linux:31356] [[18199,0],0] SENDING SIGKILL TO [[18199,1],1] >> [linux:31356] [[18199,0],0] odls:default:SENT KILL 9 TO PID 31362 >> SUCCESS >> [linux:31356] [[18199,0],0] odls:kill_local_proc working on WILDCARD >> [linux:31356] [[18199,0],0] odls:kill_local_proc checking child process >> [[18199,1],0] >> [linux:31356] [[18199,0],0] odls:kill_local_proc child [[18199,1],0] is >> not alive >> [linux:31356] [[18199,0],0] odls:kill_local_proc checking child process >> [[18199,1],1] >> [linux:31356] [[18199,0],0] odls:kill_local_proc child [[18199,1],1] is >> not alive >> >> >> Open MPI did kill both shells, and they were indeed killed as evidenced >> by ps >> >> #ps -fu gilles --forest >> UID PID PPID C STIME TTY TIME CMD >> gilles 1564 1561 0 15:39 ? 00:00:01 sshd: gilles@pts/1 >> gilles 1565 1564 0 15:39 pts/1 00:00:00 \_ -bash >> gilles 31356 1565 3 15:57 pts/1 00:00:00 \_ /home/gilles/ >> local/ompi-v2.x/bin/mpirun -np 2 -mca btl tcp,self --mca odls_base >> gilles 31364 1 1 15:57 pts/1 00:00:00 ./abort >> >> >> so trapping SIGTERM in your shell and manually killing the MPI task >> should work >> (as Jeff explained, as long as the shell script is fast enough to do >> that between SIGTERM and SIGKILL) >> >> >> if you observe a different behavior, please double check your Open MPI >> version and post the outputs of the same commands. >> >> btw, are you running from a batch manager ? if yes, which one ? >> >> Cheers, >> >> Gilles >> >> ----- Original Message ----- >> Ted, >> >> if you >> >> mpirun --mca odls_base_verbose 10 ... >> >> you will see which processes get killed and how >> >> Best regards, >> >> >> Gilles >> >> ----- Original Message ----- >> Hello Jeff, >> >> Thanks for your comments. >> >> I am not seeing behavior #4, on the two computers that I have >> tested >> on, using Open MPI >> 2.1.1. >> >> I wonder if you can duplicate my results with the files that I have >> uploaded. >> >> Regarding what is the "correct" behavior, I am willing to modify my >> application to correspond >> to Open MPI's behavior (whatever behavior the Open MPI >> developers >> decide is best) -- >> provided that Open MPI does in fact kill off both shells. >> >> So my highest priority now is to find out why Open MPI 2.1.1 does >> not >> kill off both shells on >> my computer. >> >> Sincerely, >> >> Ted Sussman >> >> On 16 Jun 2017 at 16:35, Jeff Squyres (jsquyres) wrote: >> >> Ted -- >> >> Sorry for jumping in late. Here's my $0.02... >> >> In the runtime, we can do 4 things: >> >> 1. Kill just the process that we forked. >> 2. Kill just the process(es) that call back and identify >> themselves >> as MPI processes (we don't track this right now, but we could add that >> functionality). >> 3. Union of #1 and #2. >> 4. Kill all processes (to include any intermediate processes >> that >> are not included in #1 and #2). >> >> In Open MPI 2.x, #4 is the intended behavior. There may be a >> bug >> or >> two that needs to get fixed (e.g., in your last mail, I don't see >> offhand why it waits until the MPI process finishes sleeping), but we >> should be killing the process group, which -- unless any of the >> descendant processes have explicitly left the process group -- should >> hit the entire process tree. >> >> Sidenote: there's actually a way to be a bit more aggressive >> and >> do >> a better job of ensuring that we kill *all* processes (via creative >> use >> of PR_SET_CHILD_SUBREAPER), but that's basically a future >> enhancement >> / >> optimization. >> >> I think Gilles and Ralph proposed a good point to you: if you >> want >> to be sure to be able to do cleanup after an MPI process terminates ( >> normally or abnormally), you should trap signals in your intermediate >> processes to catch what Open MPI's runtime throws and therefore know >> that it is time to cleanup. >> >> Hypothetically, this should work in all versions of Open MPI...? >> >> I think Ralph made a pull request that adds an MCA param to >> change >> the default behavior from #4 to #1. >> >> Note, however, that there's a little time between when Open >> MPI >> sends the SIGTERM and the SIGKILL, so this solution could be racy. If >> you find that you're running out of time to cleanup, we might be able >> to >> make the delay between the SIGTERM and SIGKILL be configurable >> (e.g., >> via MCA param). >> >> >> >> >> On Jun 16, 2017, at 10:08 AM, Ted Sussman >> <ted.suss...@adina.com >> >> wrote: >> >> Hello Gilles and Ralph, >> >> Thank you for your advice so far. I appreciate the time >> that >> you >> have spent to educate me about the details of Open MPI. >> >> But I think that there is something fundamental that I >> don't >> understand. Consider Example 2 run with Open MPI 2.1.1. >> >> mpirun --> shell for process 0 --> executable for process >> 0 --> >> MPI calls, MPI_Abort >> --> shell for process 1 --> executable for process 1 --> >> MPI calls >> >> After the MPI_Abort is called, ps shows that both shells >> are >> running, and that the executable for process 1 is running (in this >> case, >> process 1 is sleeping). And mpirun does not exit until process 1 is >> finished sleeping. >> >> I cannot reconcile this observed behavior with the >> statement >> >> > 2.x: each process is put into its own process group >> upon launch. When we issue a >> > "kill", we issue it to the process group. Thus, >> every >> child proc of that child proc will >> > receive it. IIRC, this was the intended behavior. >> >> I assume that, for my example, there are two process >> groups. >> The >> process group for process 0 contains the shell for process 0 and the >> executable for process 0; and the process group for process 1 contains >> the shell for process 1 and the executable for process 1. So what >> does >> MPI_ABORT do? MPI_ABORT does not kill the process group for process >> 0, >> >> since the shell for process 0 continues. And MPI_ABORT does not kill >> the process group for process 1, since both the shell and executable >> for >> process 1 continue. >> >> If I hit Ctrl-C after MPI_Abort is called, I get the message >> >> mpirun: abort is already in progress.. hit ctrl-c again to >> forcibly terminate >> >> but I don't need to hit Ctrl-C again because mpirun >> immediately >> exits. >> >> Can you shed some light on all of this? >> >> Sincerely, >> >> Ted Sussman >> >> >> On 15 Jun 2017 at 14:44, r...@open-mpi.org wrote: >> >> >> You have to understand that we have no way of >> knowing who is >> making MPI calls - all we see is >> the proc that we started, and we know someone of >> that rank is >> running (but we have no way of >> knowing which of the procs you sub-spawned it is). >> >> So the behavior you are seeking only occurred in >> some earlier >> release by sheer accident. Nor will >> you find it portable as there is no specification >> directing >> that >> behavior. >> >> The behavior I´ve provided is to either deliver the >> signal to >> _ >> all_ child processes (including >> grandchildren etc.), or _only_ the immediate child >> of the >> daemon. >> It won´t do what you describe - >> kill the mPI proc underneath the shell, but not the >> shell >> itself. >> >> What you can eventually do is use PMIx to ask the >> runtime to >> selectively deliver signals to >> pid/procs for you. We don´t have that capability >> implemented >> just yet, I´m afraid. >> >> Meantime, when I get a chance, I can code an >> option that will >> record the pid of the subproc that >> calls MPI_Init, and then let´s you deliver signals to >> just >> that >> proc. No promises as to when that will >> be done. >> >> >> On Jun 15, 2017, at 1:37 PM, Ted Sussman >> <ted.sussman@ >> adina. >> com> wrote: >> >> Hello Ralph, >> >> I am just an Open MPI end user, so I will need to >> wait for >> the next official release. >> >> mpirun --> shell for process 0 --> executable for >> process >> 0 >> --> MPI calls >> --> shell for process 1 --> executable for process >> 1 >> --> MPI calls >> ... >> >> I guess the question is, should MPI_ABORT kill the >> executables or the shells? I naively >> thought, that, since it is the executables that make >> the >> MPI >> calls, it is the executables that >> should be aborted by the call to MPI_ABORT. Since >> the >> shells don't make MPI calls, the >> shells should not be aborted. >> >> And users might have several layers of shells in >> between >> mpirun and the executable. >> >> So now I will look for the latest version of Open MPI >> that >> has the 1.4.3 behavior. >> >> Sincerely, >> >> Ted Sussman >> >> On 15 Jun 2017 at 12:31, r...@open-mpi.org wrote: >> >> > >> > Yeah, things jittered a little there as we debated >> the " >> right" behavior. Generally, when we >> see that >> > happening it means that a param is required, but >> somehow >> we never reached that point. >> > >> > See if https://github.com/open-mpi/ompi/pull/3704 >> helps >> - >> if so, I can schedule it for the next >> 2.x >> > release if the RMs agree to take it >> > >> > Ralph >> > >> > On Jun 15, 2017, at 12:20 PM, Ted Sussman <ted. >> sussman >> @adina.com > wrote: >> > >> > Thank you for your comments. >> > >> > Our application relies upon "dum.sh" to clean up >> after >> the process exits, either if the >> process >> > exits normally, or if the process exits abnormally >> because of MPI_ABORT. If the process >> > group is killed by MPI_ABORT, this clean up will not >> be performed. If exec is used to launch >> > the executable from dum.sh, then dum.sh is >> terminated >> by the exec, so dum.sh cannot >> > perform any clean up. >> > >> > I suppose that other user applications might work >> similarly, so it would be good to have an >> > MCA parameter to control the behavior of >> MPI_ABORT. >> > >> > We could rewrite our shell script that invokes >> mpirun, >> so that the cleanup that is now done >> > by >> > dum.sh is done by the invoking shell script after >> mpirun exits. Perhaps this technique is the >> > preferred way to clean up after mpirun is invoked. >> > >> > By the way, I have also tested with Open MPI >> 1.10.7, >> and Open MPI 1.10.7 has different >> > behavior than either Open MPI 1.4.3 or Open MPI >> 2.1. >> 1. >> In this explanation, it is important to >> > know that the aborttest executable sleeps for 20 >> sec. >> > >> > When running example 2: >> > >> > 1.4.3: process 1 immediately aborts >> > 1.10.7: process 1 doesn't abort and never stops. >> > 2.1.1 process 1 doesn't abort, but stops after it is >> finished sleeping >> > >> > Sincerely, >> > >> > Ted Sussman >> > >> > On 15 Jun 2017 at 9:18, r...@open-mpi.org wrote: >> > >> > Here is how the system is working: >> > >> > Master: each process is put into its own process >> group >> upon launch. When we issue a >> > "kill", however, we only issue it to the individual >> process (instead of the process group >> > that is headed by that child process). This is >> probably a bug as I don´t believe that is >> > what we intended, but set that aside for now. >> > >> > 2.x: each process is put into its own process group >> upon launch. When we issue a >> > "kill", we issue it to the process group. Thus, >> every >> child proc of that child proc will >> > receive it. IIRC, this was the intended behavior. >> > >> > It is rather trivial to make the change (it only >> involves 3 lines of code), but I´m not sure >> > of what our intended behavior is supposed to be. >> Once >> we clarify that, it is also trivial >> > to add another MCA param (you can never have too >> many!) >> to allow you to select the >> > other behavior. >> > >> > >> > On Jun 15, 2017, at 5:23 AM, Ted Sussman <ted. >> sussman@ >> adina.com > wrote: >> > >> > Hello Gilles, >> > >> > Thank you for your quick answer. I confirm that if >> exec is used, both processes >> > immediately >> > abort. >> > >> > Now suppose that the line >> > >> > echo "After aborttest: >> > >> OMPI_COMM_WORLD_RANK="$OMPI_COMM_ >> WORLD_RANK >> > >> > is added to the end of dum.sh. >> > >> > If Example 2 is run with Open MPI 1.4.3, the output >> is >> > >> > After aborttest: OMPI_COMM_WORLD_RANK=0 >> > >> > which shows that the shell script for the process >> with >> rank 0 continues after the >> > abort, >> > but that the shell script for the process with rank >> 1 >> does not continue after the >> > abort. >> > >> > If Example 2 is run with Open MPI 2.1.1, with exec >> used to invoke >> > aborttest02.exe, then >> > there is no such output, which shows that both shell >> scripts do not continue after >> > the abort. >> > >> > I prefer the Open MPI 1.4.3 behavior because our >> original application depends >> > upon the >> > Open MPI 1.4.3 behavior. (Our original application >> will also work if both >> > executables are >> > aborted, and if both shell scripts continue after >> the >> abort.) >> > >> > It might be too much to expect, but is there a way >> to >> recover the Open MPI 1.4.3 >> > behavior >> > using Open MPI 2.1.1? >> > >> > Sincerely, >> > >> > Ted Sussman >> > >> > >> > On 15 Jun 2017 at 9:50, Gilles Gouaillardet wrote: >> > >> > Ted, >> > >> > >> > fwiw, the 'master' branch has the behavior you >> expect. >> > >> > >> > meanwhile, you can simple edit your 'dum.sh' script >> and replace >> > >> > /home/buildadina/src/aborttest02/aborttest02.exe >> > >> > with >> > >> > exec /home/buildadina/src/aborttest02/aborttest02. >> exe >> > >> > >> > Cheers, >> > >> > >> > Gilles >> > >> > >> > On 6/15/2017 3:01 AM, Ted Sussman wrote: >> > Hello, >> > >> > My question concerns MPI_ABORT, indirect >> execution >> of >> > executables by mpirun and Open >> > MPI 2.1.1. When mpirun runs executables directly, >> MPI >> _ABORT >> > works as expected, but >> > when mpirun runs executables indirectly, >> MPI_ABORT >> does not >> > work as expected. >> > >> > If Open MPI 1.4.3 is used instead of Open MPI >> 2.1.1, >> MPI_ABORT >> > works as expected in all >> > cases. >> > >> > The examples given below have been simplified as >> far >> as possible >> > to show the issues. >> > >> > --- >> > >> > Example 1 >> > >> > Consider an MPI job run in the following way: >> > >> > mpirun ... -app addmpw1 >> > >> > where the appfile addmpw1 lists two executables: >> > >> > -n 1 -host gulftown ... aborttest02.exe >> > -n 1 -host gulftown ... aborttest02.exe >> > >> > The two executables are executed on the local node >> gulftown. >> > aborttest02 calls MPI_ABORT >> > for rank 0, then sleeps. >> > >> > The above MPI job runs as expected. Both >> processes >> immediately >> > abort when rank 0 calls >> > MPI_ABORT. >> > >> > --- >> > >> > Example 2 >> > >> > Now change the above example as follows: >> > >> > mpirun ... -app addmpw2 >> > >> > where the appfile addmpw2 lists shell scripts: >> > >> > -n 1 -host gulftown ... dum.sh >> > -n 1 -host gulftown ... dum.sh >> > >> > dum.sh invokes aborttest02.exe. So aborttest02.exe >> is >> executed >> > indirectly by mpirun. >> > >> > In this case, the MPI job only aborts process 0 when >> rank 0 calls >> > MPI_ABORT. Process 1 >> > continues to run. This behavior is unexpected. >> > >> > ---- >> > >> > I have attached all files to this E-mail. Since >> there >> are absolute >> > pathnames in the files, to >> > reproduce my findings, you will need to update the >> pathnames in the >> > appfiles and shell >> > scripts. To run example 1, >> > >> > sh run1.sh >> > >> > and to run example 2, >> > >> > sh run2.sh >> > >> > --- >> > >> > I have tested these examples with Open MPI 1.4.3 >> and >> 2. >> 0.3. In >> > Open MPI 1.4.3, both >> > examples work as expected. Open MPI 2.0.3 has >> the >> same behavior >> > as Open MPI 2.1.1. >> > >> > --- >> > >> > I would prefer that Open MPI 2.1.1 aborts both >> processes, even >> > when the executables are >> > invoked indirectly by mpirun. If there is an MCA >> setting that is >> > needed to make Open MPI >> > 2.1.1 abort both processes, please let me know. >> > >> > >> > Sincerely, >> > >> > Theodore Sussman >> > >> > >> > The following section of this message contains a >> file >> attachment >> > prepared for transmission using the Internet MIME >> message format. >> > If you are using Pegasus Mail, or any other MIME- >> compliant system, >> > you should be able to save it or view it from within >> your mailer. >> > If you cannot, please ask your system administrator >> for assistance. >> > >> > ---- File information ----------- >> > File: config.log.bz2 >> > Date: 14 Jun 2017, 13:35 >> > Size: 146548 bytes. >> > Type: Binary >> > >> > >> > The following section of this message contains a >> file >> attachment >> > prepared for transmission using the Internet MIME >> message format. >> > If you are using Pegasus Mail, or any other MIME- >> compliant system, >> > you should be able to save it or view it from within >> your mailer. >> > If you cannot, please ask your system administrator >> for assistance. >> > >> > ---- File information ----------- >> > File: ompi_info.bz2 >> > Date: 14 Jun 2017, 13:35 >> > Size: 24088 bytes. >> > Type: Binary >> > >> > >> > The following section of this message contains a >> file >> attachment >> > prepared for transmission using the Internet MIME >> message format. >> > If you are using Pegasus Mail, or any other MIME- >> compliant system, >> > you should be able to save it or view it from within >> your mailer. >> > If you cannot, please ask your system administrator >> for assistance. >> > >> > ---- File information ----------- >> > File: aborttest02.tgz >> > Date: 14 Jun 2017, 13:52 >> > Size: 4285 bytes. >> > Type: Binary >> > >> > >> > >> ________________________________________ >> _______ >> > users mailing list >> > users@lists.open-mpi.org >> > >> https://rfd.newmexicoconsortium.org/mailman/listin >> fo/users >> >> >> > >> > >> ________________________________________ >> _______ >> > users mailing list >> > users@lists.open-mpi.org >> > >> https://rfd.newmexicoconsortium.org/mailman/listin >> fo/users >> >> >> > >> > >> > >> > >> ________________________________________ >> _______ >> > users mailing list >> > users@lists.open-mpi.org >> > >> https://rfd.newmexicoconsortium.org/mailman/listin >> fo/users >> >> >> > >> > >> ________________________________________ >> _______ >> > users mailing list >> > users@lists.open-mpi.org >> > >> https://rfd.newmexicoconsortium.org/mailman/listin >> fo/users >> >> >> > >> > >> > >> > >> ________________________________________ >> _______ >> > users mailing list >> > users@lists.open-mpi.org >> > >> https://rfd.newmexicoconsortium.org/mailman/listin >> fo/users >> >> >> > >> >> >> __________________________________________ >> _____ >> users mailing list >> users@lists.open-mpi.org >> >> https://rfd.newmexicoconsortium.org/mailman/listin >> fo/users >> >> >> >> _____________________________________________ >> __ >> users mailing list >> users@lists.open-mpi.org >> https://rfd.newmexicoconsortium.org/mailman/listinfo/us >> ers >> >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> >> _______________________________________________ >> users mailing list >> users@lists.open-mpi.org >> https://rfd.newmexicoconsortium.org/mailman/listinfo/users >> >> >> >> _______________________________________________ >> users mailing list >> users@lists.open-mpi.org >> https://rfd.newmexicoconsortium.org/mailman/listinfo/users >> >> _______________________________________________ >> users mailing list >> users@lists.open-mpi.org >> https://rfd.newmexicoconsortium.org/mailman/listinfo/users >> >> _______________________________________________ >> users mailing list >> users@lists.open-mpi.org >> https://rfd.newmexicoconsortium.org/mailman/listinfo/users >> >> >> >> >> The following section of this message contains a file attachment >> prepared for transmission using the Internet MIME message format. >> If you are using Pegasus Mail, or any other MIME-compliant system, >> you should be able to save it or view it from within your mailer. >> If you cannot, please ask your system administrator for assistance. >> >> ---- File information ----------- >> File: aborttest10.tgz >> Date: 19 Jun 2017, 12:42 >> Size: 4740 bytes. >> Type: Binary >> <aborttest10.tgz>_______________________________________________ >> users mailing list >> users@lists.open-mpi.org >> https://rfd.newmexicoconsortium.org/mailman/listinfo/users >> > > > > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
