I've received SIGSEGV a few times for different reasons with OpenMPI Java and one of the most common reasons was the ulimit settings. You might want to look at -l (max lock memory) -u (max user processes), -n (open files).
Here's a snapshot of what we use in our clusters running OpenMPI and Java core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 515696 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 196608 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited On Wed, Jul 6, 2016 at 9:00 PM, Gilles Gouaillardet <gil...@rist.or.jp> wrote: > Gundram, > > > fwiw, i cannot reproduce the issue on my box > > - centos 7 > > - java version "1.8.0_71" > Java(TM) SE Runtime Environment (build 1.8.0_71-b15) > Java HotSpot(TM) 64-Bit Server VM (build 25.71-b15, mixed mode) > > > i noticed on non zero rank saveMem is allocated at each iteration. > ideally, the garbage collector can take care of that and this should not > be an issue. > > would you mind giving the attached file a try ? > > Cheers, > > Gilles > > > On 7/7/2016 7:41 AM, Gilles Gouaillardet wrote: > > I will have a look at it today > > how did you configure OpenMPI ? > > Cheers, > > Gilles > > On Thursday, July 7, 2016, Gundram Leifert < > <gundram.leif...@uni-rostock.de>gundram.leif...@uni-rostock.de> wrote: > >> Hello Giles, >> >> thank you for your hints! I did 3 changes, unfortunately the same error >> occures: >> >> update ompi: >> commit ae8444682f0a7aa158caea08800542ce9874455e >> Author: Ralph Castain <r...@open-mpi.org> >> Date: Tue Jul 5 20:07:16 2016 -0700 >> >> update java: >> java version "1.8.0_92" >> Java(TM) SE Runtime Environment (build 1.8.0_92-b14) >> Java HotSpot(TM) Server VM (build 25.92-b14, mixed mode) >> >> delete hashcode-lines. >> >> Now I get this error message - to 100%, after different number of >> iterations (15-300): >> >> 0/ 3:length = 100000000 >> 0/ 3:bcast length done (length = 100000000) >> 1/ 3:bcast length done (length = 100000000) >> 2/ 3:bcast length done (length = 100000000) >> # >> # A fatal error has been detected by the Java Runtime Environment: >> # >> # SIGSEGV (0xb) at pc=0x00002b3d022fcd24, pid=16578, >> tid=0x00002b3d29716700 >> # >> # JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build >> 1.8.0_92-b14) >> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode >> linux-amd64 compressed oops) >> # Problematic frame: >> # V [libjvm.so+0x414d24] ciEnv::get_field_by_index(ciInstanceKlass*, >> int)+0x94 >> # >> # Failed to write core dump. Core dumps have been disabled. To enable >> core dumping, try "ulimit -c unlimited" before starting Java again >> # >> # An error report file with more information is saved as: >> # /home/gl069/ompi/bin/executor/hs_err_pid16578.log >> # >> # Compiler replay data is saved as: >> # /home/gl069/ompi/bin/executor/replay_pid16578.log >> # >> # If you would like to submit a bug report, please visit: >> # http://bugreport.java.com/bugreport/crash.jsp >> # >> [titan01:16578] *** Process received signal *** >> [titan01:16578] Signal: Aborted (6) >> [titan01:16578] Signal code: (-6) >> [titan01:16578] [ 0] /usr/lib64/libpthread.so.0(+0xf100)[0x2b3d01500100] >> [titan01:16578] [ 1] /usr/lib64/libc.so.6(gsignal+0x37)[0x2b3d01b5c5f7] >> [titan01:16578] [ 2] /usr/lib64/libc.so.6(abort+0x148)[0x2b3d01b5dce8] >> [titan01:16578] [ 3] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x91e605)[0x2b3d02806605] >> [titan01:16578] [ 4] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0xabda63)[0x2b3d029a5a63] >> [titan01:16578] [ 5] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x14f)[0x2b3d0280be2f] >> [titan01:16578] [ 6] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x91a5c3)[0x2b3d028025c3] >> [titan01:16578] [ 7] /usr/lib64/libc.so.6(+0x35670)[0x2b3d01b5c670] >> [titan01:16578] [ 8] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x414d24)[0x2b3d022fcd24] >> [titan01:16578] [ 9] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x43c5ae)[0x2b3d023245ae] >> [titan01:16578] [10] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x369ade)[0x2b3d02251ade] >> [titan01:16578] [11] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x36eda0)[0x2b3d02256da0] >> [titan01:16578] [12] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x37091b)[0x2b3d0225891b] >> [titan01:16578] [13] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x3712b6)[0x2b3d022592b6] >> [titan01:16578] [14] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x36d2cf)[0x2b3d022552cf] >> [titan01:16578] [15] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x36e412)[0x2b3d02256412] >> [titan01:16578] [16] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x36ed8d)[0x2b3d02256d8d] >> [titan01:16578] [17] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x37091b)[0x2b3d0225891b] >> [titan01:16578] [18] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x3712b6)[0x2b3d022592b6] >> [titan01:16578] [19] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x36d2cf)[0x2b3d022552cf] >> [titan01:16578] [20] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x36e412)[0x2b3d02256412] >> [titan01:16578] [21] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x36ed8d)[0x2b3d02256d8d] >> [titan01:16578] [22] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x3708c2)[0x2b3d022588c2] >> [titan01:16578] [23] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x3724e7)[0x2b3d0225a4e7] >> [titan01:16578] [24] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x37a817)[0x2b3d02262817] >> [titan01:16578] [25] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x37a92f)[0x2b3d0226292f] >> [titan01:16578] [26] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x358edb)[0x2b3d02240edb] >> [titan01:16578] [27] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x35929e)[0x2b3d0224129e] >> [titan01:16578] [28] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x3593ce)[0x2b3d022413ce] >> [titan01:16578] [29] >> /home/gl069/bin/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so(+0x35973e)[0x2b3d0224173e] >> [titan01:16578] *** End of error message *** >> ------------------------------------------------------- >> Primary job terminated normally, but 1 process returned >> a non-zero exit code. Per user-direction, the job has been aborted. >> ------------------------------------------------------- >> -------------------------------------------------------------------------- >> mpirun noticed that process rank 2 with PID 0 on node titan01 exited on >> signal 6 (Aborted). >> -------------------------------------------------------------------------- >> >> I don't know if it is a problem of java or ompi - but the last years, >> java worked with no problems on my machine... >> >> Thank you for your tips in advance! >> Gundram >> >> On 07/06/2016 03:10 PM, Gilles Gouaillardet wrote: >> >> Note a race condition in MPI_Init has been fixed yesterday in the master. >> can you please update your OpenMPI and try again ? >> >> hopefully the hang will disappear. >> >> Can you reproduce the crash with a simpler (and ideally deterministic) >> version of your program. >> the crash occurs in hashcode, and this makes little sense to me. can you >> also update your jdk ? >> >> Cheers, >> >> Gilles >> >> On Wednesday, July 6, 2016, Gundram Leifert < >> gundram.leif...@uni-rostock.de> wrote: >> >>> Hello Jason, >>> >>> thanks for your response! I thing it is another problem. I try to send >>> 100MB bytes. So there are not many tries (between 10 and 30). I realized >>> that the execution of this code can result 3 different errors: >>> >>> 1. most often the posted error message occures. >>> >>> 2. in <10% the cases i have a live lock. I can see 3 java-processes, one >>> with 200% and two with 100% processor utilization. After ~15 minutes >>> without new system outputs this error occurs. >>> >>> >>> [thread 47499823949568 also had an error] >>> # A fatal error has been detected by the Java Runtime Environment: >>> # >>> # Internal Error (safepoint.cpp:317), pid=24256, tid=47500347131648 >>> # guarantee(PageArmed == 0) failed: invariant >>> # >>> # JRE version: 7.0_25-b15 >>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode >>> linux-amd64 compressed oops) >>> # Failed to write core dump. Core dumps have been disabled. To enable >>> core dumping, try "ulimit -c unlimited" before starting Java again >>> # >>> # An error report file with more information is saved as: >>> # /home/gl069/ompi/bin/executor/hs_err_pid24256.log >>> # >>> # If you would like to submit a bug report, please visit: >>> # http://bugreport.sun.com/bugreport/crash.jsp >>> # >>> [titan01:24256] *** Process received signal *** >>> [titan01:24256] Signal: Aborted (6) >>> [titan01:24256] Signal code: (-6) >>> [titan01:24256] [ 0] /usr/lib64/libpthread.so.0(+0xf100)[0x2b336a324100] >>> [titan01:24256] [ 1] /usr/lib64/libc.so.6(gsignal+0x37)[0x2b336a9815f7] >>> [titan01:24256] [ 2] /usr/lib64/libc.so.6(abort+0x148)[0x2b336a982ce8] >>> [titan01:24256] [ 3] >>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x742ac5)[0x2b336b44fac5] >>> [titan01:24256] [ 4] >>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x8a2137)[0x2b336b5af137] >>> [titan01:24256] [ 5] >>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x407262)[0x2b336b114262] >>> [titan01:24256] [ 6] >>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x7c6c34)[0x2b336b4d3c34] >>> [titan01:24256] [ 7] >>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x8a9c17)[0x2b336b5b6c17] >>> [titan01:24256] [ 8] >>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x8aa2c0)[0x2b336b5b72c0] >>> [titan01:24256] [ 9] >>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x744270)[0x2b336b451270] >>> [titan01:24256] [10] /usr/lib64/libpthread.so.0(+0x7dc5)[0x2b336a31cdc5] >>> [titan01:24256] [11] /usr/lib64/libc.so.6(clone+0x6d)[0x2b336aa4228d] >>> [titan01:24256] *** End of error message *** >>> ------------------------------------------------------- >>> Primary job terminated normally, but 1 process returned >>> a non-zero exit code. Per user-direction, the job has been aborted. >>> ------------------------------------------------------- >>> >>> -------------------------------------------------------------------------- >>> mpirun noticed that process rank 0 with PID 0 on node titan01 exited on >>> signal 6 (Aborted). >>> >>> -------------------------------------------------------------------------- >>> >>> >>> 3. in <10% the cases i have a dead lock while MPI.init. This stays for >>> more than 15 minutes without returning with an error message... >>> >>> Can I enable some debug-flags to see what happens on C / OpenMPI side? >>> >>> Thanks in advance for your help! >>> Gundram Leifert >>> >>> >>> On 07/05/2016 06:05 PM, Jason Maldonis wrote: >>> >>> After reading your thread looks like it may be related to an issue I had >>> a few weeks ago (I'm a novice though). Maybe my thread will be of help: >>> <https://www.open-mpi.org/community/lists/users/2016/06/29425.php> >>> https://www.open-mpi.org/community/lists/users/2016/06/29425.php >>> >>> When you say "After a specific number of repetitions the process either >>> hangs up or returns with a SIGSEGV." does you mean that a single call >>> hangs, or that at some point during the for loop a call hangs? If you mean >>> the latter, then it might relate to my issue. Otherwise my thread probably >>> won't be helpful. >>> >>> Jason Maldonis >>> Research Assistant of Professor Paul Voyles >>> Materials Science Grad Student >>> University of Wisconsin, Madison >>> 1509 University Ave, Rm M142 >>> Madison, WI 53706 >>> maldo...@wisc.edu >>> 608-295-5532 >>> >>> On Tue, Jul 5, 2016 at 9:58 AM, Gundram Leifert < >>> gundram.leif...@uni-rostock.de> wrote: >>> >>>> Hello, >>>> >>>> I try to send many byte-arrays via broadcast. After a specific number >>>> of repetitions the process either hangs up or returns with a SIGSEGV. Does >>>> any one can help me solving the problem: >>>> >>>> ########## The code: >>>> >>>> import java.util.Random; >>>> import mpi.*; >>>> >>>> public class TestSendBigFiles { >>>> >>>> public static void log(String msg) { >>>> try { >>>> System.err.println(String.format("%2d/%2d:%s", >>>> MPI.COMM_WORLD.getRank(), MPI.COMM_WORLD.getSize(), msg)); >>>> } catch (MPIException ex) { >>>> System.err.println(String.format("%2s/%2s:%s", "?", "?", >>>> msg)); >>>> } >>>> } >>>> >>>> private static int hashcode(byte[] bytearray) { >>>> if (bytearray == null) { >>>> return 0; >>>> } >>>> int hash = 39; >>>> for (int i = 0; i < bytearray.length; i++) { >>>> byte b = bytearray[i]; >>>> hash = hash * 7 + (int) b; >>>> } >>>> return hash; >>>> } >>>> >>>> public static void main(String args[]) throws MPIException { >>>> log("start main"); >>>> MPI.Init(args); >>>> try { >>>> log("initialized done"); >>>> byte[] saveMem = new byte[100000000]; >>>> MPI.COMM_WORLD.barrier(); >>>> Random r = new Random(); >>>> r.nextBytes(saveMem); >>>> if (MPI.COMM_WORLD.getRank() == 0) { >>>> for (int i = 0; i < 1000; i++) { >>>> saveMem[r.nextInt(saveMem.length)]++; >>>> log("i = " + i); >>>> int[] lengthData = new int[]{saveMem.length}; >>>> log("object hash = " + hashcode(saveMem)); >>>> log("length = " + lengthData[0]); >>>> MPI.COMM_WORLD.bcast(lengthData, 1, MPI.INT, 0); >>>> log("bcast length done (length = " + lengthData[0] >>>> + ")"); >>>> MPI.COMM_WORLD.barrier(); >>>> MPI.COMM_WORLD.bcast(saveMem, lengthData[0], >>>> MPI.BYTE, 0); >>>> log("bcast data done"); >>>> MPI.COMM_WORLD.barrier(); >>>> } >>>> MPI.COMM_WORLD.bcast(new int[]{0}, 1, MPI.INT, 0); >>>> } else { >>>> while (true) { >>>> int[] lengthData = new int[1]; >>>> MPI.COMM_WORLD.bcast(lengthData, 1, MPI.INT, 0); >>>> log("bcast length done (length = " + lengthData[0] >>>> + ")"); >>>> if (lengthData[0] == 0) { >>>> break; >>>> } >>>> MPI.COMM_WORLD.barrier(); >>>> saveMem = new byte[lengthData[0]]; >>>> MPI.COMM_WORLD.bcast(saveMem, saveMem.length, >>>> MPI.BYTE, 0); >>>> log("bcast data done"); >>>> MPI.COMM_WORLD.barrier(); >>>> log("object hash = " + hashcode(saveMem)); >>>> } >>>> } >>>> MPI.COMM_WORLD.barrier(); >>>> } catch (MPIException ex) { >>>> System.out.println("caugth error." + ex); >>>> log(ex.getMessage()); >>>> } catch (RuntimeException ex) { >>>> System.out.println("caugth error." + ex); >>>> log(ex.getMessage()); >>>> } finally { >>>> MPI.Finalize(); >>>> } >>>> >>>> } >>>> >>>> } >>>> >>>> >>>> ############ The Error (if it does not just hang up): >>>> >>>> # >>>> # A fatal error has been detected by the Java Runtime Environment: >>>> # >>>> # SIGSEGV (0xb) at pc=0x00002b7e9c86e3a1, pid=1172, tid=47822674495232 >>>> # >>>> # >>>> # A fatal error has been detected by the Java Runtime Environment: >>>> # JRE version: 7.0_25-b15 >>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode >>>> linux-amd64 compressed oops) >>>> # Problematic frame: >>>> # # >>>> # SIGSEGV (0xb) at pc=0x00002af69c0693a1, pid=1173, tid=47238546896640 >>>> # >>>> # JRE version: 7.0_25-b15 >>>> J de.uros.citlab.executor.test.TestSendBigFiles.hashcode([B)I >>>> # >>>> # Failed to write core dump. Core dumps have been disabled. To enable >>>> core dumping, try "ulimit -c unlimited" before starting Java again >>>> # >>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode >>>> linux-amd64 compressed oops) >>>> # Problematic frame: >>>> # J de.uros.citlab.executor.test.TestSendBigFiles.hashcode([B)I >>>> # >>>> # Failed to write core dump. Core dumps have been disabled. To enable >>>> core dumping, try "ulimit -c unlimited" before starting Java again >>>> # >>>> # An error report file with more information is saved as: >>>> # /home/gl069/ompi/bin/executor/hs_err_pid1172.log >>>> # An error report file with more information is saved as: >>>> # /home/gl069/ompi/bin/executor/hs_err_pid1173.log >>>> # >>>> # If you would like to submit a bug report, please visit: >>>> # http://bugreport.sun.com/bugreport/crash.jsp >>>> # >>>> # >>>> # If you would like to submit a bug report, please visit: >>>> # http://bugreport.sun.com/bugreport/crash.jsp >>>> # >>>> [titan01:01172] *** Process received signal *** >>>> [titan01:01172] Signal: Aborted (6) >>>> [titan01:01172] Signal code: (-6) >>>> [titan01:01173] *** Process received signal *** >>>> [titan01:01173] Signal: Aborted (6) >>>> [titan01:01173] Signal code: (-6) >>>> [titan01:01172] [ 0] /usr/lib64/libpthread.so.0(+0xf100)[0x2b7e9596a100] >>>> [titan01:01172] [ 1] /usr/lib64/libc.so.6(gsignal+0x37)[0x2b7e95fc75f7] >>>> [titan01:01172] [ 2] /usr/lib64/libc.so.6(abort+0x148)[0x2b7e95fc8ce8] >>>> [titan01:01172] [ 3] >>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x742ac5)[0x2b7e96a95ac5] >>>> [titan01:01172] [ 4] >>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x8a2137)[0x2b7e96bf5137] >>>> [titan01:01172] [ 5] >>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x140)[0x2b7e96a995e0] >>>> [titan01:01172] [ 6] [titan01:01173] [ 0] >>>> /usr/lib64/libpthread.so.0(+0xf100)[0x2af694ded100] >>>> [titan01:01173] [ 1] /usr/lib64/libc.so.6(+0x35670)[0x2b7e95fc7670] >>>> [titan01:01172] [ 7] [0x2b7e9c86e3a1] >>>> [titan01:01172] *** End of error message *** >>>> /usr/lib64/libc.so.6(gsignal+0x37)[0x2af69544a5f7] >>>> [titan01:01173] [ 2] /usr/lib64/libc.so.6(abort+0x148)[0x2af69544bce8] >>>> [titan01:01173] [ 3] >>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x742ac5)[0x2af695f18ac5] >>>> [titan01:01173] [ 4] >>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(+0x8a2137)[0x2af696078137] >>>> [titan01:01173] [ 5] >>>> /home/gl069/bin/jdk1.7.0_25/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x140)[0x2af695f1c5e0] >>>> [titan01:01173] [ 6] /usr/lib64/libc.so.6(+0x35670)[0x2af69544a670] >>>> [titan01:01173] [ 7] [0x2af69c0693a1] >>>> [titan01:01173] *** End of error message *** >>>> ------------------------------------------------------- >>>> Primary job terminated normally, but 1 process returned >>>> a non-zero exit code. Per user-direction, the job has been aborted. >>>> ------------------------------------------------------- >>>> >>>> -------------------------------------------------------------------------- >>>> mpirun noticed that process rank 1 with PID 0 on node titan01 exited on >>>> signal 6 (Aborted). >>>> >>>> >>>> ########CONFIGURATION: >>>> I used the ompi master sources from github: >>>> commit 267821f0dd405b5f4370017a287d9a49f92e734a >>>> Author: Gilles Gouaillardet <gil...@rist.or.jp> >>>> Date: Tue Jul 5 13:47:50 2016 +0900 >>>> >>>> ./configure --enable-mpi-java >>>> --with-jdk-dir=/home/gl069/bin/jdk1.7.0_25 --disable-dlopen >>>> --disable-mca-dso >>>> >>>> Thanks a lot for your help! >>>> Gundram >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users >>>> Link to this post: >>>> <http://www.open-mpi.org/community/lists/users/2016/07/29584.php> >>>> http://www.open-mpi.org/community/lists/users/2016/07/29584.php >>>> >>> >>> >>> >>> _______________________________________________ >>> users mailing listus...@open-mpi.org >>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/07/29585.php >>> >>> >>> >> >> _______________________________________________ >> users mailing listus...@open-mpi.org >> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/07/29587.php >> >> >> > > _______________________________________________ > users mailing listus...@open-mpi.org > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/07/29589.php > > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/07/29590.php > -- Saliya Ekanayake Ph.D. Candidate | Research Assistant School of Informatics and Computing | Digital Science Center Indiana University, Bloomington
