On May 2, 2011, at 7:56 AM, jody wrote: > Hi Ralph > > Thank You for doing the fix. > > Do you perhaps also have an idea what is going on when i try to start > xterm (or probably an other X application) on a remote host? > In this case it is not enough to specify the '--leave-session-attached' > option. > > These calls won't open any xterms > mpirun -np 4 -host squid_0 -mca plm_rsh_agent "ssh -Y" -mca > plm_base_verbose 1 xterm > mpirun -np 4 -host squid_0 -mca plm_rsh_agent "ssh -Y" > --leave-session-attached xterm > mpirun -np 4 -host squid_0 -mca plm_rsh_agent "ssh -Y" -mca > odls_base_verbose 5 xterm > mpirun -np 4 -host squid_0 -mca plm_rsh_agent "ssh -Y" -mca > odls_base_verbose 5 --leave-session-attached xterm > > But this will open the xterms: > mpirun -np 4 -host squid_0 -mca plm_rsh_agent "ssh -Y" -mca > plm_base_verbose 1 --leave-session-attached xterm > > Any verbosity level > 0 will open xterms, but with ' -mca > plm_base_verbose 0' there are again no xterms. >
No earthly idea...this seems to contradict what you had below. You said you were seeing the xterms with this cmd line: >>> I just found that everything works as expected if i use the the >>> '--leave-session-attached' option (without the debug options): >>> jody@chefli ~/share/neander $ mpirun -np 4 -host squid_0 -mca >>> plm_rsh_agent "ssh -Y" --leave-session-attached --xterm 0,1,2,3! >>> ./HelloMPI >>> The xterms are also opened if i do not use the '!' hold option. >> Did I miss something? > Thank You > Jody > > On Mon, May 2, 2011 at 2:29 PM, Ralph Castain <r...@open-mpi.org> wrote: >> >> On May 2, 2011, at 2:34 AM, jody wrote: >> >>> Hi Ralph >>> >>> I rebuilt open MPI 1.4.2 with the debug option on both chefli and squid_0. >>> The results are interesting! >>> >>> I wrote a small HelloMPI app which basically calls usleep for a pause >>> of 5 seconds. >>> >>> Now calling it as i did before, no MPI errors appear anymore, only the >>> display problems: >>> jody@chefli ~/share/neander $ mpirun -np 1 -host squid_0 -mca >>> plm_rsh_agent "ssh -Y" --xterm 0 ./HelloMPI >>> /usr/bin/xterm Xt error: Can't open display: localhost:10.0 >>> >>> When i do the same call *with* the debug option, the xterm appears and >>> shows the output of HelloMPI! >>> I attach the output in ompidbg_1.txt (It also works if i call with >>> '-np 4' and '--xterm 0,1,2,3' >> >> Good! >> >>> >>> Calling hostname the same way does not open an xterm (cf. ompidbg_2.txt). >>> >>> If i use the hold-option, the xterm appears with the output of >>> 'hostrname' (cf. ompidbg_3.txt) >>> The xterm opens after the line "launch complete for job..." has been >>> written (line 59) >> >> Okay, that's also expected. Like I said, without the "hold", the output is >> generated so quickly that the window just flashes at best. I've had similar >> experiences - hence the "hold" option. >> >>> >>> I just found that everything works as expected if i use the the >>> '--leave-session-attached' option (without the debug options): >>> jody@chefli ~/share/neander $ mpirun -np 4 -host squid_0 -mca >>> plm_rsh_agent "ssh -Y" --leave-session-attached --xterm 0,1,2,3! >>> ./HelloMPI >>> The xterms are also opened if i do not use the '!' hold option. >> >> Okay, I can understand why. The --leave-session-attached option just tells >> mpirun to not daemonize the backend daemons - thus leaving the ssh session >> alive. The debug options do the same thing, but turn on all the debug output. >> >> The problem is that if you don't leave the ssh session alive, then the xterm >> has no way back to your screen. By daemonizing, we severe that connection. >> >> What I should do (and maybe used to do, but it got removed) is automatically >> turn "on" the leave-session-attached option if you give --xterm. I can enter >> that patch. >> >> Note that this does limit the size of the launch to the number of ssh >> sessions the system allows you to have open at the same time. We default to >> a limit of 128 nodes, which is likely adequate for an xterm-based debugging >> session. However, you can increase it using an mca param (see ompi_info) to >> as high as the system allows. >> >> Thanks for helping debug this! I'll add you to the patch list so you can >> track it. >> >>> >>> What does *not* work is >>> jody@aim-triops ~/share/neander $ mpirun -np 2 -host squid_0 -mca >>> plm_rsh_agent "ssh -Y" --leave-session-attached xterm >>> xterm Xt error: Can't open display: >>> xterm: DISPLAY is not set >>> xterm Xt error: Can't open display: >>> xterm: DISPLAY is not set >>> >>> But then again, this call works (i.e. an xterm is opened) if all the >>> debug-options are used (ompidbg_4.txt). >>> Here the '--leave-session-attached' is necessary - without it, no xterm. >>> >>>> From these results i would say that there is no basic mishandling of >>> 'ssh', though i have no idea >>> what internal differences the use of the '-leave-session-attached' >>> option or the debug options make. >>> >>> I hope these observations are helpful >>> Jody >>> >>> >>> On Fri, Apr 29, 2011 at 12:08 AM, jody <jody....@gmail.com> wrote: >>>> Hi Ralph >>>> >>>> Thank you for your suggestions. >>>> I'll be happy to help you. >>>> I'm not sure if i'll get around to this tomorrow, >>>> but i certainly will do so on Monday. >>>> >>>> Thanks >>>> Jody >>>> >>>> On Thu, Apr 28, 2011 at 11:53 PM, Ralph Castain <r...@open-mpi.org> wrote: >>>>> Hi Jody >>>>> >>>>> I'm not sure when I'll get a chance to work on this - got a deadline to >>>>> meet. I do have a couple of suggestions, if you wouldn't mind helping >>>>> debug the problem? >>>>> >>>>> It looks to me like the problem is that mpirun is crashing or terminating >>>>> early for some reason - hence the failures to send msgs to it, and the >>>>> "lifeline lost" error that leads to the termination of the daemon. If you >>>>> build a debug version of the code (i.e., --enable-debug on configure), >>>>> you can get a lot of debug info that traces the behavior. >>>>> >>>>> If you could then run your program with >>>>> >>>>> -mca plm_base_verbose 5 -mca odls_base_verbose 5 --leave-session-attached >>>>> >>>>> and send it to me, we'll see what ORTE thinks it is doing. >>>>> >>>>> You could also take a look at the code for implementing the xterm option. >>>>> You'll find it in >>>>> >>>>> orte/mca/odls/base/odls_base_default_fns.c >>>>> >>>>> around line 1115. The xterm command syntax is defined in >>>>> >>>>> orte/mca/odls/base/odls_base_open.c >>>>> >>>>> around line 233 and following. Note that we use "xterm -T" as the cmd. >>>>> Perhaps you can spot an error in the way we treat xterm? >>>>> >>>>> Also, remember that you have to specify that you want us to "hold" the >>>>> xterm window open even after the process terminates. If you don't specify >>>>> it, the window automatically closes upon completion of the process. So a >>>>> fast-running cmd like "hostname" might disappear so quickly that it >>>>> causes a race condition problem. >>>>> >>>>> You might want to try a spinner application - i.e.., output something and >>>>> then sit in a loop or sleep for some period of time. Or, use the "hold" >>>>> option to keep the window open - you designate "hold" by putting a '!' >>>>> before the rank, e.g., "mpirun -np 2 -xterm \!2 hostname" >>>>> >>>>> >>>>> On Apr 28, 2011, at 8:38 AM, jody wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> Unfortunately this does not solve my problem. >>>>>> While i can do >>>>>> ssh -Y squid_0 xterm >>>>>> and this will open an xterm on m,y machiine (chefli), >>>>>> i run into problems with the -xterm option of openmpi: >>>>>> >>>>>> jody@chefli ~/share/neander $ mpirun -np 4 -mca plm_rsh_agent "ssh >>>>>> -Y" -host squid_0 --xterm 1 hostname >>>>>> squid_0 >>>>>> [squid_0:28046] [[35219,0],1]->[[35219,0],0] >>>>>> mca_oob_tcp_msg_send_handler: writev failed: Bad file descriptor (9) >>>>>> [sd = 8] >>>>>> [squid_0:28046] [[35219,0],1] routed:binomial: Connection to >>>>>> lifeline [[35219,0],0] lost >>>>>> [squid_0:28046] [[35219,0],1]->[[35219,0],0] >>>>>> mca_oob_tcp_msg_send_handler: writev failed: Bad file descriptor (9) >>>>>> [sd = 8] >>>>>> [squid_0:28046] [[35219,0],1] routed:binomial: Connection to >>>>>> lifeline [[35219,0],0] lost >>>>>> /usr/bin/xterm Xt error: Can't open display: localhost:11.0 >>>>>> >>>>>> By the way when i look at the DISPLAY variable in the xterm window >>>>>> opened via squid_0, >>>>>> i also have the display variable "localhost:11.0" >>>>>> >>>>>> Actually, the difference with using the "-mca plm_rsh_agent" is that >>>>>> the lines wiht the warnings about "xauth" and "untrusted X" do not >>>>>> appear: >>>>>> >>>>>> jody@chefli ~/share/neander $ mpirun -np 4 -host squid_0 -xterm 1 >>>>>> hostname >>>>>> Warning: untrusted X11 forwarding setup failed: xauth key data not >>>>>> generated >>>>>> Warning: No xauth data; using fake authentication data for X11 >>>>>> forwarding. >>>>>> squid_0 >>>>>> [squid_0:28337] [[34926,0],1]->[[34926,0],0] >>>>>> mca_oob_tcp_msg_send_handler: writev failed: Bad file descriptor (9) >>>>>> [sd = 8] >>>>>> [squid_0:28337] [[34926,0],1] routed:binomial: Connection to >>>>>> lifeline [[34926,0],0] lost >>>>>> [squid_0:28337] [[34926,0],1]->[[34926,0],0] >>>>>> mca_oob_tcp_msg_send_handler: writev failed: Bad file descriptor (9) >>>>>> [sd = 8] >>>>>> [squid_0:28337] [[34926,0],1] routed:binomial: Connection to >>>>>> lifeline [[34926,0],0] lost >>>>>> /usr/bin/xterm Xt error: Can't open display: localhost:11.0 >>>>>> >>>>>> >>>>>> I have doubts that the "-Y" is passed correctly: >>>>>> jody@triops ~/share/neander $ mpirun -np -mca plm_rsh_agent "ssh >>>>>> -Y" -host squid_0 xterm >>>>>> xterm Xt error: Can't open display: >>>>>> xterm: DISPLAY is not set >>>>>> xterm Xt error: Can't open display: >>>>>> xterm: DISPLAY is not set >>>>>> >>>>>> >>>>>> ---> as a matter of fact i noticed that the xterm option doesn't work >>>>>> locally: >>>>>> mpirun -np 4 -xterm 1 /usr/bin/printenv >>>>>> prints verything onto the console. >>>>>> >>>>>> Do you have any other suggestions i could try? >>>>>> >>>>>> Thank You >>>>>> Jody >>>>>> >>>>>> On Thu, Apr 28, 2011 at 3:06 PM, Ralph Castain <r...@open-mpi.org> wrote: >>>>>>> Should be able to just set >>>>>>> >>>>>>> -mca plm_rsh_agent "ssh -Y" >>>>>>> >>>>>>> on your cmd line, I believe >>>>>>> >>>>>>> On Apr 28, 2011, at 12:53 AM, jody wrote: >>>>>>> >>>>>>>> Hi Ralph >>>>>>>> >>>>>>>> Is there an easy way i could modify the OpenMPI code so that it would >>>>>>>> use >>>>>>>> the -Y option for ssh when connecting to remote machines? >>>>>>>> >>>>>>>> Thank You >>>>>>>> Jody >>>>>>>> >>>>>>>> On Thu, Apr 7, 2011 at 4:01 PM, jody <jody....@gmail.com> wrote: >>>>>>>>> Hi Ralph >>>>>>>>> thank you for your suggestions. After some fiddling, i found that >>>>>>>>> after my >>>>>>>>> last update (gentoo) my sshd_config had been overwritten >>>>>>>>> (X11Forwarding was set to 'no'). >>>>>>>>> >>>>>>>>> After correcting that, i can now open remote terminals with 'ssh -Y' >>>>>>>>> and with 'ssh -X' >>>>>>>>> (but with '-X' is till get those xauth warnings) >>>>>>>>> >>>>>>>>> But the xterm option still doesn't work: >>>>>>>>> jody@chefli ~/share/neander $ mpirun -np 4 -host squid_0 -xterm 1,2 >>>>>>>>> printenv | grep WORLD_RANK >>>>>>>>> Warning: untrusted X11 forwarding setup failed: xauth key data not >>>>>>>>> generated >>>>>>>>> Warning: No xauth data; using fake authentication data for X11 >>>>>>>>> forwarding. >>>>>>>>> /usr/bin/xterm Xt error: Can't open display: localhost:11.0 >>>>>>>>> /usr/bin/xterm Xt error: Can't open display: localhost:11.0 >>>>>>>>> OMPI_COMM_WORLD_RANK=0 >>>>>>>>> [aim-squid_0:09856] [[54132,0],1]->[[54132,0],0] >>>>>>>>> mca_oob_tcp_msg_send_handler: writev failed: Bad file descriptor (9) >>>>>>>>> [sd = 8] >>>>>>>>> [aim-squid_0:09856] [[54132,0],1] routed:binomial: Connection to >>>>>>>>> lifeline [[54132,0],0] lost >>>>>>>>> >>>>>>>>> So it looks like the two processes from squid_0 can't open the >>>>>>>>> display this way, >>>>>>>>> but one of them writes the output to the console... >>>>>>>>> Surprisingly, they are trying 'localhost:11.0' whereas when i use >>>>>>>>> 'ssh -Y' the >>>>>>>>> DISPLAY variable is set to 'localhost:10.0' >>>>>>>>> >>>>>>>>> So in what way would OMPI have to be adapted, so -xterm would work? >>>>>>>>> >>>>>>>>> Thank You >>>>>>>>> Jody >>>>>>>>> >>>>>>>>> On Wed, Apr 6, 2011 at 8:32 PM, Ralph Castain <r...@open-mpi.org> >>>>>>>>> wrote: >>>>>>>>>> Here's a little more info - it's for Cygwin, but I don't see anything >>>>>>>>>> Cygwin-specific in the answers: >>>>>>>>>> http://x.cygwin.com/docs/faq/cygwin-x-faq.html#q-ssh-no-x11forwarding >>>>>>>>>> >>>>>>>>>> On Apr 6, 2011, at 12:30 PM, Ralph Castain wrote: >>>>>>>>>> >>>>>>>>>> Sorry Jody - I should have read your note more carefully to see that >>>>>>>>>> you >>>>>>>>>> already tried -Y. :-( >>>>>>>>>> Not sure what to suggest... >>>>>>>>>> >>>>>>>>>> On Apr 6, 2011, at 12:29 PM, Ralph Castain wrote: >>>>>>>>>> >>>>>>>>>> Like I said, I'm not expert. However, a quick "google" of revealed >>>>>>>>>> this >>>>>>>>>> result: >>>>>>>>>> >>>>>>>>>> When trying to set up x11 forwarding over an ssh session to a remote >>>>>>>>>> server >>>>>>>>>> with the -X switch, I was getting an error like Warning: No xauth >>>>>>>>>> data; using fake authentication data for X11 forwarding. >>>>>>>>>> >>>>>>>>>> When doing something like: >>>>>>>>>> ssh -Xl root 10.1.1.9 to a remote server, the authentication worked, >>>>>>>>>> but I >>>>>>>>>> got an error message like: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> jason@badman ~/bin $ ssh -Xl root 10.1.1.9 >>>>>>>>>> Warning: untrusted X11 forwarding setup failed: xauth key data not >>>>>>>>>> generated >>>>>>>>>> Warning: No xauth data; using fake authentication data for X11 >>>>>>>>>> forwarding. >>>>>>>>>> Last login: Wed Apr 14 18:18:39 2010 from 10.1.1.5 >>>>>>>>>> [root@RHEL ~]# >>>>>>>>>> and any X programs I ran would not display on my local system.. >>>>>>>>>> >>>>>>>>>> Turns out the solution is to use the -Y switch instead. >>>>>>>>>> >>>>>>>>>> ssh -Yl root 10.1.1.9 >>>>>>>>>> >>>>>>>>>> and that worked fine. >>>>>>>>>> >>>>>>>>>> See if that works for you - if it does, we may have to modify OMPI to >>>>>>>>>> accommodate. >>>>>>>>>> >>>>>>>>>> On Apr 6, 2011, at 9:19 AM, jody wrote: >>>>>>>>>> >>>>>>>>>> Hi Ralph >>>>>>>>>> No, after the above error message mpirun has exited. >>>>>>>>>> >>>>>>>>>> But i also noticed that it is to ssh into squid_0 and open a xterm >>>>>>>>>> there: >>>>>>>>>> >>>>>>>>>> jody@chefli ~/share/neander $ ssh -Y squid_0 >>>>>>>>>> Last login: Wed Apr 6 17:14:02 CEST 2011 from chefli.uzh.ch on >>>>>>>>>> pts/0 >>>>>>>>>> jody@squid_0 ~ $ xterm >>>>>>>>>> xterm Xt error: Can't open display: >>>>>>>>>> xterm: DISPLAY is not set >>>>>>>>>> jody@squid_0 ~ $ export DISPLAY=130.60.126.74:0.0 >>>>>>>>>> jody@squid_0 ~ $ xterm >>>>>>>>>> xterm Xt error: Can't open display: 130.60.126.74:0.0 >>>>>>>>>> jody@squid_0 ~ $ export DISPLAY=chefli.uzh.ch:0.0 >>>>>>>>>> jody@squid_0 ~ $ xterm >>>>>>>>>> xterm Xt error: Can't open display: chefli.uzh.ch:0.0 >>>>>>>>>> jody@squid_0 ~ $ exit >>>>>>>>>> logout >>>>>>>>>> >>>>>>>>>> same thing with ssh -X, but here i get the same warning/error message >>>>>>>>>> as with mpirun: >>>>>>>>>> >>>>>>>>>> jody@chefli ~/share/neander $ ssh -X squid_0 >>>>>>>>>> Warning: untrusted X11 forwarding setup failed: xauth key data not >>>>>>>>>> generated >>>>>>>>>> Warning: No xauth data; using fake authentication data for X11 >>>>>>>>>> forwarding. >>>>>>>>>> Last login: Wed Apr 6 17:12:31 CEST 2011 from chefli.uzh.ch on ssh >>>>>>>>>> >>>>>>>>>> So perhaps the whole problem is linked to that xauth-thing. >>>>>>>>>> Do you have a suggestion how this can be solved? >>>>>>>>>> >>>>>>>>>> Thank You >>>>>>>>>> Jody >>>>>>>>>> >>>>>>>>>> On Wed, Apr 6, 2011 at 4:41 PM, Ralph Castain <r...@open-mpi.org> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> If I read your error messages correctly, it looks like mpirun is >>>>>>>>>> crashing - >>>>>>>>>> the daemon is complaining that it lost the socket connection back to >>>>>>>>>> mpirun, >>>>>>>>>> and hence will abort. >>>>>>>>>> >>>>>>>>>> Are you seeing mpirun still alive? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Apr 5, 2011, at 4:46 AM, jody wrote: >>>>>>>>>> >>>>>>>>>> Hi >>>>>>>>>> >>>>>>>>>> On my workstation and the cluster i set up OpenMPI (v 1.4.2) so that >>>>>>>>>> >>>>>>>>>> it works in "text-mode": >>>>>>>>>> >>>>>>>>>> $ mpirun -np 4 -x DISPLAY -host squid_0 printenv | grep >>>>>>>>>> WORLD_RANK >>>>>>>>>> >>>>>>>>>> OMPI_COMM_WORLD_RANK=0 >>>>>>>>>> >>>>>>>>>> OMPI_COMM_WORLD_RANK=1 >>>>>>>>>> >>>>>>>>>> OMPI_COMM_WORLD_RANK=2 >>>>>>>>>> >>>>>>>>>> OMPI_COMM_WORLD_RANK=3 >>>>>>>>>> >>>>>>>>>> but when i use the -xterm option to mpirun, it doesn't work >>>>>>>>>> >>>>>>>>>> $ mpirun -np 4 -x DISPLAY -host squid_0 -xterm 1,2 printenv | grep >>>>>>>>>> WORLD_RANK >>>>>>>>>> >>>>>>>>>> Warning: untrusted X11 forwarding setup failed: xauth key data not >>>>>>>>>> generated >>>>>>>>>> >>>>>>>>>> Warning: No xauth data; using fake authentication data for X11 >>>>>>>>>> forwarding. >>>>>>>>>> >>>>>>>>>> OMPI_COMM_WORLD_RANK=0 >>>>>>>>>> >>>>>>>>>> [squid_0:05266] [[55607,0],1]->[[55607,0],0] >>>>>>>>>> >>>>>>>>>> mca_oob_tcp_msg_send_handler: writev failed: Bad file descriptor (9) >>>>>>>>>> >>>>>>>>>> [sd = 8] >>>>>>>>>> >>>>>>>>>> [squid_0:05266] [[55607,0],1] routed:binomial: Connection to >>>>>>>>>> >>>>>>>>>> lifeline [[55607,0],0] lost >>>>>>>>>> >>>>>>>>>> /usr/bin/xterm Xt error: Can't open display: chefli.uzh.ch:0.0 >>>>>>>>>> >>>>>>>>>> /usr/bin/xterm Xt error: Can't open display: chefli.uzh.ch:0.0 >>>>>>>>>> >>>>>>>>>> (strange: somebody wrote his message to the console) >>>>>>>>>> >>>>>>>>>> No matter whether i set the DISPLAY variable to the full hostname of >>>>>>>>>> >>>>>>>>>> the workstation, >>>>>>>>>> >>>>>>>>>> to the IP-Adress of the workstation or simply to ":0.0", it doesn't >>>>>>>>>> work >>>>>>>>>> >>>>>>>>>> But i do have xauth data (as far as i know): >>>>>>>>>> >>>>>>>>>> On the remote (squid_0): >>>>>>>>>> >>>>>>>>>> jody@squid_0 ~ $ xauth list >>>>>>>>>> >>>>>>>>>> chefli/unix:10 MIT-MAGIC-COOKIE-1 5293e179bc7b2036d87cbcdf14891d0c >>>>>>>>>> >>>>>>>>>> chefli/unix:0 MIT-MAGIC-COOKIE-1 146c7f438fab79deb8a8a7df242b6f4b >>>>>>>>>> >>>>>>>>>> chefli.uzh.ch:0 MIT-MAGIC-COOKIE-1 >>>>>>>>>> 146c7f438fab79deb8a8a7df242b6f4b >>>>>>>>>> >>>>>>>>>> on the workstation: >>>>>>>>>> >>>>>>>>>> $ xauth list >>>>>>>>>> >>>>>>>>>> chefli/unix:10 MIT-MAGIC-COOKIE-1 5293e179bc7b2036d87cbcdf14891d0c >>>>>>>>>> >>>>>>>>>> chefli/unix:0 MIT-MAGIC-COOKIE-1 146c7f438fab79deb8a8a7df242b6f4b >>>>>>>>>> >>>>>>>>>> localhost.localdomain/unix:0 MIT-MAGIC-COOKIE-1 >>>>>>>>>> >>>>>>>>>> 146c7f438fab79deb8a8a7df242b6f4b >>>>>>>>>> >>>>>>>>>> chefli.uzh.ch/unix:0 MIT-MAGIC-COOKIE-1 >>>>>>>>>> 146c7f438fab79deb8a8a7df242b6f4b >>>>>>>>>> >>>>>>>>>> In sshd_config on the workstation i have 'X11Forwarding yes' >>>>>>>>>> >>>>>>>>>> I have also done >>>>>>>>>> >>>>>>>>>> xhost + squid_0 >>>>>>>>>> >>>>>>>>>> on the workstation. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> How can i get the -xterm option running? >>>>>>>>>> >>>>>>>>>> Thank You >>>>>>>>>> >>>>>>>>>> Jody >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> >>>>>>>>>> users mailing list >>>>>>>>>> >>>>>>>>>> us...@open-mpi.org >>>>>>>>>> >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> >>>>>>>>>> users mailing list >>>>>>>>>> >>>>>>>>>> us...@open-mpi.org >>>>>>>>>> >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> users mailing list >>>>>>>>>> us...@open-mpi.org >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> users mailing list >>>>>>>>>> us...@open-mpi.org >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> us...@open-mpi.org >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>> >>> <ompidbg_1.txt><ompidbg_2.txt><ompidbg_3.txt><ompidbg_4.txt>_______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
