Hi Ralph,
I tested it with the trunk r29228. I still have the following problem. Now, it
even spawns the daemon on the new node through torque but then suddently quits.
The following is the output. Can you please have a look?
Thanks
Suraj
[grsacc20:04511] [[6253,0],0] plm:base:receive processing msg
[grsacc20:04511] [[6253,0],0] plm:base:receive job launch command from
[[6253,1],0]
[grsacc20:04511] [[6253,0],0] plm:base:receive adding hosts
[grsacc20:04511] [[6253,0],0] plm:base:receive calling spawn
[grsacc20:04511] [[6253,0],0] plm:base:receive done processing commands
[grsacc20:04511] [[6253,0],0] plm:base:setup_job
[grsacc20:04511] [[6253,0],0] plm:base:setup_vm
[grsacc20:04511] [[6253,0],0] plm:base:setup_vm add new daemon [[6253,0],2]
[grsacc20:04511] [[6253,0],0] plm:base:setup_vm assigning new daemon
[[6253,0],2] to node grsacc18
[grsacc20:04511] [[6253,0],0] plm:tm: launching vm
[grsacc20:04511] [[6253,0],0] plm:tm: final top-level argv:
orted -mca ess tm -mca orte_ess_jobid 409796608 -mca orte_ess_vpid
<template> -mca orte_ess_num_procs 3 -mca orte_hnp_uri
"409796608.0;tcp://192.168.222.20:53097" -mca plm_base_verbose 6
[grsacc20:04511] [[6253,0],0] plm:tm: launching on node grsacc19
[grsacc20:04511] [[6253,0],0] plm:tm: executing:
orted -mca ess tm -mca orte_ess_jobid 409796608 -mca orte_ess_vpid 1
-mca orte_ess_num_procs 3 -mca orte_hnp_uri
"409796608.0;tcp://192.168.222.20:53097" -mca plm_base_verbose 6
[grsacc20:04511] [[6253,0],0] plm:tm: launching on node grsacc18
[grsacc20:04511] [[6253,0],0] plm:tm: executing:
orted -mca ess tm -mca orte_ess_jobid 409796608 -mca orte_ess_vpid 2
-mca orte_ess_num_procs 3 -mca orte_hnp_uri
"409796608.0;tcp://192.168.222.20:53097" -mca plm_base_verbose 6
[grsacc20:04511] [[6253,0],0] plm:tm:launch: finished spawning orteds
[grsacc19:28754] mca:base:select:( plm) Querying component [rsh]
[grsacc19:28754] [[6253,0],1] plm:rsh_lookup on agent ssh : rsh path NULL
[grsacc19:28754] mca:base:select:( plm) Query of component [rsh] set priority
to 10
[grsacc19:28754] mca:base:select:( plm) Selected component [rsh]
[grsacc19:28754] [[6253,0],1] plm:rsh_setup on agent ssh : rsh path NULL
[grsacc19:28754] [[6253,0],1] plm:base:receive start comm
[grsacc19:28754] [[6253,0],1] plm:base:receive stop comm
[grsacc18:16648] mca:base:select:( plm) Querying component [rsh]
[grsacc18:16648] [[6253,0],2] plm:rsh_lookup on agent ssh : rsh path NULL
[grsacc18:16648] mca:base:select:( plm) Query of component [rsh] set priority
to 10
[grsacc18:16648] mca:base:select:( plm) Selected component [rsh]
[grsacc18:16648] [[6253,0],2] plm:rsh_setup on agent ssh : rsh path NULL
[grsacc18:16648] [[6253,0],2] plm:base:receive start comm
[grsacc20:04511] [[6253,0],0] plm:base:orted_report_launch from daemon
[[6253,0],2]
[grsacc20:04511] [[6253,0],0] plm:base:orted_report_launch from daemon
[[6253,0],2] on node grsacc18
[grsacc20:04511] [[6253,0],0] plm:base:orted_report_launch completed for daemon
[[6253,0],2] at contact 409796608.2;tcp://192.168.222.18:47974
[grsacc20:04511] [[6253,0],0] plm:base:launch_apps for job [6253,2]
[grsacc20:04511] 1 more process has sent help message help-mpi-btl-openib.txt /
no active ports found
[grsacc20:04511] Set MCA parameter "orte_base_help_aggregate" to 0 to see all
help / error messages
[grsacc20:04511] 1 more process has sent help message help-mpi-btl-base.txt /
btl:no-nics
[grsacc20:04511] [[6253,0],0] plm:base:receive processing msg
[grsacc20:04511] [[6253,0],0] plm:base:receive update proc state command from
[[6253,0],2]
[grsacc20:04511] [[6253,0],0] plm:base:receive got update_proc_state for job
[6253,2]
[grsacc20:04511] [[6253,0],0] plm:base:receive got update_proc_state for vpid 0
state RUNNING exit_code 0
[grsacc20:04511] [[6253,0],0] plm:base:receive done processing commands
[grsacc20:04511] [[6253,0],0] plm:base:launch wiring up iof for job [6253,2]
[grsacc20:04511] [[6253,0],0] plm:base:receive processing msg
[grsacc20:04511] [[6253,0],0] plm:base:receive done processing commands
[grsacc20:04511] [[6253,0],0] plm:base:launch registered event
[grsacc20:04511] [[6253,0],0] plm:base:launch sending dyn release of job
[6253,2] to [[6253,1],0]
[grsacc20:04511] [[6253,0],0] plm:base:orted_cmd sending orted_exit commands
[grsacc19:28747] [[6253,0],1] plm:base:receive stop comm
[grsacc20:04511] [[6253,0],0] plm:base:receive stop comm
-bash-4.1$ [grsacc18:16648] [[6253,0],2] plm:base:receive stop comm
On Sep 23, 2013, at 1:55 AM, Ralph Castain wrote:
> Found a bug in the Torque support - we were trying to connect to the MOM
> again, which would hang (I imagine). I pushed a fix to the trunk (r29227) and
> scheduled it to come to 1.7.3 if you want to try it again.
>
>
> On Sep 22, 2013, at 4:21 PM, Suraj Prabhakaran <[email protected]>
> wrote:
>
>> Dear Ralph,
>>
>> This is the output I get when I execute with the verbose option.
>>
>> [grsacc20:21012] [[23526,0],0] plm:base:receive processing msg
>> [grsacc20:21012] [[23526,0],0] plm:base:receive job launch command from
>> [[23526,1],0]
>> [grsacc20:21012] [[23526,0],0] plm:base:receive adding hosts
>> [grsacc20:21012] [[23526,0],0] plm:base:receive calling spawn
>> [grsacc20:21012] [[23526,0],0] plm:base:receive done processing commands
>> [grsacc20:21012] [[23526,0],0] plm:base:setup_job
>> [grsacc20:21012] [[23526,0],0] plm:base:setup_vm
>> [grsacc20:21012] [[23526,0],0] plm:base:setup_vm add new daemon [[23526,0],2]
>> [grsacc20:21012] [[23526,0],0] plm:base:setup_vm assigning new daemon
>> [[23526,0],2] to node grsacc17/1-4
>> [grsacc20:21012] [[23526,0],0] plm:base:setup_vm add new daemon [[23526,0],3]
>> [grsacc20:21012] [[23526,0],0] plm:base:setup_vm assigning new daemon
>> [[23526,0],3] to node grsacc17/0-5
>> [grsacc20:21012] [[23526,0],0] plm:tm: launching vm
>> [grsacc20:21012] [[23526,0],0] plm:tm: final top-level argv:
>> orted -mca ess tm -mca orte_ess_jobid 1541799936 -mca orte_ess_vpid
>> <template> -mca orte_ess_num_procs 4 -mca orte_hnp_uri
>> "1541799936.0;tcp://192.168.222.20:49049" -mca plm_base_verbose 5
>> [warn] opal_libevent2021_event_base_loop: reentrant invocation. Only one
>> event_base_loop can run on each event_base at once.
>> [grsacc20:21012] [[23526,0],0] plm:base:orted_cmd sending orted_exit commands
>> [grsacc20:21012] [[23526,0],0] plm:base:receive stop comm
>>
>> Says something?
>>
>> Best,
>> Suraj
>>
>>
>>
>>
>> On Sep 22, 2013, at 9:45 PM, Ralph Castain wrote:
>>
>>> I'll still need to look at the intercomm_create issue, but I just tested
>>> both the trunk and current 1.7.3 branch for "add-host" and both worked just
>>> fine. This was on my little test cluster which only has rsh available - no
>>> Torque.
>>>
>>> You might add "-mca plm_base_verbose 5" to your cmd line to get some debug
>>> output as to the problem.
>>>
>>>
>>> On Sep 21, 2013, at 5:48 PM, Ralph Castain <[email protected]> wrote:
>>>
>>>>
>>>> On Sep 21, 2013, at 4:54 PM, Suraj Prabhakaran
>>>> <[email protected]> wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> Really thanks a lot for your efforts. I too downloaded the trunk to check
>>>>> if it works for my case and as of revision 29215, it works for the
>>>>> original case I reported. Although it works, I still see the following in
>>>>> the output. Does it mean anything?
>>>>> [grsacc17][[13611,1],0][btl_openib_proc.c:157:mca_btl_openib_proc_create]
>>>>> [btl_openib_proc.c:157] ompi_modex_recv failed for peer [[13611,2],0]
>>>>
>>>> Yes - it means we don't quite have this right yet :-(
>>>>
>>>>>
>>>>> However, on another topic relevant to my use case, I have another problem
>>>>> to report. I am having problems using the "add-host" info to the
>>>>> MPI_Comm_spawn() when MPI is compiled with support for Torque resource
>>>>> manager. This problem is totally new in the 1.7 series and it worked
>>>>> perfectly until 1.6.5
>>>>>
>>>>> Basically, I am working on implementing dynamic resource management
>>>>> facilities in the Torque/Maui batch system. Through a new tm call, an
>>>>> application can get new resources for a job.
>>>>
>>>> FWIW: you'll find that we added an API to the orte RAS framework to
>>>> support precisely that operation. It allows an application to request that
>>>> we dynamically obtain additional resources during execution (e.g., as part
>>>> of a Comm_spawn call via an info_key). We originally implemented this with
>>>> Slurm, but you could add the calls into the Torque component as well if
>>>> you like.
>>>>
>>>> This is in the trunk now - will come over to 1.7.4
>>>>
>>>>
>>>>> I want to use MPI_Comm_spawn() to spawn new processes in the new hosts.
>>>>> With my extended torque/maui batch system, I was able to perfectly use
>>>>> the "add-host" info argument to MPI_Comm_spawn() to spawn new processes
>>>>> on these hosts. Since MPI and Torque refer to the hosts through the
>>>>> nodeids, I made sure that OpenMPI uses the correct nodeid's for these new
>>>>> hosts.
>>>>> Until 1.6.5, this worked perfectly fine, except that due to the
>>>>> Intercomm_merge problem, I could not really run a real application to its
>>>>> completion.
>>>>>
>>>>> While this is now fixed in the trunk, I found that, however, when using
>>>>> the "add-host" info argument, everything collapses after printing out the
>>>>> following error.
>>>>>
>>>>> [warn] opal_libevent2021_event_base_loop: reentrant invocation. Only one
>>>>> event_base_loop can run on each event_base at once.
>>>>
>>>> I'll take a look - probably some stale code that hasn't been updated yet
>>>> for async ORTE operations
>>>>
>>>>>
>>>>> And due to this, I am still not really able to run my application! I also
>>>>> compiled the MPI without any Torque/PBS support and just used the
>>>>> "add-host" argument normally. Again, this worked perfectly in 1.6.5. But
>>>>> in the 1.7 series, it works but after printing out the following error.
>>>>>
>>>>> [grsacc17][[13731,1],0][btl_openib_proc.c:157:mca_btl_openib_proc_create]
>>>>> [btl_openib_proc.c:157] ompi_modex_recv failed for peer [[13731,2],0]
>>>>> [grsacc17][[13731,1],1][btl_openib_proc.c:157:mca_btl_openib_proc_create]
>>>>> [btl_openib_proc.c:157] ompi_modex_recv failed for peer [[13731,2],0]
>>>>
>>>> Yeah, the 1.7 series doesn't have the reentrant test in it - so we
>>>> "illegally" re-enter libevent. The error again means we don't have
>>>> Intercomm_create correct just yet.
>>>>
>>>> I'll see what I can do about this and get back to you
>>>>
>>>>>
>>>>> In short, with pbs/torque support, it fails and without pbs/torque
>>>>> support, it runs after spitting the above lines.
>>>>>
>>>>> I would really appreciate some help on this, since I need these features
>>>>> to actually test my case and (at least in my short experience) no other
>>>>> MPI implementation seem friendly to such dynamic scenarios.
>>>>>
>>>>> Thanks a lot!
>>>>>
>>>>> Best,
>>>>> Suraj
>>>>>
>>>>>
>>>>>
>>>>> On Sep 20, 2013, at 4:58 PM, Jeff Squyres (jsquyres) wrote:
>>>>>
>>>>>> Just to close my end of this loop: as of trunk r29213, it all works for
>>>>>> me. Thanks!
>>>>>>
>>>>>>
>>>>>> On Sep 18, 2013, at 12:52 PM, Ralph Castain <[email protected]> wrote:
>>>>>>
>>>>>>> Thanks George - much appreciated
>>>>>>>
>>>>>>> On Sep 18, 2013, at 9:49 AM, George Bosilca <[email protected]> wrote:
>>>>>>>
>>>>>>>> The test case was broken. I just pushed a fix.
>>>>>>>>
>>>>>>>> George.
>>>>>>>>
>>>>>>>> On Sep 18, 2013, at 16:49 , Ralph Castain <[email protected]> wrote:
>>>>>>>>
>>>>>>>>> Hangs with any np > 1
>>>>>>>>>
>>>>>>>>> However, I'm not sure if that's an issue with the test vs the
>>>>>>>>> underlying implementation
>>>>>>>>>
>>>>>>>>> On Sep 18, 2013, at 7:40 AM, "Jeff Squyres (jsquyres)"
>>>>>>>>> <[email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> Does it hang when you run with -np 4?
>>>>>>>>>>
>>>>>>>>>> Sent from my phone. No type good.
>>>>>>>>>>
>>>>>>>>>> On Sep 18, 2013, at 4:10 PM, "Ralph Castain" <[email protected]>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Strange - it works fine for me on my Mac. However, I see one
>>>>>>>>>>> difference - I only run it with np=1
>>>>>>>>>>>
>>>>>>>>>>> On Sep 18, 2013, at 2:22 AM, Jeff Squyres (jsquyres)
>>>>>>>>>>> <[email protected]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Sep 18, 2013, at 9:33 AM, George Bosilca <[email protected]>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> 1. sm doesn't work between spawned processes. So you must have
>>>>>>>>>>>>> another network enabled.
>>>>>>>>>>>>
>>>>>>>>>>>> I know :-). I have tcp available as well (OMPI will abort if you
>>>>>>>>>>>> only run with sm,self because the comm_spawn will fail with
>>>>>>>>>>>> unreachable errors -- I just tested/proved this to myself).
>>>>>>>>>>>>
>>>>>>>>>>>>> 2. Don't use the test case attached to my email, I left an xterm
>>>>>>>>>>>>> based spawn and the debugging. It can't work without xterm
>>>>>>>>>>>>> support. Instead try using the test case from the trunk, the one
>>>>>>>>>>>>> committed by Ralph.
>>>>>>>>>>>>
>>>>>>>>>>>> I didn't see any "xterm" strings in there, but ok. :-) I ran
>>>>>>>>>>>> with orte/test/mpi/intercomm_create.c, and that hangs for me as
>>>>>>>>>>>> well:
>>>>>>>>>>>>
>>>>>>>>>>>> -----
>>>>>>>>>>>> ❯❯❯ mpicc intercomm_create.c -o intercomm_create
>>>>>>>>>>>> ❯❯❯ mpirun -np 4 intercomm_create
>>>>>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201,
>>>>>>>>>>>> &inter) [rank 4]
>>>>>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201,
>>>>>>>>>>>> &inter) [rank 5]
>>>>>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201,
>>>>>>>>>>>> &inter) [rank 6]
>>>>>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201,
>>>>>>>>>>>> &inter) [rank 7]
>>>>>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201, &inter)
>>>>>>>>>>>> [rank 4]
>>>>>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201, &inter)
>>>>>>>>>>>> [rank 5]
>>>>>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201, &inter)
>>>>>>>>>>>> [rank 6]
>>>>>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201, &inter)
>>>>>>>>>>>> [rank 7]
>>>>>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter) (0)
>>>>>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter) (0)
>>>>>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter) (0)
>>>>>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter) (0)
>>>>>>>>>>>> [hang]
>>>>>>>>>>>> -----
>>>>>>>>>>>>
>>>>>>>>>>>> Similarly, on my Mac, it hangs with no output:
>>>>>>>>>>>>
>>>>>>>>>>>> -----
>>>>>>>>>>>> ❯❯❯ mpicc intercomm_create.c -o intercomm_create
>>>>>>>>>>>> ❯❯❯ mpirun -np 4 intercomm_create
>>>>>>>>>>>> [hang]
>>>>>>>>>>>> -----
>>>>>>>>>>>>
>>>>>>>>>>>>> George.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sep 18, 2013, at 07:53 , "Jeff Squyres (jsquyres)"
>>>>>>>>>>>>> <[email protected]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> George --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When I build the SVN trunk (r29201) on 64 bit linux, your
>>>>>>>>>>>>>> attached test case hangs:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -----
>>>>>>>>>>>>>> ❯❯❯ mpicc intercomm_create.c -o intercomm_create
>>>>>>>>>>>>>> ❯❯❯ mpirun -np 4 intercomm_create
>>>>>>>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201,
>>>>>>>>>>>>>> &inter) [rank 4]
>>>>>>>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201,
>>>>>>>>>>>>>> &inter) [rank 5]
>>>>>>>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201,
>>>>>>>>>>>>>> &inter) [rank 6]
>>>>>>>>>>>>>> b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201,
>>>>>>>>>>>>>> &inter) [rank 7]
>>>>>>>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter)
>>>>>>>>>>>>>> (0)
>>>>>>>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter)
>>>>>>>>>>>>>> (0)
>>>>>>>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter)
>>>>>>>>>>>>>> (0)
>>>>>>>>>>>>>> a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 0, 201, &inter)
>>>>>>>>>>>>>> (0)
>>>>>>>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201,
>>>>>>>>>>>>>> &inter) [rank 4]
>>>>>>>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201,
>>>>>>>>>>>>>> &inter) [rank 5]
>>>>>>>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201,
>>>>>>>>>>>>>> &inter) [rank 6]
>>>>>>>>>>>>>> c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201,
>>>>>>>>>>>>>> &inter) [rank 7]
>>>>>>>>>>>>>> [hang]
>>>>>>>>>>>>>> -----
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On my Mac, it hangs without printing anything:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -----
>>>>>>>>>>>>>> ❯❯❯ mpicc intercomm_create.c -o intercomm_create
>>>>>>>>>>>>>> ❯❯❯ mpirun -np 4 intercomm_create
>>>>>>>>>>>>>> [hang]
>>>>>>>>>>>>>> -----
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sep 18, 2013, at 1:48 AM, George Bosilca
>>>>>>>>>>>>>> <[email protected]> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Here is a quick (and definitively not the cleanest) patch that
>>>>>>>>>>>>>>> addresses the MPI_Intercomm issue at the MPI level. It should
>>>>>>>>>>>>>>> be applied after removal of 29166.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I also added the corrected test case stressing the corner cases
>>>>>>>>>>>>>>> by doing barriers at every inter-comm creation and doing a
>>>>>>>>>>>>>>> clean disconnect.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Jeff Squyres
>>>>>>>>>>>>>> [email protected]
>>>>>>>>>>>>>> For corporate legal information go to:
>>>>>>>>>>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> devel mailing list
>>>>>>>>>>>>>> [email protected]
>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> devel mailing list
>>>>>>>>>>>>> [email protected]
>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Jeff Squyres
>>>>>>>>>>>> [email protected]
>>>>>>>>>>>> For corporate legal information go to:
>>>>>>>>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> devel mailing list
>>>>>>>>>>>> [email protected]
>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> devel mailing list
>>>>>>>>>>> [email protected]
>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>>>>> _______________________________________________
>>>>>>>>>> devel mailing list
>>>>>>>>>> [email protected]
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> devel mailing list
>>>>>>>>> [email protected]
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> devel mailing list
>>>>>>>> [email protected]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> devel mailing list
>>>>>>> [email protected]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jeff Squyres
>>>>>> [email protected]
>>>>>> For corporate legal information go to:
>>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>>>>
>>>>>> _______________________________________________
>>>>>> devel mailing list
>>>>>> [email protected]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>
>>>>> _______________________________________________
>>>>> devel mailing list
>>>>> [email protected]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>
>>>
>>> _______________________________________________
>>> devel mailing list
>>> [email protected]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> _______________________________________________
>> devel mailing list
>> [email protected]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> [email protected]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
