Re: [OMPI users] Big jump from OFED 1.5.4.1 -> recent (stable). Any suggestions?

[Mehmet Belgin]

Hi Llolsten,

We are trying to keep up with the latest Open MPI as we can and keep a 
few old versions around (oldest is 1.6), with RHEL 6.5. The OFED upgrade 
will complement planned OS upgrades to the latest stable RHEL 6.x  
(probably 6.7, not sure if we will go with 6.8).


Did you have to recompile Open MPI stacks or any of the existing MPI 
software?


Thank you for your input!

-Memo



On 6/13/16 10:57 PM, Llolsten Kaonga wrote:


Hello Mehmet,

OFED is now around 3.18.2-rc2 and there is talk of an rc3.

We have used many different versions of OFED, and we are now running 
OFED 3.18.1 rc2 with the latest version of Open MPI with no trouble 
(OS is CentOS 7.2). What version of Open MPI are you planning to use? 
What OS, and version?


Good luck.

--

Llolsten

*From:*users [mailto:users-boun...@open-mpi.org] *On Behalf Of *Mehmet 
Belgin

*Sent:* Monday, June 13, 2016 7:05 PM
*To:* us...@open-mpi.org
*Subject:* [OMPI users] Big jump from OFED 1.5.4.1 -> recent (stable). 
Any suggestions?


Greetings!

We have not upgraded our OFED stack for a very long time, and still 
running on an ancient version (1.5.4.1, yeah we know). We are now 
considering a big jump from this version to a tested and stable recent 
version and would really appreciate any suggestions from the community.


In particular, is there a particular recent OFED version that's known 
to work well with Open MPI? Any versions we need to avoid? Do we need 
to recompile existing Open MPI stacks? How about existing scientific 
software that are compiled with it?


We will appreciate *any* suggestions/warnings (horror stories?) that 
you won't mind sharing.


Thank you very much in advance!

-Mehmet

--

=
Mehmet Belgin, Ph.D. (mehmet.bel...@oit.gatech.edu 
)
Scientific Computing Consultant | OIT - Academic and Research Technologies
Georgia Institute of Technology
258 4th Str NW, Rich Building, Room 326
Atlanta, GA  30332-0700
Office: (404) 385-0665


___
users mailing list
us...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2016/06/29429.php


--
=
Mehmet Belgin, Ph.D. (mehmet.bel...@oit.gatech.edu)
Scientific Computing Consultant | OIT - Academic and Research Technologies
Georgia Institute of Technology
258 4th St NW, Rich Building, Room 326
Atlanta, GA  30332-0700
Office: (404) 385-0665

Re: [OMPI users] runtime error in orte/loop_spawn test using OMPI 1.10.2

[Jason Maldonis]

Thanks Ralph for all the help. I will do that until it gets fixed.

Nathan, I am very very interested in this working because we are developing
some new cool code for research in materials science. This is the last
piece of the puzzle for us I believe. I can use TCP for now though of
course. While I doubt I can help, if you are having trouble reproducing the
problem or something else, feel free to let me know. I understand you
probably have a bunch of other things on your plate too, but if there is
something I can do to speed up the process, just let me know.

Lastly, what are the chances there is a place/website/etc where I can watch
to see when the fix for this has been made?

Thanks everyone!
Jason

Jason Maldonis
Research Assistant of Professor Paul Voyles
Materials Science Grad Student
University of Wisconsin, Madison
1509 University Ave, Rm M142
Madison, WI 53706
maldo...@wisc.edu
608-295-5532

On Tue, Jun 14, 2016 at 4:51 PM, Ralph Castain  wrote:

> You don’t want to always use those options as your performance will take a
> hit - TCP vs Infiniband isn’t a good option. Sadly, this is something we
> need someone like Nathan to address as it is a bug in the code base, and in
> an area I’m not familiar with
>
> For now, just use TCP so you can move forward
>
>
> On Jun 14, 2016, at 2:14 PM, Jason Maldonis  wrote:
>
> Ralph, The problem *does* go away if I add "-mca btl tcp,sm,self" to the
> mpiexec cmd line. (By the way, I am using mpiexec rather than mpirun; do
> you recommend one over the other?) Will you tell me what this means for me?
> For example, should I always append these arguments to mpiexec for my
> non-test jobs as well?   I do not know what you mean by fabric
> unfortunately, but I can give you some system information (see end of
> email). Unfortunately I am not a system admin so I do not have sudo rights.
> Just let me know if I can tell you something more specific though and I
> will get it.
>
> Nathan,  Thank you for your response. Unfortunately I have no idea what
> that means :(  I can forward that to our cluster managers, but I do not
> know if that is enough information for them to understand what they might
> need to do to help me with this issue.
>
> $ lscpu
> Architecture:  x86_64
> CPU op-mode(s):32-bit, 64-bit
> Byte Order:Little Endian
> CPU(s):20
> On-line CPU(s) list:   0-19
> Thread(s) per core:1
> Core(s) per socket:10
> Socket(s): 2
> NUMA node(s):  2
> Vendor ID: GenuineIntel
> CPU family:6
> Model: 63
> Stepping:  2
> CPU MHz:   2594.159
> BogoMIPS:  5187.59
> Virtualization:VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache:  256K
> L3 cache:  25600K
> NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18
> NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19
>
> Thanks,
> Jason
>
> Jason Maldonis
> Research Assistant of Professor Paul Voyles
> Materials Science Grad Student
> University of Wisconsin, Madison
> 1509 University Ave, Rm M142
> Madison, WI 53706
> maldo...@wisc.edu
> 608-295-5532
>
> On Tue, Jun 14, 2016 at 1:27 PM, Nathan Hjelm  wrote:
>
>> That message is coming from udcm in the openib btl. It indicates some
>> sort of failure in the connection mechanism. It can happen if the listening
>> thread no longer exists or is taking too long to process messages.
>>
>> -Nathan
>>
>>
>> On Jun 14, 2016, at 12:20 PM, Ralph Castain  wrote:
>>
>> Hmm…I’m unable to replicate a problem on my machines. What fabric are you
>> using? Does the problem go away if you add “-mca btl tcp,sm,self” to the
>> mpirun cmd line?
>>
>> On Jun 14, 2016, at 11:15 AM, Jason Maldonis  wrote:
>> Hi Ralph, et. al,
>>
>> Great, thank you for the help. I downloaded the mpi loop spawn test
>> directly from what I think is the master repo on github:
>> https://github.com/open-mpi/ompi/blob/master/orte/test/mpi/loop_spawn.c
>> I am still using the mpi code from 1.10.2, however.
>>
>> Is that test updated with the correct code? If so, I am still getting the
>> same "too many retries sending message to 0x0184:0x1d27, giving up"
>> errors. I also just downloaded the June 14 nightly tarball (7.79MB) from:
>> https://www.open-mpi.org/nightly/v2.x/ and I get the same error.
>>
>> Could you please point me to the correct code?
>>
>> If you need me to provide more information please let me know.
>>
>> Thank you,
>> Jason
>>
>> Jason Maldonis
>> Research Assistant of Professor Paul Voyles
>> Materials Science Grad Student
>> University of Wisconsin, Madison
>> 1509 University Ave, Rm M142
>> Madison, WI 53706
>> maldo...@wisc.edu
>> 608-295-5532
>>
>> On Tue, Jun 14, 2016 at 10:59 AM, Ralph Castain  wrote:
>>
>>> I dug into this a bit (with some help from others) and found that the
>>> spawn code 

Re: [OMPI users] runtime error in orte/loop_spawn test using OMPI 1.10.2

[Ralph Castain]

You don’t want to always use those options as your performance will take a hit 
- TCP vs Infiniband isn’t a good option. Sadly, this is something we need 
someone like Nathan to address as it is a bug in the code base, and in an area 
I’m not familiar with

For now, just use TCP so you can move forward


> On Jun 14, 2016, at 2:14 PM, Jason Maldonis  wrote:
> 
> Ralph, The problem *does* go away if I add "-mca btl tcp,sm,self" to the 
> mpiexec cmd line. (By the way, I am using mpiexec rather than mpirun; do you 
> recommend one over the other?) Will you tell me what this means for me? For 
> example, should I always append these arguments to mpiexec for my non-test 
> jobs as well?   I do not know what you mean by fabric unfortunately, but I 
> can give you some system information (see end of email). Unfortunately I am 
> not a system admin so I do not have sudo rights. Just let me know if I can 
> tell you something more specific though and I will get it.
> 
> Nathan,  Thank you for your response. Unfortunately I have no idea what that 
> means :(  I can forward that to our cluster managers, but I do not know if 
> that is enough information for them to understand what they might need to do 
> to help me with this issue.
> 
> $ lscpu
> Architecture:  x86_64
> CPU op-mode(s):32-bit, 64-bit
> Byte Order:Little Endian
> CPU(s):20
> On-line CPU(s) list:   0-19
> Thread(s) per core:1
> Core(s) per socket:10
> Socket(s): 2
> NUMA node(s):  2
> Vendor ID: GenuineIntel
> CPU family:6
> Model: 63
> Stepping:  2
> CPU MHz:   2594.159
> BogoMIPS:  5187.59
> Virtualization:VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache:  256K
> L3 cache:  25600K
> NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18
> NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19
> 
> Thanks,
> Jason
> 
> Jason Maldonis
> Research Assistant of Professor Paul Voyles
> Materials Science Grad Student
> University of Wisconsin, Madison
> 1509 University Ave, Rm M142
> Madison, WI 53706
> maldo...@wisc.edu 
> 608-295-5532
> 
> On Tue, Jun 14, 2016 at 1:27 PM, Nathan Hjelm  > wrote:
> That message is coming from udcm in the openib btl. It indicates some sort of 
> failure in the connection mechanism. It can happen if the listening thread no 
> longer exists or is taking too long to process messages.
> 
> -Nathan
> 
> 
> On Jun 14, 2016, at 12:20 PM, Ralph Castain  > wrote:
> 
>> Hmm…I’m unable to replicate a problem on my machines. What fabric are you 
>> using? Does the problem go away if you add “-mca btl tcp,sm,self” to the 
>> mpirun cmd line?
>> 
>>> On Jun 14, 2016, at 11:15 AM, Jason Maldonis >> > wrote:
>>> Hi Ralph, et. al,
>>> 
>>> Great, thank you for the help. I downloaded the mpi loop spawn test 
>>> directly from what I think is the master repo on github:  
>>> https://github.com/open-mpi/ompi/blob/master/orte/test/mpi/loop_spawn.c 
>>> 
>>> I am still using the mpi code from 1.10.2, however.
>>> 
>>> Is that test updated with the correct code? If so, I am still getting the 
>>> same "too many retries sending message to 0x0184:0x1d27, giving up" 
>>> errors. I also just downloaded the June 14 nightly tarball (7.79MB) from: 
>>> https://www.open-mpi.org/nightly/v2.x/ 
>>>  and I get the same error.
>>> 
>>> Could you please point me to the correct code?
>>> 
>>> If you need me to provide more information please let me know.
>>> 
>>> Thank you,
>>> Jason
>>> 
>>> Jason Maldonis
>>> Research Assistant of Professor Paul Voyles
>>> Materials Science Grad Student
>>> University of Wisconsin, Madison
>>> 1509 University Ave, Rm M142
>>> Madison, WI 53706
>>> maldo...@wisc.edu 
>>> 608-295-5532 
>>> On Tue, Jun 14, 2016 at 10:59 AM, Ralph Castain >> > wrote:
>>> I dug into this a bit (with some help from others) and found that the spawn 
>>> code appears to be working correctly - it is the test in orte/test that is 
>>> wrong. The test has been correctly updated in the 2.x and master repos, but 
>>> we failed to backport it to the 1.10 series. I have done so this morning, 
>>> and it will be in the upcoming 1.10.3 release (out very soon).
>>> 
>>> 
 On Jun 13, 2016, at 3:53 PM, Ralph Castain > wrote:
 
 No, that PR has nothing to do with loop_spawn. I’ll try to take a look at 
 the problem.
 
> On Jun 13, 2016, at 3:47 PM, Jason Maldonis  > wrote:
> 
> 

Re: [OMPI users] runtime error in orte/loop_spawn test using OMPI 1.10.2

[Jason Maldonis]

Ralph, The problem *does* go away if I add "-mca btl tcp,sm,self" to the
mpiexec cmd line. (By the way, I am using mpiexec rather than mpirun; do
you recommend one over the other?) Will you tell me what this means for me?
For example, should I always append these arguments to mpiexec for my
non-test jobs as well?   I do not know what you mean by fabric
unfortunately, but I can give you some system information (see end of
email). Unfortunately I am not a system admin so I do not have sudo rights.
Just let me know if I can tell you something more specific though and I
will get it.

Nathan,  Thank you for your response. Unfortunately I have no idea what
that means :(  I can forward that to our cluster managers, but I do not
know if that is enough information for them to understand what they might
need to do to help me with this issue.

$ lscpu
Architecture:  x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:Little Endian
CPU(s):20
On-line CPU(s) list:   0-19
Thread(s) per core:1
Core(s) per socket:10
Socket(s): 2
NUMA node(s):  2
Vendor ID: GenuineIntel
CPU family:6
Model: 63
Stepping:  2
CPU MHz:   2594.159
BogoMIPS:  5187.59
Virtualization:VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache:  256K
L3 cache:  25600K
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19

Thanks,
Jason

Jason Maldonis
Research Assistant of Professor Paul Voyles
Materials Science Grad Student
University of Wisconsin, Madison
1509 University Ave, Rm M142
Madison, WI 53706
maldo...@wisc.edu
608-295-5532

On Tue, Jun 14, 2016 at 1:27 PM, Nathan Hjelm  wrote:

> That message is coming from udcm in the openib btl. It indicates some sort
> of failure in the connection mechanism. It can happen if the listening
> thread no longer exists or is taking too long to process messages.
>
> -Nathan
>
>
> On Jun 14, 2016, at 12:20 PM, Ralph Castain  wrote:
>
> Hmm…I’m unable to replicate a problem on my machines. What fabric are you
> using? Does the problem go away if you add “-mca btl tcp,sm,self” to the
> mpirun cmd line?
>
> On Jun 14, 2016, at 11:15 AM, Jason Maldonis  wrote:
> Hi Ralph, et. al,
>
> Great, thank you for the help. I downloaded the mpi loop spawn test
> directly from what I think is the master repo on github:
> https://github.com/open-mpi/ompi/blob/master/orte/test/mpi/loop_spawn.c
> I am still using the mpi code from 1.10.2, however.
>
> Is that test updated with the correct code? If so, I am still getting the
> same "too many retries sending message to 0x0184:0x1d27, giving up"
> errors. I also just downloaded the June 14 nightly tarball (7.79MB) from:
> https://www.open-mpi.org/nightly/v2.x/ and I get the same error.
>
> Could you please point me to the correct code?
>
> If you need me to provide more information please let me know.
>
> Thank you,
> Jason
>
> Jason Maldonis
> Research Assistant of Professor Paul Voyles
> Materials Science Grad Student
> University of Wisconsin, Madison
> 1509 University Ave, Rm M142
> Madison, WI 53706
> maldo...@wisc.edu
> 608-295-5532
>
> On Tue, Jun 14, 2016 at 10:59 AM, Ralph Castain  wrote:
>
>> I dug into this a bit (with some help from others) and found that the
>> spawn code appears to be working correctly - it is the test in orte/test
>> that is wrong. The test has been correctly updated in the 2.x and master
>> repos, but we failed to backport it to the 1.10 series. I have done so this
>> morning, and it will be in the upcoming 1.10.3 release (out very soon).
>>
>>
>> On Jun 13, 2016, at 3:53 PM, Ralph Castain  wrote:
>>
>> No, that PR has nothing to do with loop_spawn. I’ll try to take a look at
>> the problem.
>>
>> On Jun 13, 2016, at 3:47 PM, Jason Maldonis  wrote:
>>
>> Hello,
>>
>> I am using OpenMPI 1.10.2 compiled with Intel. I am trying to get the
>> spawn functionality to work inside a for loop, but continue to get the
>> error "too many retries sending message to , giving up" somewhere
>> down the line in the for loop, seemingly because the processors are not
>> being fully freed when disconnecting/finishing. I found the
>> orte/test/mpi/loop_spawn.c
>> 
>>  example/test, and it has the exact same problem. I also found this
>>  mailing
>> list post from ~ a month and a half ago.
>>
>> Is this PR (https://github.com/open-mpi/ompi/pull/1473) about the same
>> issue I am having (ie the loop_spawn example not working)? If so, do you
>> know if we can downgrade to e.g. 1.10.1 or another version? Or is there
>> another solution to fix this bug until you get a new release out (or is one
>> 

Re: [OMPI users] Client-Server Shared Memory Transport

[Ralph Castain]

Nope - we don’t currently support cross-job shared memory operations. Nathan 
has talked about doing so for vader, but not at this time.


> On Jun 14, 2016, at 12:38 PM, Louis Williams  
> wrote:
> 
> Hi,
> 
> I am attempting to use the sm and vader BTLs between a client and server 
> process, but it seems impossible to use fast transports (i.e. not TCP) 
> between two independent groups started with two separate mpirun invocations. 
> Am I correct, or is there a way to communicate using shared memory between a 
> client and server like this? It seems this might be the case: 
> https://github.com/open-mpi/ompi/blob/master/ompi/dpm/dpm.c#L495 
> 
> 
> The server calls MPI::COMM_WORLD.Accept() and the client calls 
> MPI::COMM_WORLD.Connect(). Each program is started with "mpirun --np 1 --mca 
> btl self,sm,vader " where the executable is either the client or 
> server program. When no BTL is specified, both establish a TCP connection 
> just fine. But when the sm and vader BTLs are specified, immediately after 
> the Connect() call, both client and server exit with the message, copied at 
> the end. It seems as though intergroup communication can't use fast transport 
> and only uses TCP. 
> 
> Also, as expected, running the Accept() and Connect() calls within a single 
> program with "mpirun -np 2 --mca btl self,sm,vader ..." uses shared memory as 
> transport.
> 
> $> mpirun --ompi-server "3414491136.0;tcp://10.4.131.16:49775 
> " -np 1 --mca btl self,vader ./server
> 
> At least one pair of MPI processes are unable to reach each other for
> MPI communications.  This means that no Open MPI device has indicated
> that it can be used to communicate between these processes.  This is
> an error; Open MPI requires that all MPI processes be able to reach
> each other.  This error can sometimes be the result of forgetting to
> specify the "self" BTL.
> 
>   Process 1 ([[50012,1],0]) is on host: MacBook-Pro-80
>   Process 2 ([[50010,1],0]) is on host: MacBook-Pro-80
>   BTLs attempted: self
> 
> Your MPI job is now going to abort; sorry.
> --
> [MacBook-Pro-80.local:57315] [[50012,1],0] ORTE_ERROR_LOG: Unreachable in 
> file dpm_orte.c at line 523
> [MacBook-Pro-80:57315] *** An error occurred in MPI_Comm_accept
> [MacBook-Pro-80:57315] *** reported by process [7572553729,4294967296]
> [MacBook-Pro-80:57315] *** on communicator MPI_COMM_WORLD
> [MacBook-Pro-80:57315] *** MPI_ERR_INTERN: internal error
> [MacBook-Pro-80:57315] *** MPI_ERRORS_ARE_FATAL (processes in this 
> communicator will now abort,
> [MacBook-Pro-80:57315] ***and potentially your MPI job)
> ---
> Primary job  terminated normally, but 1 process returned
> a non-zero exit code.. Per user-direction, the job has been aborted.
> ---
> --
> mpirun detected that one or more processes exited with non-zero status, thus 
> causing
> the job to be terminated. The first process to do so was:
> 
>   Process name: [[50012,1],0]
>   Exit code:17
> -- 
> 
> Thanks,
> Louis
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2016/06/29441.php

[OMPI users] Client-Server Shared Memory Transport

[Louis Williams]

Hi,

I am attempting to use the sm and vader BTLs between a client and server
process, but it seems impossible to use fast transports (i.e. not TCP)
between two independent groups started with two separate mpirun
invocations. Am I correct, or is there a way to communicate using shared
memory between a client and server like this? It seems this might be the
case: https://github.com/open-mpi/ompi/blob/master/ompi/dpm/dpm.c#L495

The server calls MPI::COMM_WORLD.Accept() and the client calls
MPI::COMM_WORLD.Connect(). Each program is started with "mpirun --np 1
--mca btl self,sm,vader " where the executable is either the
client or server program. When no BTL is specified, both establish a TCP
connection just fine. But when the sm and vader BTLs are specified,
immediately after the Connect() call, both client and server exit with the
message, copied at the end. It seems as though intergroup communication
can't use fast transport and only uses TCP.

Also, as expected, running the Accept() and Connect() calls within a single
program with "mpirun -np 2 --mca btl self,sm,vader ..." uses shared memory
as transport.

$> mpirun --ompi-server "3414491136.0;tcp://10.4.131.16:49775" -np 1 --mca
btl self,vader ./server

At least one pair of MPI processes are unable to reach each other for
MPI communications.  This means that no Open MPI device has indicated
that it can be used to communicate between these processes.  This is
an error; Open MPI requires that all MPI processes be able to reach
each other.  This error can sometimes be the result of forgetting to
specify the "self" BTL.

  Process 1 ([[50012,1],0]) is on host: MacBook-Pro-80
  Process 2 ([[50010,1],0]) is on host: MacBook-Pro-80
  BTLs attempted: self

Your MPI job is now going to abort; sorry.
--
[MacBook-Pro-80.local:57315] [[50012,1],0] ORTE_ERROR_LOG: Unreachable in
file dpm_orte.c at line 523
[MacBook-Pro-80:57315] *** An error occurred in MPI_Comm_accept
[MacBook-Pro-80:57315] *** reported by process [7572553729,4294967296]
[MacBook-Pro-80:57315] *** on communicator MPI_COMM_WORLD
[MacBook-Pro-80:57315] *** MPI_ERR_INTERN: internal error
[MacBook-Pro-80:57315] *** MPI_ERRORS_ARE_FATAL (processes in this
communicator will now abort,
[MacBook-Pro-80:57315] ***and potentially your MPI job)
---
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
---
--
mpirun detected that one or more processes exited with non-zero status,
thus causing
the job to be terminated. The first process to do so was:

  Process name: [[50012,1],0]
  Exit code:17
--

Thanks,
Louis

Re: [OMPI users] runtime error in orte/loop_spawn test using OMPI 1.10.2

[Nathan Hjelm]

That message is coming from udcm in the openib btl. It indicates some sort of 
failure in the connection mechanism. It can happen if the listening thread no 
longer exists or is taking too long to process messages.

-Nathan


On Jun 14, 2016, at 12:20 PM, Ralph Castain  wrote:

Hmm…I’m unable to replicate a problem on my machines. What fabric are you 
using? Does the problem go away if you add “-mca btl tcp,sm,self” to the mpirun 
cmd line?

On Jun 14, 2016, at 11:15 AM, Jason Maldonis  wrote:
Hi Ralph, et. al,

Great, thank you for the help. I downloaded the mpi loop spawn test directly 
from what I think is the master repo on github:  
https://github.com/open-mpi/ompi/blob/master/orte/test/mpi/loop_spawn.c
I am still using the mpi code from 1.10.2, however.

Is that test updated with the correct code? If so, I am still getting the same "too 
many retries sending message to 0x0184:0x1d27, giving up" errors. I also just 
downloaded the June 14 nightly tarball (7.79MB) from: 
https://www.open-mpi.org/nightly/v2.x/ and I get the same error.

Could you please point me to the correct code?

If you need me to provide more information please let me know.

Thank you,
Jason

Jason Maldonis
Research Assistant of Professor Paul Voyles
Materials Science Grad Student
University of Wisconsin, Madison
1509 University Ave, Rm M142
Madison, WI 53706
maldo...@wisc.edu
608-295-5532

On Tue, Jun 14, 2016 at 10:59 AM, Ralph Castain  wrote:
I dug into this a bit (with some help from others) and found that the spawn 
code appears to be working correctly - it is the test in orte/test that is 
wrong. The test has been correctly updated in the 2.x and master repos, but we 
failed to backport it to the 1.10 series. I have done so this morning, and it 
will be in the upcoming 1.10.3 release (out very soon).


On Jun 13, 2016, at 3:53 PM, Ralph Castain  wrote:

No, that PR has nothing to do with loop_spawn. I’ll try to take a look at the 
problem.

On Jun 13, 2016, at 3:47 PM, Jason Maldonis  wrote:

Hello,

I am using OpenMPI 1.10.2 compiled with Intel. I am trying to get the spawn functionality to 
work inside a for loop, but continue to get the error "too many retries sending message to 
, giving up" somewhere down the line in the for loop, seemingly because the 
processors are not being fully freed when disconnecting/finishing. I found the 
orte/test/mpi/loop_spawn.c example/test, and it has the exact same problem. I also found this 
mailing list post from ~ a month and a half ago.

Is this PR (https://github.com/open-mpi/ompi/pull/1473) about the same issue I 
am having (ie the loop_spawn example not working)? If so, do you know if we can 
downgrade to e.g. 1.10.1 or another version? Or is there another solution to 
fix this bug until you get a new release out (or is one coming shortly to fix 
this maybe?)?

Below is the output of the loop_spawn test on our university's cluster, which I 
know very little about in terms of architecture but can get information if it's 
helpful. The large group of people who manage this cluster are very good.

Thanks for your time.

Jason

mpiexec -np 5 loop_spawn
parent***
parent: Launching MPI*
parent***
parent: Launching MPI*
parent***
parent: Launching MPI*
parent***
parent: Launching MPI*
parent***
parent: Launching MPI*
parent: MPI_Comm_spawn #0 return : 0
parent: MPI_Comm_spawn #0 return : 0
parent: MPI_Comm_spawn #0 return : 0
parent: MPI_Comm_spawn #0 return : 0
parent: MPI_Comm_spawn #0 return : 0
Child: launch
Child merged rank = 5, size = 6
parent: MPI_Comm_spawn #0 rank 4, size 6
parent: MPI_Comm_spawn #0 rank 0, size 6
parent: MPI_Comm_spawn #0 rank 2, size 6
parent: MPI_Comm_spawn #0 rank 3, size 6
parent: MPI_Comm_spawn #0 rank 1, size 6
Child 329941: exiting
parent: MPI_Comm_spawn #1 return : 0
parent: MPI_Comm_spawn #1 return : 0
parent: MPI_Comm_spawn #1 return : 0
parent: MPI_Comm_spawn #1 return : 0
parent: MPI_Comm_spawn #1 return : 0
Child: launch
parent: MPI_Comm_spawn #1 rank 0, size 6
parent: MPI_Comm_spawn #1 rank 2, size 6
parent: MPI_Comm_spawn #1 rank 1, size 6
parent: MPI_Comm_spawn #1 rank 3, size 6
Child merged rank = 5, size = 6
parent: MPI_Comm_spawn #1 rank 4, size 6
Child 329945: exiting
parent: MPI_Comm_spawn #2 return : 0
parent: MPI_Comm_spawn #2 return : 0
parent: MPI_Comm_spawn #2 return : 0
parent: MPI_Comm_spawn #2 return : 0
parent: MPI_Comm_spawn #2 return : 0
Child: launch
parent: MPI_Comm_spawn #2 rank 3, size 6
parent: MPI_Comm_spawn #2 rank 0, size 6
parent: MPI_Comm_spawn #2 rank 2, size 6
Child merged rank = 5, size = 6
parent: MPI_Comm_spawn #2 rank 1, size 6
parent: MPI_Comm_spawn #2 rank 4, size 6
Child 329949: exiting
parent: MPI_Comm_spawn #3 return : 0
parent: MPI_Comm_spawn #3 return : 0
parent: 

Re: [OMPI users] runtime error in orte/loop_spawn test using OMPI 1.10.2

[Ralph Castain]

Hmm…I’m unable to replicate a problem on my machines. What fabric are you 
using? Does the problem go away if you add “-mca btl tcp,sm,self” to the mpirun 
cmd line?

> On Jun 14, 2016, at 11:15 AM, Jason Maldonis  wrote:
> 
> Hi Ralph, et. al,
> 
> Great, thank you for the help. I downloaded the mpi loop spawn test directly 
> from what I think is the master repo on github:  
> https://github.com/open-mpi/ompi/blob/master/orte/test/mpi/loop_spawn.c 
> 
> I am still using the mpi code from 1.10.2, however.
> 
> Is that test updated with the correct code? If so, I am still getting the 
> same "too many retries sending message to 0x0184:0x1d27, giving up" 
> errors. I also just downloaded the June 14 nightly tarball (7.79MB) from: 
> https://www.open-mpi.org/nightly/v2.x/ 
>  and I get the same error.
> 
> Could you please point me to the correct code?
> 
> If you need me to provide more information please let me know.
> 
> Thank you,
> Jason
> 
> Jason Maldonis
> Research Assistant of Professor Paul Voyles
> Materials Science Grad Student
> University of Wisconsin, Madison
> 1509 University Ave, Rm M142
> Madison, WI 53706
> maldo...@wisc.edu 
> 608-295-5532
> 
> On Tue, Jun 14, 2016 at 10:59 AM, Ralph Castain  > wrote:
> I dug into this a bit (with some help from others) and found that the spawn 
> code appears to be working correctly - it is the test in orte/test that is 
> wrong. The test has been correctly updated in the 2.x and master repos, but 
> we failed to backport it to the 1.10 series. I have done so this morning, and 
> it will be in the upcoming 1.10.3 release (out very soon).
> 
> 
>> On Jun 13, 2016, at 3:53 PM, Ralph Castain > > wrote:
>> 
>> No, that PR has nothing to do with loop_spawn. I’ll try to take a look at 
>> the problem.
>> 
>>> On Jun 13, 2016, at 3:47 PM, Jason Maldonis >> > wrote:
>>> 
>>> Hello,
>>> 
>>> I am using OpenMPI 1.10.2 compiled with Intel. I am trying to get the spawn 
>>> functionality to work inside a for loop, but continue to get the error "too 
>>> many retries sending message to , giving up" somewhere down the line 
>>> in the for loop, seemingly because the processors are not being fully freed 
>>> when disconnecting/finishing. I found the orte/test/mpi/loop_spawn.c 
>>>  
>>> example/test, and it has the exact same problem. I also found this 
>>>  mailing 
>>> list post from ~ a month and a half ago.
>>> 
>>> Is this PR (https://github.com/open-mpi/ompi/pull/1473 
>>> ) about the same issue I am 
>>> having (ie the loop_spawn example not working)? If so, do you know if we 
>>> can downgrade to e.g. 1.10.1 or another version? Or is there another 
>>> solution to fix this bug until you get a new release out (or is one coming 
>>> shortly to fix this maybe?)?
>>> 
>>> Below is the output of the loop_spawn test on our university's cluster, 
>>> which I know very little about in terms of architecture but can get 
>>> information if it's helpful. The large group of people who manage this 
>>> cluster are very good.
>>> 
>>> Thanks for your time.
>>> 
>>> Jason
>>> 
>>> mpiexec -np 5 loop_spawn
>>> parent***
>>> parent: Launching MPI*
>>> parent***
>>> parent: Launching MPI*
>>> parent***
>>> parent: Launching MPI*
>>> parent***
>>> parent: Launching MPI*
>>> parent***
>>> parent: Launching MPI*
>>> parent: MPI_Comm_spawn #0 return : 0
>>> parent: MPI_Comm_spawn #0 return : 0
>>> parent: MPI_Comm_spawn #0 return : 0
>>> parent: MPI_Comm_spawn #0 return : 0
>>> parent: MPI_Comm_spawn #0 return : 0
>>> Child: launch
>>> Child merged rank = 5, size = 6
>>> parent: MPI_Comm_spawn #0 rank 4, size 6
>>> parent: MPI_Comm_spawn #0 rank 0, size 6
>>> parent: MPI_Comm_spawn #0 rank 2, size 6
>>> parent: MPI_Comm_spawn #0 rank 3, size 6
>>> parent: MPI_Comm_spawn #0 rank 1, size 6
>>> Child 329941: exiting
>>> parent: MPI_Comm_spawn #1 return : 0
>>> parent: MPI_Comm_spawn #1 return : 0
>>> parent: MPI_Comm_spawn #1 return : 0
>>> parent: MPI_Comm_spawn #1 return : 0
>>> parent: MPI_Comm_spawn #1 return : 0
>>> Child: launch
>>> parent: MPI_Comm_spawn #1 rank 0, size 6
>>> parent: MPI_Comm_spawn #1 rank 2, size 6
>>> parent: MPI_Comm_spawn #1 rank 1, size 6
>>> parent: MPI_Comm_spawn #1 rank 3, size 6
>>> Child merged rank = 5, size = 6
>>> parent: MPI_Comm_spawn #1 rank 4, size 6
>>> Child 329945: exiting
>>> parent: MPI_Comm_spawn #2 return : 0
>>> parent: 

Re: [OMPI users] runtime error in orte/loop_spawn test using OMPI 1.10.2

[Jason Maldonis]

Hi Ralph, et. al,

Great, thank you for the help. I downloaded the mpi loop spawn test
directly from what I think is the master repo on github:
https://github.com/open-mpi/ompi/blob/master/orte/test/mpi/loop_spawn.c
I am still using the mpi code from 1.10.2, however.

Is that test updated with the correct code? If so, I am still getting the
same "too many retries sending message to 0x0184:0x1d27, giving up"
errors. I also just downloaded the June 14 nightly tarball (7.79MB) from:
https://www.open-mpi.org/nightly/v2.x/ and I get the same error.

Could you please point me to the correct code?

If you need me to provide more information please let me know.

Thank you,
Jason

Jason Maldonis
Research Assistant of Professor Paul Voyles
Materials Science Grad Student
University of Wisconsin, Madison
1509 University Ave, Rm M142
Madison, WI 53706
maldo...@wisc.edu
608-295-5532

On Tue, Jun 14, 2016 at 10:59 AM, Ralph Castain  wrote:

> I dug into this a bit (with some help from others) and found that the
> spawn code appears to be working correctly - it is the test in orte/test
> that is wrong. The test has been correctly updated in the 2.x and master
> repos, but we failed to backport it to the 1.10 series. I have done so this
> morning, and it will be in the upcoming 1.10.3 release (out very soon).
>
>
> On Jun 13, 2016, at 3:53 PM, Ralph Castain  wrote:
>
> No, that PR has nothing to do with loop_spawn. I’ll try to take a look at
> the problem.
>
> On Jun 13, 2016, at 3:47 PM, Jason Maldonis  wrote:
>
> Hello,
>
> I am using OpenMPI 1.10.2 compiled with Intel. I am trying to get the
> spawn functionality to work inside a for loop, but continue to get the
> error "too many retries sending message to , giving up" somewhere
> down the line in the for loop, seemingly because the processors are not
> being fully freed when disconnecting/finishing. I found the
> orte/test/mpi/loop_spawn.c
>  
> example/test,
> and it has the exact same problem. I also found this
>  mailing
> list post from ~ a month and a half ago.
>
> Is this PR (https://github.com/open-mpi/ompi/pull/1473) about the same
> issue I am having (ie the loop_spawn example not working)? If so, do you
> know if we can downgrade to e.g. 1.10.1 or another version? Or is there
> another solution to fix this bug until you get a new release out (or is one
> coming shortly to fix this maybe?)?
>
> Below is the output of the loop_spawn test on our university's cluster,
> which I know very little about in terms of architecture but can get
> information if it's helpful. The large group of people who manage this
> cluster are very good.
>
> Thanks for your time.
>
> Jason
>
> mpiexec -np 5 loop_spawn
> parent***
> parent: Launching MPI*
> parent***
> parent: Launching MPI*
> parent***
> parent: Launching MPI*
> parent***
> parent: Launching MPI*
> parent***
> parent: Launching MPI*
> parent: MPI_Comm_spawn #0 return : 0
> parent: MPI_Comm_spawn #0 return : 0
> parent: MPI_Comm_spawn #0 return : 0
> parent: MPI_Comm_spawn #0 return : 0
> parent: MPI_Comm_spawn #0 return : 0
> Child: launch
> Child merged rank = 5, size = 6
> parent: MPI_Comm_spawn #0 rank 4, size 6
> parent: MPI_Comm_spawn #0 rank 0, size 6
> parent: MPI_Comm_spawn #0 rank 2, size 6
> parent: MPI_Comm_spawn #0 rank 3, size 6
> parent: MPI_Comm_spawn #0 rank 1, size 6
> Child 329941: exiting
> parent: MPI_Comm_spawn #1 return : 0
> parent: MPI_Comm_spawn #1 return : 0
> parent: MPI_Comm_spawn #1 return : 0
> parent: MPI_Comm_spawn #1 return : 0
> parent: MPI_Comm_spawn #1 return : 0
> Child: launch
> parent: MPI_Comm_spawn #1 rank 0, size 6
> parent: MPI_Comm_spawn #1 rank 2, size 6
> parent: MPI_Comm_spawn #1 rank 1, size 6
> parent: MPI_Comm_spawn #1 rank 3, size 6
> Child merged rank = 5, size = 6
> parent: MPI_Comm_spawn #1 rank 4, size 6
> Child 329945: exiting
> parent: MPI_Comm_spawn #2 return : 0
> parent: MPI_Comm_spawn #2 return : 0
> parent: MPI_Comm_spawn #2 return : 0
> parent: MPI_Comm_spawn #2 return : 0
> parent: MPI_Comm_spawn #2 return : 0
> Child: launch
> parent: MPI_Comm_spawn #2 rank 3, size 6
> parent: MPI_Comm_spawn #2 rank 0, size 6
> parent: MPI_Comm_spawn #2 rank 2, size 6
> Child merged rank = 5, size = 6
> parent: MPI_Comm_spawn #2 rank 1, size 6
> parent: MPI_Comm_spawn #2 rank 4, size 6
> Child 329949: exiting
> parent: MPI_Comm_spawn #3 return : 0
> parent: MPI_Comm_spawn #3 return : 0
> parent: MPI_Comm_spawn #3 return : 0
> parent: MPI_Comm_spawn #3 return : 0
> parent: MPI_Comm_spawn #3 return : 0
> Child: launch
> [node:port?] too many retries sending message to , giving up
> ---

Re: [OMPI users] Big jump from OFED 1.5.4.1 -> recent (stable). Any suggestions?

[Llolsten Kaonga]

Hello Grigory,

I am not sure what Redhat does exactly but when you install the OS, there is
always an InfiniBand Support module during the installation process. We
never check/install that module when we do OS installations because it is
usually several versions of OFED behind (almost obsolete).

-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Grigory Shamov
Sent: Tuesday, June 14, 2016 12:21 PM
To: Open MPI Users 
Subject: Re: [OMPI users] Big jump from OFED 1.5.4.1 -> recent (stable). Any
suggestions?



On 2016-06-14, 3:42 AM, "users on behalf of Peter Kjellström"
 wrote:

>On Mon, 13 Jun 2016 19:04:59 -0400
>Mehmet Belgin  wrote:
>
>> Greetings!
>> 
>> We have not upgraded our OFED stack for a very long time, and still 
>> running on an ancient version (1.5.4.1, yeah we know). We are now 
>> considering a big jump from this version to a tested and stable 
>> recent version and would really appreciate any suggestions from the 
>> community.
>
>Some thoughts on the subject.
>
>* Not installing an external ibstack is quite attractive imo.
>  RHEL/CentOS stack (not based on any direct OFED version) works fine
>  for us. It simplifies cluster maintenance (kernel updates etc.).


I am curious on how Redhat stack is ³not based on any direct OFED version²? 
Doesn¹t Redhat just ship an old OFED build, or they do their own changes to
it like to the kernel?

--
Grigory Shamov

Westgrid/ComputeCanada Site Lead
University of Manitoba
E2-588 EITC Building,
(204) 474-9625



___
users mailing list
us...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2016/06/29436.php

Re: [OMPI users] Big jump from OFED 1.5.4.1 -> recent (stable). Any suggestions?

[Grigory Shamov]

On 2016-06-14, 3:42 AM, "users on behalf of Peter Kjellström"
 wrote:

>On Mon, 13 Jun 2016 19:04:59 -0400
>Mehmet Belgin  wrote:
>
>> Greetings!
>> 
>> We have not upgraded our OFED stack for a very long time, and still
>> running on an ancient version (1.5.4.1, yeah we know). We are now
>> considering a big jump from this version to a tested and stable
>> recent version and would really appreciate any suggestions from the
>> community.
>
>Some thoughts on the subject.
>
>* Not installing an external ibstack is quite attractive imo.
>  RHEL/CentOS stack (not based on any direct OFED version) works fine
>  for us. It simplifies cluster maintenance (kernel updates etc.).


I am curious on how Redhat stack is ³not based on any direct OFED
version²? 
Doesn¹t Redhat just ship an old OFED build, or they do their own changes
to it like to the kernel?

-- 
Grigory Shamov

Westgrid/ComputeCanada Site Lead
University of Manitoba
E2-588 EITC Building,
(204) 474-9625

Re: [OMPI users] runtime error in orte/loop_spawn test using OMPI 1.10.2

[Ralph Castain]

I dug into this a bit (with some help from others) and found that the spawn 
code appears to be working correctly - it is the test in orte/test that is 
wrong. The test has been correctly updated in the 2.x and master repos, but we 
failed to backport it to the 1.10 series. I have done so this morning, and it 
will be in the upcoming 1.10.3 release (out very soon).


> On Jun 13, 2016, at 3:53 PM, Ralph Castain  wrote:
> 
> No, that PR has nothing to do with loop_spawn. I’ll try to take a look at the 
> problem.
> 
>> On Jun 13, 2016, at 3:47 PM, Jason Maldonis > > wrote:
>> 
>> Hello,
>> 
>> I am using OpenMPI 1.10.2 compiled with Intel. I am trying to get the spawn 
>> functionality to work inside a for loop, but continue to get the error "too 
>> many retries sending message to , giving up" somewhere down the line 
>> in the for loop, seemingly because the processors are not being fully freed 
>> when disconnecting/finishing. I found the orte/test/mpi/loop_spawn.c 
>>  
>> example/test, and it has the exact same problem. I also found this 
>>  mailing 
>> list post from ~ a month and a half ago.
>> 
>> Is this PR (https://github.com/open-mpi/ompi/pull/1473 
>> ) about the same issue I am 
>> having (ie the loop_spawn example not working)? If so, do you know if we can 
>> downgrade to e.g. 1.10.1 or another version? Or is there another solution to 
>> fix this bug until you get a new release out (or is one coming shortly to 
>> fix this maybe?)?
>> 
>> Below is the output of the loop_spawn test on our university's cluster, 
>> which I know very little about in terms of architecture but can get 
>> information if it's helpful. The large group of people who manage this 
>> cluster are very good.
>> 
>> Thanks for your time.
>> 
>> Jason
>> 
>> mpiexec -np 5 loop_spawn
>> parent***
>> parent: Launching MPI*
>> parent***
>> parent: Launching MPI*
>> parent***
>> parent: Launching MPI*
>> parent***
>> parent: Launching MPI*
>> parent***
>> parent: Launching MPI*
>> parent: MPI_Comm_spawn #0 return : 0
>> parent: MPI_Comm_spawn #0 return : 0
>> parent: MPI_Comm_spawn #0 return : 0
>> parent: MPI_Comm_spawn #0 return : 0
>> parent: MPI_Comm_spawn #0 return : 0
>> Child: launch
>> Child merged rank = 5, size = 6
>> parent: MPI_Comm_spawn #0 rank 4, size 6
>> parent: MPI_Comm_spawn #0 rank 0, size 6
>> parent: MPI_Comm_spawn #0 rank 2, size 6
>> parent: MPI_Comm_spawn #0 rank 3, size 6
>> parent: MPI_Comm_spawn #0 rank 1, size 6
>> Child 329941: exiting
>> parent: MPI_Comm_spawn #1 return : 0
>> parent: MPI_Comm_spawn #1 return : 0
>> parent: MPI_Comm_spawn #1 return : 0
>> parent: MPI_Comm_spawn #1 return : 0
>> parent: MPI_Comm_spawn #1 return : 0
>> Child: launch
>> parent: MPI_Comm_spawn #1 rank 0, size 6
>> parent: MPI_Comm_spawn #1 rank 2, size 6
>> parent: MPI_Comm_spawn #1 rank 1, size 6
>> parent: MPI_Comm_spawn #1 rank 3, size 6
>> Child merged rank = 5, size = 6
>> parent: MPI_Comm_spawn #1 rank 4, size 6
>> Child 329945: exiting
>> parent: MPI_Comm_spawn #2 return : 0
>> parent: MPI_Comm_spawn #2 return : 0
>> parent: MPI_Comm_spawn #2 return : 0
>> parent: MPI_Comm_spawn #2 return : 0
>> parent: MPI_Comm_spawn #2 return : 0
>> Child: launch
>> parent: MPI_Comm_spawn #2 rank 3, size 6
>> parent: MPI_Comm_spawn #2 rank 0, size 6
>> parent: MPI_Comm_spawn #2 rank 2, size 6
>> Child merged rank = 5, size = 6
>> parent: MPI_Comm_spawn #2 rank 1, size 6
>> parent: MPI_Comm_spawn #2 rank 4, size 6
>> Child 329949: exiting
>> parent: MPI_Comm_spawn #3 return : 0
>> parent: MPI_Comm_spawn #3 return : 0
>> parent: MPI_Comm_spawn #3 return : 0
>> parent: MPI_Comm_spawn #3 return : 0
>> parent: MPI_Comm_spawn #3 return : 0
>> Child: launch
>> [node:port?] too many retries sending message to , giving up
>> ---
>> Child job 5 terminated normally, but 1 process returned
>> a non-zero exit code.. Per user-direction, the job has been aborted.
>> ---
>> --
>> mpiexec detected that one or more processes exited with non-zero status, 
>> thus causing
>> the job to be terminated. The first process to do so was:
>> 
>>   Process name: [[...],0]
>>   Exit code:255
>> --
>> ___
>> users mailing list
>> us...@open-mpi.org 
>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> 

[OMPI users] MPI_File_read+MPI_BOTTOM crash on NFS ?

[Nicolas Joly]

Hi,

At work, i do have some mpi codes that make use of custom datatypes to
call MPI_File_read with MPI_BOTTOM ... It mostly works, except when
the underlying filesystem is NFS where if crash with SIGSEGV.

The attached sample (code + data) works just fine with 1.10.1 on my
NetBSD/amd64 workstation using the UFS romio backend, but crash if
switched to NFS :

njoly@issan [~]> mpirun --version
mpirun (Open MPI) 1.10.1
njoly@issan [~]> mpicc -g -Wall -o sample sample.c
njoly@issan [~]> mpirun -n 2 ./sample ufs:data.txt
rank1 ... 113355
rank0 ... 002244
njoly@issan [~]> mpirun -n 2 ./sample nfs:data.txt
[issan:20563] *** Process received signal ***
[issan:08879] *** Process received signal ***
[issan:20563] Signal: Segmentation fault (11)
[issan:20563] Signal code: Address not mapped (1)
[issan:20563] Failing at address: 0xb1309240
[issan:08879] Signal: Segmentation fault (11)
[issan:08879] Signal code: Address not mapped (1)
[issan:08879] Failing at address: 0x881b0420
[issan:08879] [ 0] [issan:20563] [ 0] 0x7dafb14a52b0 <__sigtramp_siginfo_2> at 
/usr/lib/libc.so.12
[issan:20563] *** End of error message ***
0x78b9886a52b0 <__sigtramp_siginfo_2> at /usr/lib/libc.so.12
[issan:08879] *** End of error message ***
--
mpirun noticed that process rank 0 with PID 20563 on node issan exited on 
signal 11 (Segmentation fault).
--
njoly@issan [~]> gdb sample sample.core
GNU gdb (GDB) 7.10.1
[...]
Core was generated by `sample'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x78b98871971f in memcpy () from /usr/lib/libc.so.12
[Current thread is 1 (LWP 1)]
(gdb) bt
#0  0x78b98871971f in memcpy () from /usr/lib/libc.so.12
#1  0x78b974010edf in ADIOI_NFS_ReadStrided () from 
/usr/pkg/lib/openmpi/mca_io_romio.so
#2  0x78b97400bacf in MPIOI_File_read () from 
/usr/pkg/lib/openmpi/mca_io_romio.so
#3  0x78b97400bc72 in mca_io_romio_dist_MPI_File_read () from 
/usr/pkg/lib/openmpi/mca_io_romio.so
#4  0x78b988e72b38 in PMPI_File_read () from /usr/pkg/lib/libmpi.so.12
#5  0x004013a4 in main (argc=2, argv=0x7f7fff7b0f00) at sample.c:63

Thanks.

-- 
Nicolas Joly

Cluster & Computing Group
Biology IT Center
Institut Pasteur, Paris.

#include 
#include 
#include 
#include 
#include 
#include 

int main(int argc, char **argv) {
  char *file, *rbuf;
  int i, res, rank, size, count;
  size_t len;

  MPI_Comm comm = MPI_COMM_WORLD;
  MPI_Datatype data, view;
  MPI_File fh;
  MPI_Offset fs;
  MPI_Status sts;

  int lens[6], offs[6];
  MPI_Aint addr, disp[6];

  file = argv[1];

  res = MPI_Init(, );
  assert(res == MPI_SUCCESS);
  res = MPI_Comm_size(comm, );
  assert(res == MPI_SUCCESS);
  res = MPI_Comm_rank(comm, );
  assert(res == MPI_SUCCESS);

  res = MPI_File_open(comm, file, MPI_MODE_RDONLY, MPI_INFO_NULL, );
  assert(res == MPI_SUCCESS);
  res = MPI_File_get_size(fh, );
  assert(res == MPI_SUCCESS);

  count = fs / 10 / size;
  len = count * 10;
  rbuf = malloc(len+1);
  assert(rbuf != NULL);
  memset(rbuf, 0, len+1);

  for (i = 0; i < count; i++) {
lens[i] = 10;
offs[i] = rank * 10 + size * i * 10;
res = MPI_Get_address(rbuf + i * 10, );
assert(res == MPI_SUCCESS);
disp[i] = addr;
  }

  res = MPI_Type_create_hindexed(count, lens, disp, MPI_CHAR, );
  assert(res == MPI_SUCCESS);
  res = MPI_Type_commit();
  assert(res == MPI_SUCCESS);

  res = MPI_Type_indexed(count, lens, offs, MPI_CHAR, );
  assert(res == MPI_SUCCESS);
  res = MPI_Type_commit();
  assert(res == MPI_SUCCESS);

  res = MPI_File_set_view(fh, 0, MPI_CHAR, view, "native", MPI_INFO_NULL);
  assert(res == MPI_SUCCESS);
  res = MPI_File_read(fh, MPI_BOTTOM, 1, data, );
  assert(res == MPI_SUCCESS);
  res = MPI_Get_count(, data, );
  assert(res == MPI_SUCCESS);
  assert(count == 1);

  printf("rank%d ... %s\n", rank, rbuf);

  res = MPI_Type_free();
  assert(res == MPI_SUCCESS);

  res = MPI_Type_free();
  assert(res == MPI_SUCCESS);

  free(rbuf);

  res = MPI_File_close();
  assert(res == MPI_SUCCESS);

  res = MPI_Finalize();
  assert(res == MPI_SUCCESS);

  return 0; }
001122334455

Re: [OMPI users] scatter/gather, tcp, 3 nodes, homogeneous, # RAM

[Gilles Gouaillardet]

Note if your program is synchronous, it will run at the speed of the
slowest task.
(E.g. Tasks on node 2, 1GB per task, will wait for the other tasks, 2 GB
per task)

You can use MPI_Comm_split_type in order to create inter node communicators.
Then you can find how much memory is available per task, MPI_Gather that on
the master task, and MPI_Scatterv/MPI_Gatherv to distribute/consolidate the
data

Cheers,

Gilles

On Tuesday, June 14, 2016, MM  wrote:

> Hello,
> I have the following 3 1-socket nodes:
>
> node1:  4GB RAM 2-core: rank 0  rank 1
> node2:  4GB RAM 4-core: rank 2  rank 3 rank 4 rank 5
> node3:  8GB RAM 4-core: rank 6  rank 7 rank 8 rank 9
>
> I have a model that takes a input and produces a output, and I want to run
> this model for N possible combinations of inputs. N is very big and i am
> limited by memory capacity.
>
> I am using the world collective and I want to know how to distribute N
> over the 10 ranks, given the mem specs of each node.
>
> For now, i have been simply dividing N by the number of ranks and
> scatter/gather that way.
> How can I improve without hardcoding the specs in my own c++ code?
>
> thanks,
>
>

[OMPI users] scatter/gather, tcp, 3 nodes, homogeneous, # RAM

[MM]

Hello,
I have the following 3 1-socket nodes:

node1:  4GB RAM 2-core: rank 0  rank 1
node2:  4GB RAM 4-core: rank 2  rank 3 rank 4 rank 5
node3:  8GB RAM 4-core: rank 6  rank 7 rank 8 rank 9

I have a model that takes a input and produces a output, and I want to run
this model for N possible combinations of inputs. N is very big and i am
limited by memory capacity.

I am using the world collective and I want to know how to distribute N over
the 10 ranks, given the mem specs of each node.

For now, i have been simply dividing N by the number of ranks and
scatter/gather that way.
How can I improve without hardcoding the specs in my own c++ code?

thanks,

Re: [OMPI users] OMPI users] Big jump from OFED 1.5.4.1 -> recent (stable). Any suggestions?

[Gilles Gouaillardet]

HI,

Unless you provide Open MPI static libraries, you might not be required to 
rebuild your apps.
You will likely have to / should rebuild OpenMPI though

Cheers,

Gilles

Peter Kjellström  wrote:
>On Mon, 13 Jun 2016 19:04:59 -0400
>Mehmet Belgin  wrote:
>
>> Greetings!
>> 
>> We have not upgraded our OFED stack for a very long time, and still 
>> running on an ancient version (1.5.4.1, yeah we know). We are now 
>> considering a big jump from this version to a tested and stable
>> recent version and would really appreciate any suggestions from the
>> community.
>
>Some thoughts on the subject.
>
>* Not installing an external ibstack is quite attractive imo.
>  RHEL/CentOS stack (not based on any direct OFED version) works fine
>  for us. It simplifies cluster maintenance (kernel updates etc.).
>
>* If you use an external IB-stack consider the constraints it may put
>  on your update plans (for example, you want to update to CentOS-7.3
>  but your OFED only supports 7.2...).
>
>* Also consider updates for the stack itself wrt. security. Upstream
>  OFED has been quite good at patching security bus but they DO NOT
>  maintain older releases (-> you may have to run a nightly build of
>  latest). Mellanox has patched when poked at but also only for latest
>  version. Intel does not seem to do security afaict and with a dist
>  stack it's covered by the normal dist updates.
>
>/Peter K
>___
>users mailing list
>us...@open-mpi.org
>Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/users
>Link to this post: 
>http://www.open-mpi.org/community/lists/users/2016/06/29430.php

Re: [OMPI users] Big jump from OFED 1.5.4.1 -> recent (stable). Any suggestions?

[Peter Kjellström]

On Mon, 13 Jun 2016 19:04:59 -0400
Mehmet Belgin  wrote:

> Greetings!
> 
> We have not upgraded our OFED stack for a very long time, and still 
> running on an ancient version (1.5.4.1, yeah we know). We are now 
> considering a big jump from this version to a tested and stable
> recent version and would really appreciate any suggestions from the
> community.

Some thoughts on the subject.

* Not installing an external ibstack is quite attractive imo.
  RHEL/CentOS stack (not based on any direct OFED version) works fine
  for us. It simplifies cluster maintenance (kernel updates etc.).

* If you use an external IB-stack consider the constraints it may put
  on your update plans (for example, you want to update to CentOS-7.3
  but your OFED only supports 7.2...).

* Also consider updates for the stack itself wrt. security. Upstream
  OFED has been quite good at patching security bus but they DO NOT
  maintain older releases (-> you may have to run a nightly build of
  latest). Mellanox has patched when poked at but also only for latest
  version. Intel does not seem to do security afaict and with a dist
  stack it's covered by the normal dist updates.

/Peter K