Re: [OMPI users] glut display 'occasionally' opens

[Ralph Castain]

Hmmm...yes, the code does seem to handle that '=' being in there. Forgot it was 
there.

Depending on the version you are using, mpirun could just open the display for 
you. There is an mpirun option that tells us to please start each app in its 
own xterm.

You shouldn't need forwarding if you are going to see it on a local display 
(i.e., one physically attached to the node), assuming you are logged into those 
nodes (otherwise you don't own the display).

If you are trying to view it on your own local display, then you do need 
forwarding setup.


On Dec 6, 2010, at 8:36 PM, brad baker wrote:

> Without including the -x DISPLAY, glut doesn't know what display to open.  
> For instance, without the -x DISPLAY parameter glut returns an error from 
> each process stating that it could not find display "" (empty string).  This 
> strategy is briefly described in the openmpi   FAQs for launching gui 
> applications with openmpi. 
> 
> I'm assuming that by setting the DISPLAY envar to :0.0, each process will 
> render to their local display, which is my intention, and as I previously 
> stated works for up to 2 processes. So I believe it to be necessary.  
> 
> But I'm thinking I may have to configure some kind of  X11 forwarding.  I'm 
> not sure...
> 
> Thanks for your reply! Any more ideas?
> Brad
> 
> 
> On Mon, Dec 6, 2010 at 6:31 PM, Ralph Castain  wrote:
> Guess I'm not entirely sure I understand how this is supposed to work. All 
> the -x does is tell us to pickup an envar of the given name and forward its 
> value to the remote apps. You can't set the envar's value on the cmd line. So 
> you told mpirun to pickup the value of an envar called "DISPLAY=:0.0".
> 
> So yes - I would expect this would be behaving strangely.
> 
> If you tell us -x DISPLAY, we'll pickup the local value of DISPLAY and 
> forward it. What that will cause your app to do is, I suppose, up to it.
> 
> 
> On Dec 6, 2010, at 12:42 PM, brad baker wrote:
> 
> > Hello,
> >
> > I'm working on an mpi application that opens a glut display on each node of 
> > a small cluster for opengl rendering (each node has its own display). My 
> > current implementation scales great with mpich2, but I'd like to use 
> > openmpi infiniband, which is giving me trouble.
> >
> > I've had some success with the -x DISPLAY=:0.0 parameter to mpirun, which 
> > will open the display on up to 2 of my nodes... any 2.  But when I attempt 
> > to run the application on 4 nodes, the display is non-deterministic. If any 
> > open at all process 0 definately will, and sometimes process 3 along with 
> > that.  I haven't observed much behavior from process 1 or 2.
> >
> > My command:
> >
> > mpirun -x DISPLAY=:0.0 -np 4 -hostfile ~/openmpi.hosts ./myapp
> >
> > I've tried adding the -d option with no success.
> >
> > Does anyone have any suggestions or comments?  I'll certainly provide more 
> > information if required.
> >
> > Thanks,
> > Brad
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_Send doesn't work if the data >= 2GB

[Gus Correa]

Hi Xianjun

Suggestions/Questions:

1) Did you check if malloc returns a non-NULL pointer?
Your program is assuming this, but it may not be true,
and in this case the problem is not with MPI.
You can print a message and call MPI_Abort if it doesn't.

2) Have you tried MPI_Isend/MPI_Irecv?
Or perhaps the buffered cousin MPI_Ibsend?

3) Why do you want to send these huge messages?
Wouldn't it be less of a trouble to send several
smaller messages?

I hope it helps,
Gus Correa

Xianjun wrote:


Hi

Are you running on two processes (mpiexec -n 2)?
Yes

Have you tried to print Gsize?
Yes, I had checked my codes several times, and I thought the errors came 
from the OpenMpi. :)


The command line I used:
"mpirun -hostfile ./Serverlist -np 2 ./test". The "Serverlist" file 
include several computers in my network.


The command line that I used to build the openmpi-1.4.1:
./configure --enable-debug --prefix=/usr/work/openmpi ; make all install;

What interconnect do you use?
It is normal TCP/IP interconnect with 1GB network card. when I debugged 
my codes(and the openmpi codes), I found the openMpi do call the 
"mca_pml_ob1_send_request_start_rdma(...)" function, but I was not quite 
sure which protocal was used when transfer 2BG data. Do you have any 
opinions? Thanks


Best Regards
Xianjun Meng

2010/12/7 Gus Correa >

Hi Xianjun

Are you running on two processes (mpiexec -n 2)?
I think this code will deadlock for more than two processes.
The MPI_Recv won't have a matching send for rank>1.

Also, this is C, not MPI,
but you may be wrapping into the negative numbers.
Have you tried to print Gsize?
It is probably -2147483648 in 32bit and 64bit machines.

My two cents.
Gus Correa

Mike Dubman wrote:

Hi,
What interconnect and command line do you use? For InfiniBand
openib component there is a known issue with large transfers (2GB)

https://svn.open-mpi.org/trac/ompi/ticket/2623

try disabling memory pinning:

http://www.open-mpi.org/faq/?category=openfabrics#large-message-leave-pinned


regards
M


2010/12/6 
>>


   hi,

   In my computers(X86-64), the sizeof(int)=4, but the
   sizeof(long)=sizeof(double)=sizeof(size_t)=8. when I checked my
   mpi.h file, I found that the definition about the sizeof(int) is
   correct. meanwhile, I think the mpi.h file was generated
according
   to my compute environment when I compiled the Openmpi. So, my
codes
   still don't work. :(

   Further, I found when I called the collective routines(such as,
   MPI_Allgatherv(...)) which are implemented by the Point 2 Point
   don't work either when the data > 2GB.

   Thanks
   Xianjun

   2010/12/6 Tim Prince 
>>


   On 12/5/2010 7:13 PM, Xianjun wrote:

   hi,

   I met a question recently when I tested the MPI_send and
   MPI_Recv
   functions. When I run the following codes, the processes
   hanged and I
   found there was not data transmission in my network
at all.

   BTW: I finished this test on two X86-64 computers
with 16GB
   memory and
   installed Linux.

   1 #include 
   2 #include 
   3 #include 
   4 #include 
   5
   6
   7 int main(int argc, char** argv)
   8 {
   9 int localID;
   10 int numOfPros;
   11 size_t Gsize = (size_t)2 * 1024 * 1024 * 1024;
   12
   13 char* g = (char*)malloc(Gsize);
   14
   15 MPI_Init(, );
   16 MPI_Comm_size(MPI_COMM_WORLD, );
   17 MPI_Comm_rank(MPI_COMM_WORLD, );
   18
   19 MPI_Datatype MPI_Type_lkchar;
   20 MPI_Type_contiguous(2048, MPI_BYTE, _Type_lkchar);
   21 MPI_Type_commit(_Type_lkchar);
   22
   23 if (localID == 0)
   24 {
   25 MPI_Send(g, 1024*1024, MPI_Type_lkchar, 1, 1,
   MPI_COMM_WORLD);
   26 }
   27
   28 if (localID != 0)
   29 {
   30 MPI_Status status;
   31 MPI_Recv(g, 1024*1024, MPI_Type_lkchar, 0, 1, \
   32 MPI_COMM_WORLD, );
   33 }
   34
   

Re: [OMPI users] glut display 'occasionally' opens

[brad baker]

Without including the -x DISPLAY, glut doesn't know what display to open.
 For instance, without the -x DISPLAY parameter glut returns an error from
each process stating that it could not find display "" (empty string).  This
strategy is briefly described in the openmpi
FAQs for
launching gui applications with openmpi.

I'm assuming that by setting the DISPLAY envar to :0.0, each process will
render to their local display, which is my intention, and as I previously
stated works for up to 2 processes. So I believe it to be necessary.

But I'm thinking I may have to configure some kind of  X11 forwarding.  I'm
not sure...

Thanks for your reply! Any more ideas?
Brad


On Mon, Dec 6, 2010 at 6:31 PM, Ralph Castain  wrote:

> Guess I'm not entirely sure I understand how this is supposed to work. All
> the -x does is tell us to pickup an envar of the given name and forward its
> value to the remote apps. You can't set the envar's value on the cmd line.
> So you told mpirun to pickup the value of an envar called "DISPLAY=:0.0".
>
> So yes - I would expect this would be behaving strangely.
>
> If you tell us -x DISPLAY, we'll pickup the local value of DISPLAY and
> forward it. What that will cause your app to do is, I suppose, up to it.
>
>
> On Dec 6, 2010, at 12:42 PM, brad baker wrote:
>
> > Hello,
> >
> > I'm working on an mpi application that opens a glut display on each node
> of a small cluster for opengl rendering (each node has its own display). My
> current implementation scales great with mpich2, but I'd like to use openmpi
> infiniband, which is giving me trouble.
> >
> > I've had some success with the -x DISPLAY=:0.0 parameter to mpirun, which
> will open the display on up to 2 of my nodes... any 2.  But when I attempt
> to run the application on 4 nodes, the display is non-deterministic. If any
> open at all process 0 definately will, and sometimes process 3 along with
> that.  I haven't observed much behavior from process 1 or 2.
> >
> > My command:
> >
> > mpirun -x DISPLAY=:0.0 -np 4 -hostfile ~/openmpi.hosts ./myapp
> >
> > I've tried adding the -d option with no success.
> >
> > Does anyone have any suggestions or comments?  I'll certainly provide
> more information if required.
> >
> > Thanks,
> > Brad
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] MPI_Send doesn't work if the data >= 2GB

[孟宪军]

Hi

Are you running on two processes (mpiexec -n 2)?
Yes

Have you tried to print Gsize?
Yes, I had checked my codes several times, and I thought the errors came
from the OpenMpi. :)

The command line I used:
"mpirun -hostfile ./Serverlist -np 2 ./test". The "Serverlist" file include
several computers in my network.

The command line that I used to build the openmpi-1.4.1:
./configure --enable-debug --prefix=/usr/work/openmpi ; make all install;

What interconnect do you use?
It is normal TCP/IP interconnect with 1GB network card. when I debugged my
codes(and the openmpi codes), I found the openMpi do call the
"mca_pml_ob1_send_request_start_rdma(...)" function, but I was not quite
sure which protocal was used when transfer 2BG data. Do you have any
opinions? Thanks

Best Regards
Xianjun Meng

2010/12/7 Gus Correa 

> Hi Xianjun
>
> Are you running on two processes (mpiexec -n 2)?
> I think this code will deadlock for more than two processes.
> The MPI_Recv won't have a matching send for rank>1.
>
> Also, this is C, not MPI,
> but you may be wrapping into the negative numbers.
> Have you tried to print Gsize?
> It is probably -2147483648 in 32bit and 64bit machines.
>
> My two cents.
> Gus Correa
>
> Mike Dubman wrote:
>
>> Hi,
>> What interconnect and command line do you use? For InfiniBand openib
>> component there is a known issue with large transfers (2GB)
>>
>> https://svn.open-mpi.org/trac/ompi/ticket/2623
>>
>> try disabling memory pinning:
>> http://www.open-mpi.org/faq/?category=openfabrics#large-message-leave-pinned
>>
>>
>> regards
>> M
>>
>>
>> 2010/12/6 >
>>
>>
>>hi,
>>
>>In my computers(X86-64), the sizeof(int)=4, but the
>>sizeof(long)=sizeof(double)=sizeof(size_t)=8. when I checked my
>>mpi.h file, I found that the definition about the sizeof(int) is
>>correct. meanwhile, I think the mpi.h file was generated according
>>to my compute environment when I compiled the Openmpi. So, my codes
>>still don't work. :(
>>
>>Further, I found when I called the collective routines(such as,
>>MPI_Allgatherv(...)) which are implemented by the Point 2 Point
>>don't work either when the data > 2GB.
>>
>>Thanks
>>Xianjun
>>
>>2010/12/6 Tim Prince >
>>
>>
>>On 12/5/2010 7:13 PM, Xianjun wrote:
>>
>>hi,
>>
>>I met a question recently when I tested the MPI_send and
>>MPI_Recv
>>functions. When I run the following codes, the processes
>>hanged and I
>>found there was not data transmission in my network at all.
>>
>>BTW: I finished this test on two X86-64 computers with 16GB
>>memory and
>>installed Linux.
>>
>>1 #include 
>>2 #include 
>>3 #include 
>>4 #include 
>>5
>>6
>>7 int main(int argc, char** argv)
>>8 {
>>9 int localID;
>>10 int numOfPros;
>>11 size_t Gsize = (size_t)2 * 1024 * 1024 * 1024;
>>12
>>13 char* g = (char*)malloc(Gsize);
>>14
>>15 MPI_Init(, );
>>16 MPI_Comm_size(MPI_COMM_WORLD, );
>>17 MPI_Comm_rank(MPI_COMM_WORLD, );
>>18
>>19 MPI_Datatype MPI_Type_lkchar;
>>20 MPI_Type_contiguous(2048, MPI_BYTE, _Type_lkchar);
>>21 MPI_Type_commit(_Type_lkchar);
>>22
>>23 if (localID == 0)
>>24 {
>>25 MPI_Send(g, 1024*1024, MPI_Type_lkchar, 1, 1,
>>MPI_COMM_WORLD);
>>26 }
>>27
>>28 if (localID != 0)
>>29 {
>>30 MPI_Status status;
>>31 MPI_Recv(g, 1024*1024, MPI_Type_lkchar, 0, 1, \
>>32 MPI_COMM_WORLD, );
>>33 }
>>34
>>35 MPI_Finalize();
>>36
>>37 return 0;
>>38 }
>>
>>You supplied all your constants as 32-bit signed data, so, even
>>if the count for MPI_Send() and MPI_Recv() were a larger data
>>type, you would see this limit. Did you look at your  ?
>>
>>-- Tim Prince
>>
>>___
>>users mailing list
>>us...@open-mpi.org 
>>
>>http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>>
>>___
>>users mailing list
>>us...@open-mpi.org 
>>
>>http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>>
>> 
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
> 

Re: [OMPI users] glut display 'occasionally' opens

[Ralph Castain]

Guess I'm not entirely sure I understand how this is supposed to work. All the 
-x does is tell us to pickup an envar of the given name and forward its value 
to the remote apps. You can't set the envar's value on the cmd line. So you 
told mpirun to pickup the value of an envar called "DISPLAY=:0.0".

So yes - I would expect this would be behaving strangely.

If you tell us -x DISPLAY, we'll pickup the local value of DISPLAY and forward 
it. What that will cause your app to do is, I suppose, up to it.


On Dec 6, 2010, at 12:42 PM, brad baker wrote:

> Hello, 
> 
> I'm working on an mpi application that opens a glut display on each node of a 
> small cluster for opengl rendering (each node has its own display). My 
> current implementation scales great with mpich2, but I'd like to use openmpi 
> infiniband, which is giving me trouble.
> 
> I've had some success with the -x DISPLAY=:0.0 parameter to mpirun, which 
> will open the display on up to 2 of my nodes... any 2.  But when I attempt to 
> run the application on 4 nodes, the display is non-deterministic. If any open 
> at all process 0 definately will, and sometimes process 3 along with that.  I 
> haven't observed much behavior from process 1 or 2.  
> 
> My command:
> 
> mpirun -x DISPLAY=:0.0 -np 4 -hostfile ~/openmpi.hosts ./myapp
> 
> I've tried adding the -d option with no success.
> 
> Does anyone have any suggestions or comments?  I'll certainly provide more 
> information if required.
> 
> Thanks,
> Brad
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] glut display 'occasionally' opens

[brad baker]

Hello,

I'm working on an mpi application that opens a glut display on each node of
a small cluster for opengl rendering (each node has its own display). My
current implementation scales great with mpich2, but I'd like to use openmpi
infiniband, which is giving me trouble.

I've had some success with the -x DISPLAY=:0.0 parameter to mpirun, which
will open the display on up to 2 of my nodes... any 2.  But when I attempt
to run the application on 4 nodes, the display is *non-deterministic. If any
open at all process 0 definately will, and sometimes process 3 along with
that.  I haven't observed much behavior from process 1 or 2.

My command:

mpirun -x DISPLAY=:0.0 -np 4 -hostfile ~/openmpi.hosts ./myapp

I've tried adding the -d option with no success.

Does anyone have any suggestions or comments?  I'll certainly provide more
information if required.

Thanks,
Brad
*

Re: [OMPI users] MPI_Send doesn't work if the data >= 2GB

[Gus Correa]

Hi Xianjun

Are you running on two processes (mpiexec -n 2)?
I think this code will deadlock for more than two processes.
The MPI_Recv won't have a matching send for rank>1.

Also, this is C, not MPI,
but you may be wrapping into the negative numbers.
Have you tried to print Gsize?
It is probably -2147483648 in 32bit and 64bit machines.

My two cents.
Gus Correa

Mike Dubman wrote:

Hi,
What interconnect and command line do you use? For InfiniBand openib 
component there is a known issue with large transfers (2GB)


https://svn.open-mpi.org/trac/ompi/ticket/2623

try disabling memory 
pinning: http://www.open-mpi.org/faq/?category=openfabrics#large-message-leave-pinned



regards
M


2010/12/6 >

hi,

In my computers(X86-64), the sizeof(int)=4, but the
sizeof(long)=sizeof(double)=sizeof(size_t)=8. when I checked my
mpi.h file, I found that the definition about the sizeof(int) is
correct. meanwhile, I think the mpi.h file was generated according
to my compute environment when I compiled the Openmpi. So, my codes
still don't work. :(

Further, I found when I called the collective routines(such as,
MPI_Allgatherv(...)) which are implemented by the Point 2 Point
don't work either when the data > 2GB.

Thanks
Xianjun

2010/12/6 Tim Prince >

On 12/5/2010 7:13 PM, Xianjun wrote:

hi,

I met a question recently when I tested the MPI_send and
MPI_Recv
functions. When I run the following codes, the processes
hanged and I
found there was not data transmission in my network at all.

BTW: I finished this test on two X86-64 computers with 16GB
memory and
installed Linux.

1 #include 
2 #include 
3 #include 
4 #include 
5
6
7 int main(int argc, char** argv)
8 {
9 int localID;
10 int numOfPros;
11 size_t Gsize = (size_t)2 * 1024 * 1024 * 1024;
12
13 char* g = (char*)malloc(Gsize);
14
15 MPI_Init(, );
16 MPI_Comm_size(MPI_COMM_WORLD, );
17 MPI_Comm_rank(MPI_COMM_WORLD, );
18
19 MPI_Datatype MPI_Type_lkchar;
20 MPI_Type_contiguous(2048, MPI_BYTE, _Type_lkchar);
21 MPI_Type_commit(_Type_lkchar);
22
23 if (localID == 0)
24 {
25 MPI_Send(g, 1024*1024, MPI_Type_lkchar, 1, 1,
MPI_COMM_WORLD);
26 }
27
28 if (localID != 0)
29 {
30 MPI_Status status;
31 MPI_Recv(g, 1024*1024, MPI_Type_lkchar, 0, 1, \
32 MPI_COMM_WORLD, );
33 }
34
35 MPI_Finalize();
36
37 return 0;
38 }

You supplied all your constants as 32-bit signed data, so, even
if the count for MPI_Send() and MPI_Recv() were a larger data
type, you would see this limit. Did you look at your  ?

-- 
Tim Prince


___
users mailing list
us...@open-mpi.org 
http://www.open-mpi.org/mailman/listinfo.cgi/users



___
users mailing list
us...@open-mpi.org 
http://www.open-mpi.org/mailman/listinfo.cgi/users





___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_Send doesn't work if the data >= 2GB

[Mike Dubman]

Hi,
What interconnect and command line do you use? For InfiniBand openib
component there is a known issue with large transfers (2GB)

https://svn.open-mpi.org/trac/ompi/ticket/2623

try disabling memory pinning:
http://www.open-mpi.org/faq/?category=openfabrics#large-message-leave-pinned


regards
M


2010/12/6 孟宪军 

> hi,
>
> In my computers(X86-64), the sizeof(int)=4, but the
> sizeof(long)=sizeof(double)=sizeof(size_t)=8. when I checked my mpi.h file,
> I found that the definition about the sizeof(int) is correct. meanwhile, I
> think the mpi.h file was generated according to my compute environment when
> I compiled the Openmpi. So, my codes still don't work. :(
>
> Further, I found when I called the collective routines(such as,
> MPI_Allgatherv(...)) which are implemented by the Point 2 Point don't work
> either when the data > 2GB.
>
> Thanks
> Xianjun
>
> 2010/12/6 Tim Prince 
>
> On 12/5/2010 7:13 PM, 孟宪军 wrote:
>>
>>> hi,
>>>
>>> I met a question recently when I tested the MPI_send and MPI_Recv
>>> functions. When I run the following codes, the processes hanged and I
>>> found there was not data transmission in my network at all.
>>>
>>> BTW: I finished this test on two X86-64 computers with 16GB memory and
>>> installed Linux.
>>>
>>> 1 #include 
>>> 2 #include 
>>> 3 #include 
>>> 4 #include 
>>> 5
>>> 6
>>> 7 int main(int argc, char** argv)
>>> 8 {
>>> 9 int localID;
>>> 10 int numOfPros;
>>> 11 size_t Gsize = (size_t)2 * 1024 * 1024 * 1024;
>>> 12
>>> 13 char* g = (char*)malloc(Gsize);
>>> 14
>>> 15 MPI_Init(, );
>>> 16 MPI_Comm_size(MPI_COMM_WORLD, );
>>> 17 MPI_Comm_rank(MPI_COMM_WORLD, );
>>> 18
>>> 19 MPI_Datatype MPI_Type_lkchar;
>>> 20 MPI_Type_contiguous(2048, MPI_BYTE, _Type_lkchar);
>>> 21 MPI_Type_commit(_Type_lkchar);
>>> 22
>>> 23 if (localID == 0)
>>> 24 {
>>> 25 MPI_Send(g, 1024*1024, MPI_Type_lkchar, 1, 1, MPI_COMM_WORLD);
>>> 26 }
>>> 27
>>> 28 if (localID != 0)
>>> 29 {
>>> 30 MPI_Status status;
>>> 31 MPI_Recv(g, 1024*1024, MPI_Type_lkchar, 0, 1, \
>>> 32 MPI_COMM_WORLD, );
>>> 33 }
>>> 34
>>> 35 MPI_Finalize();
>>> 36
>>> 37 return 0;
>>> 38 }
>>>
>>>  You supplied all your constants as 32-bit signed data, so, even if the
>> count for MPI_Send() and MPI_Recv() were a larger data type, you would see
>> this limit. Did you look at your  ?
>>
>> --
>> Tim Prince
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] difference between single and double precision

[Peter Kjellström]

On Monday 06 December 2010 15:03:13 Mathieu Gontier wrote:
> Hi,
> 
> A small update.
> My colleague made a mistake and there is no arithmetic performance
> issue. Sorry for bothering you.
> 
> Nevertheless, one can observed some differences between MPICH and
> OpenMPI from 25% to 100% depending on the options we are using into our
> software. Tests are lead on a single SGI node on 6 or 12 processes, and
> thus, I am focused on the sm option.

A few previous threads on sm performance have been related to what /tmp is. 
OpenMPI relies on (or at least used to rely on) this being backed by page 
cache (tmpfs, a local ext3 or similar). I'm not sure what the behaviour is in 
the latest version but then again you didn't say which version you've tried.

/Peter


signature.asc
Description: This is a digitally signed message part.

[OMPI users] Segmentation fault in mca_pml_ob1.so

[Grzegorz Maj]

Hi,
I'm using mkl scalapack in my project. Recently, I was trying to run
my application on new set of nodes. Unfortunately, when I try to
execute more than about 20 processes, I get segmentation fault.

[compn7:03552] *** Process received signal ***
[compn7:03552] Signal: Segmentation fault (11)
[compn7:03552] Signal code: Address not mapped (1)
[compn7:03552] Failing at address: 0x20b2e68
[compn7:03552] [ 0] /lib64/libpthread.so.0(+0xf3c0) [0x7f46e0fc33c0]
[compn7:03552] [ 1]
/home/gmaj/lib/openmpi/lib/openmpi/mca_pml_ob1.so(+0xd577)
[0x7f46dd093577]
[compn7:03552] [ 2]
/home/gmaj/lib/openmpi/lib/openmpi/mca_btl_tcp.so(+0x5b4c)
[0x7f46dc5edb4c]
[compn7:03552] [ 3]
/home/gmaj/lib/openmpi/lib/libopen-pal.so.0(+0x1dbe8) [0x7f46e0679be8]
[compn7:03552] [ 4]
(home/gmaj/lib/openmpi/lib/libopen-pal.so.0(opal_progress+0xa1)
[0x7f46e066dbf1]
[compn7:03552] [ 5]
/home/gmaj/lib/openmpi/lib/openmpi/mca_pml_ob1.so(+0x5945)
[0x7f46dd08b945]
[compn7:03552] [ 6]
/home/gmaj/lib/openmpi/lib/libmpi.so.0(MPI_Send+0x6a) [0x7f46e0b4f10a]
[compn7:03552] [ 7] /home/gmaj/matrix/matrix(BI_Ssend+0x21) [0x49cc11]
[compn7:03552] [ 8] /home/gmaj/matrix/matrix(BI_IdringBR+0x79) [0x49c579]
[compn7:03552] [ 9] /home/gmaj/matrix/matrix(ilp64_Cdgebr2d+0x221) [0x495bb1]
[compn7:03552] [10] /home/gmaj/matrix/matrix(Cdgebr2d+0xd0) [0x47ffb0]
[compn7:03552] [11]
/home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so(PB_CInV2+0x1304)
[0x7f46e27f5124]
[compn7:03552] *** End of error message ***

This error appears during some scalapack computation. My processes do
some mpi communication before this error appears.

I found out, that by modifying btl_tcp_eager_limit and
btl_tcp_max_send_size parameters, I can run more processes - the
smaller those values are, the more processes I can run. Unfortunately,
by this method I've succeeded to run up to 30 processes, which is
still far to small.

Some clue may be what valgrind says:

==3894== Syscall param writev(vector[...]) points to uninitialised byte(s)
==3894==at 0x82D009B: writev (in /lib64/libc-2.12.90.so)
==3894==by 0xBA2136D: mca_btl_tcp_frag_send (in
/home/gmaj/lib/openmpi/lib/openmpi/mca_btl_tcp.so)
==3894==by 0xBA203D0: mca_btl_tcp_endpoint_send (in
/home/gmaj/lib/openmpi/lib/openmpi/mca_btl_tcp.so)
==3894==by 0xB003583: mca_pml_ob1_send_request_start_rdma (in
/home/gmaj/lib/openmpi/lib/openmpi/mca_pml_ob1.so)
==3894==by 0xAFFA7C9: mca_pml_ob1_send (in
/home/gmaj/lib/openmpi/lib/openmpi/mca_pml_ob1.so)
==3894==by 0x6D4B109: PMPI_Send (in /home/gmaj/lib/openmpi/lib/libmpi.so.0)
==3894==by 0x49CC10: BI_Ssend (in /home/gmaj/matrix/matrix)
==3894==by 0x49C578: BI_IdringBR (in /home/gmaj/matrix/matrix)
==3894==by 0x495BB0: ilp64_Cdgebr2d (in /home/gmaj/matrix/matrix)
==3894==by 0x47FFAF: Cdgebr2d (in /home/gmaj/matrix/matrix)
==3894==by 0x51B38E0: PB_CInV2 (in
/home/gmaj/lib/intel_mkl/10.2.6/lib/em64t/libmkl_scalapack_ilp64.so)
==3894==by 0x51DB89B: PB_CpgemmAB (in
/home/gmaj/lib/intel_mkl/10.2.6/lib/em64t/libmkl_scalapack_ilp64.so)
==3894==  Address 0xadecdce is 461,886 bytes inside a block of size
527,544 alloc'd
==3894==at 0x4C2615D: malloc (vg_replace_malloc.c:195)
==3894==by 0x6D0BBA3: ompi_free_list_grow (in
/home/gmaj/lib/openmpi/lib/libmpi.so.0)
==3894==by 0xBA1E1A4: mca_btl_tcp_component_init (in
/home/gmaj/lib/openmpi/lib/openmpi/mca_btl_tcp.so)
==3894==by 0x6D5C909: mca_btl_base_select (in
/home/gmaj/lib/openmpi/lib/libmpi.so.0)
==3894==by 0xB40E950: mca_bml_r2_component_init (in
/home/gmaj/lib/openmpi/lib/openmpi/mca_bml_r2.so)
==3894==by 0x6D5C07E: mca_bml_base_init (in
/home/gmaj/lib/openmpi/lib/libmpi.so.0)
==3894==by 0xAFF8A0E: mca_pml_ob1_component_init (in
/home/gmaj/lib/openmpi/lib/openmpi/mca_pml_ob1.so)
==3894==by 0x6D663B2: mca_pml_base_select (in
/home/gmaj/lib/openmpi/lib/libmpi.so.0)
==3894==by 0x6D25D20: ompi_mpi_init (in
/home/gmaj/lib/openmpi/lib/libmpi.so.0)
==3894==by 0x6D45987: PMPI_Init_thread (in
/home/gmaj/lib/openmpi/lib/libmpi.so.0)
==3894==by 0x42490A: MPI::Init_thread(int&, char**&, int)
(functions_inln.h:150)
==3894==by 0x41F483: main (matrix.cpp:83)

I've tried to configure open-mpi with option --without-memory-manager,
but it didn't help.

I can successfully run exactly the same application on other machines,
having the number of nodes even over 800.

Does anyone have any idea how to further debug this issue? Any help
would be appreciated.

Thanks,
Grzegorz Maj

Re: [OMPI users] difference between single and double precision

[Mathieu Gontier]

Hi,

A small update.
My colleague made a mistake and there is no arithmetic performance 
issue. Sorry for bothering you.


Nevertheless, one can observed some differences between MPICH and 
OpenMPI from 25% to 100% depending on the options we are using into our 
software. Tests are lead on a single SGI node on 6 or 12 processes, and 
thus, I am focused on the sm option.


So, I have two questions:
1/ does the option--mca mpool_sm_max_size= can change something (I 
am wondering if the value is not too small and, as consequence, a set of 
small messages is sent instead of a big one)
2/ is there a difference between --mca btl tcp,sm,self and --mca btl 
self,sm,tcp (or not put any explicit mca option)?


Best regards,
Mathieu.

On 12/05/2010 06:10 PM, Eugene Loh wrote:

Mathieu Gontier wrote:


  Dear OpenMPI users

I am dealing with an arithmetic problem. In fact, I have two variants 
of my code: one in single precision, one in double precision. When I 
compare the two executable built with MPICH, one can observed an 
expected difference of performance: 115.7-sec in single precision 
against 178.68-sec in double precision (+54%).


The thing is, when I use OpenMPI, the difference is really bigger: 
238.5-sec in single precision against 403.19-sec double precision 
(+69%).


Our experiences have already shown OpenMPI is less efficient than 
MPICH on Ethernet with a small number of processes. This explain the 
differences between the first set of results with MPICH and the 
second set with OpenMPI. (But if someone have more information about 
that or even a solution, I am of course interested.)
But, using OpenMPI increases the difference between the two 
arithmetic. Is it the accentuation of the OpenMPI+Ethernet loss of 
performance, is it another issue into OpenMPI or is there any option 
a can use?


It is also unusual that the performance difference between MPICH and 
OMPI is so large.  You say that OMPI is slower than MPICH even at 
small process counts.  Can you confirm that this is because MPI calls 
are slower?  Some of the biggest performance differences I've seen 
between MPI implementations had nothing to do with the performance of 
MPI calls at all.  It had to do with process binding or other factors 
that impacted the computational (non-MPI) performance of the code.  
The performance of MPI calls was basically irrelevant.


In this particular case, I'm not convinced since neither OMPI nor 
MPICH binds processes by default.


Still, can you do some basic performance profiling to confirm what 
aspect of your application is consuming so much time?  Is it a 
particular MPI call?  If your application is spending almost all of 
its time in MPI calls, do you have some way of judging whether the 
faster performance is acceptable?  That is, is 238 secs acceptable and 
403 secs slow?  Or, are both timings unacceptable -- e.g., the code 
"should" be running in about 30 secs.

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] meaning of MPI_THREAD_*

[Tim Prince]

On 12/6/2010 3:16 AM, Hicham Mouline wrote:

Hello,

1. MPI_THREAD_SINGLE: Only one thread will execute.
Does this really mean the process cannot have any other threads at all, even if 
they doen't deal with MPI at all?
I'm curious as to how this case affects the openmpi implementation?
Essentially, what is the difference between MPI_THREAD_SINGLE and 
MPI_THREAD_FUNNELED?

2. In my case, I'm interested in MPI_THREAD_SERIALIZED. However if it's 
available, I can use MPI_THREAD_FUNNELED.
What cmake flags do I need to enable to allow this mode?

3. Assume I assign only 1 thread in my program to deal with MPI. What is the 
difference between
int MPI::Init_thread(MPI_THREAD_SINGLE)
int MPI::Init_thread(MPI_THREAD_FUNNELED)
int MPI::Init()

You're question is too broad; perhaps you didn't intend it that way.  
Are you trying to do something which may work only with a specific 
version of openmpi, or are you willing to adhere to portable practice?

I tend to believe what it says at
http://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-2.0/node165.htm 


including:
A call to MPI_INIT has the same effect as a call to MPI_INIT_THREAD with 
a required = MPI_THREAD_SINGLE


You would likely use one of those if all your MPI calls are from a 
single thread, and you don't perform any threading inside MPI.  MPI 
implementations vary on the extent to which a higher level of threading 
than what is declared can be used successfully (there's no guarantee of 
bad results if you exceed what was set by MPI_INIT).   There shouldn't 
be any bad effect from setting a higher level of thread support which 
you never use.


I would think your question about cmake flags would apply only once you 
chose a compiler.  I have never seen anyone try mixing 
auto-parallelization with MPI; that would require MPI_THREAD_MULTIPLE 
but still appears unpredictable.  MPI_THREAD_FUNNELED is used often with 
OpenMP parallelization inside MPI.


--
Tim Prince

Re: [OMPI users] Scalability issue

[Gustavo Correa]

Hi Benjamin

I guess you could compile OpenMPI with standard integer and real sizes.
Then compile your application (DRAGON) with the flags to change to 8-byte
integers and 8-byte reals.
We have some programs here that use real8 and are compiled this way,
and run without a problem.
I guess this is what Tim Prince was also telling you in his comments.

You can pass those flags to the MPI compiler wrappers (mpif77 etc),
which will relay them to gfortran when you compile DRAGON.

I am not even sure if those flags would be accepted or ignored by OpenMPI
when you build it.
I guess they will be ignored.
You could check this out by looking at the MPI type sizes in your header
files in the include directory and subdirectories.

Maybe an OpenMPI developer could shed some light here.

Moreover, if I remember right, 
the MPI address type complies with the machine architecture,
i.e., 32 bits if your machine is 32-bit, 64-bits if the machine is 64-bit, 
and you don't need to force it to be 8-bytes with compilation flags.

Unfortunately mixing pointers ("Cray pointers", I suppose) 
with integers is a common source of headaches, if DRAGON does this.
It is yet another possible situation where negative integers could crop in
and lead to segmentation fault.
At least one ocean circulation model we run here had
many problems because of this mix of integers and (Cray) pointers
spread all across the code.

Gus Correa

On Dec 5, 2010, at 7:17 PM, Benjamin Toueg wrote:

> Unfortunately DRAGON is old FORTRAN77. Integers have been used instead of 
> pointers. If I compile it in 64bits without -f-default-integer-8, the 
> so-called pointers will remain in 32bits. Problems could also arise from its 
> data structure handlers.
> 
> Therefore -f-default-integer-8 is absolutely necessary.
> 
> Futhermore MPI_SEND and MPI_RECEIVE are called a dozen times in only one 
> source file (used for passing a data structure from one node to another) and 
> it has proved to be working in every situtation.
> 
> Not knowing which line is causing my segfault is annoying. <323.gif>
> 
> Regards,
> Benjamin
> 
> 2010/12/6 Gustavo Correa 
> Hi Benjamin
> 
> I would just rebuild OpenMPI withOUT the compiler flags that change the 
> standard
> sizes of  "int" and "float" (do a "make cleandist" first!), then recompile 
> your program,
> and see how it goes.
> I don't think you are gaining anything by trying to change the standard 
> "int/integer" and
> "real/float" sizdes, and most likely they are inviting trouble, making things 
> more confusing.
> Worst scenario, you will at least be sure that the bug is somewhere else, not 
> on the mismatch
> of basic type sizes.
> 
> If you need to pass 8-byte real buffers, use MPI_DOUBLE_PRECISION, or 
> MPI_REAL8
> in your (Fortran) MPI calls, and declare them in the Fortran code accordingly
> (double precision or real(kind=8)).
> 
> If I remember right, there is no  8-byte integer support in the Fortran MPI 
> bindings,
> only in the C bindings, but some OpenMPI expert could clarify this.
> Hence, if you are passing 8-byte integers in your MPI calls this may be also 
> problematic.
> 
> My two cents,
> Gus Correa
> 
> On Dec 5, 2010, at 3:04 PM, Benjamin Toueg wrote:
> 
> > Hi,
> >
> > First of all thanks for your insight !
> >
> > Do you get a corefile?
> > I don't get a core file, but I get a file called _FIL001. It doesn't 
> > contain any debugging symbols. It's most likely a digested version of the 
> > input file given to the executable : ./myexec < inputfile.
> >
> > there's no line numbers printed in the stack trace
> > I would love to see those, but even if I compile openmpi with -debug 
> > -mem-debug -mem-profile, they don't show up. I recompiled my sources to be 
> > sure to properly link them to the newly debugged version of openmpi. I 
> > assumed I didn't need to compile my own sources with -g option since it 
> > crashes in openmpi itself ? I didn't try to run mpiexec via gdb either, I 
> > guess it wont help since I already get the trace.
> >
> > the -fdefault-integer-8 options ought to be highly dangerous
> > Thanks for noting. Indeed I had some issues with this option. For instance 
> > I have to declare some arguments as INTEGER*4 like RANK,SIZE,IERR in :
> > CALL MPI_COMM_RANK(MPI_COMM_WORLD,RANK,IERR)
> > CALL MPI_COMM_SIZE(MPI_COMM_WORLD,SIZE,IERR)
> > In your example "call MPI_Send(buf, count, MPI_INTEGER, dest, tag, 
> > MPI_COMM_WORLD, mpierr)" I checked that count is never bigger than 2000 (as 
> > you mentioned it could flip to the negative). However I haven't declared it 
> > as INTEGER*4 and I think I should.
> > When I said "I had to raise the number of data strucutures to be sent", I 
> > meant that I had to call MPI_SEND many more times, not that buffers were 
> > bigger than before.
> >
> > I'll get back to you with more info when I'll be able to fix my connexion 
> > problem to the cluster...
> >
> > Thanks,
> > Benjamin
> >
> > 2010/12/3 Martin Siegert