Re: [OMPI users] MPI_Send doesn't work if the data >= 2GB

[孟宪军]

Hi Bowen,

Thanks very much. I had checked my writev system call, I thought it was him
that caused all these bad things :)


Best Regards
Xianjun Meng

2010/12/8 Bowen Zhou 

> On 12/05/2010 10:13 PM,
>
>> hi,
>>
>> I met a question recently when I tested the MPI_send and MPI_Recv
>> functions.  When I run the following codes, the  processes hanged and I
>> found there was not data transmission in my network at all.
>>
>> BTW: I finished this test on two X86-64 computers with 16GB memory and
>> installed Linux.
>>
>>  1 #include 
>>  2 #include 
>>  3 #include 
>>  4 #include 
>>  5
>>  6
>>  7 int main(int argc, char** argv)
>>  8 {
>>  9   int localID;
>>  10 int numOfPros;
>>  11 size_t Gsize = (size_t)2 * 1024 * 1024 * 1024;
>>  12
>>  13 char* g = (char*)malloc(Gsize);
>>  14
>>  15 MPI_Init(&argc, &argv);
>>  16 MPI_Comm_size(MPI_COMM_WORLD, &numOfPros);
>>  17 MPI_Comm_rank(MPI_COMM_WORLD, &localID);
>>  18
>>  19 MPI_Datatype MPI_Type_lkchar;
>>  20 MPI_Type_contiguous(2048, MPI_BYTE, &MPI_Type_lkchar);
>>  21 MPI_Type_commit(&MPI_Type_lkchar);
>>  22
>>  23 if (localID == 0)
>>  24 {
>>  25 MPI_Send(g, 1024*1024, MPI_Type_lkchar, 1, 1, MPI_COMM_WORLD);
>>  26 }
>>  27
>>  28 if (localID != 0)
>>  29 {
>>  30 MPI_Status status;
>>  31 MPI_Recv(g, 1024*1024, MPI_Type_lkchar, 0, 1, \
>>  32 MPI_COMM_WORLD, &status);
>>  33 }
>>  34
>>  35 MPI_Finalize();
>>  36
>>  37 return 0;
>>  38 }
>>
>> Thanks
>> Xianjun
>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
> Hi,
>
> It might be a problem with the Linux TCP stack. The similar issue happens
> with MPICH2 too. Can you try with message size (2G-1) bytes?
> http://trac.mcs.anl.gov/projects/mpich2/ticket/1080
>
> Best,
> Bowen Zhou
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] glut display 'occasionally' opens

[Ralph Castain]

FWIW: I just tested the -x option on a multi-node system and had no problem 
getting the value of DISPLAY to propagate. I was able to define it on the cmd 
line, saw it set correctly on every process, etc.

This was with our devel trunk - not sure what version you are using.


On Dec 7, 2010, at 12:12 PM, brad baker wrote:

> Thanks for your responses!  I'm at home today so I can't actually do any 
> tests to 'see' if anything works. But I logged in remotely and I did as Ralph 
> suggested and ran env as my app.  No process returned a value for DISPLAY.  
> Then I made a small program that calls getenv("DISPLAY") to run with mpi, and 
> each process returns NULL.
> 
> I did some googling and found in the mpirun man page:
> 
> "Exported Environment Variables
> The -x option to mpirun can be used to export specific environment variables 
> to the new processes. While the syntax of the -x option allows the definition 
> of new variables, note that the parser for this option is currently not very 
> sophisticated - it does not even understand quoted values. Users are advised 
> to set variables in the environment and use -x to export them; not to define 
> them."
> So it looks like I need to manually set them, possible how Jeff suggested.  
> I'll do some more research on this and get back after I've tried a few things 
> in the lab.
> 
> Thanks again!
> Brad
> 
> 
> On Tue, Dec 7, 2010 at 10:26 AM, Jeff Squyres  wrote:
> Are you using ssh to launch OMPI between your nodes?  (i.e., is mpirun using 
> ssh under the covers to launch on remote nodes)
> 
> If so, you might want to just set OMPI to use "ssh -X", which sets up SSH 
> tunneled X forwarding, and therefore it sets DISPLAY for you properly on all 
> the remote nodes automatically.  But it does have the disadvantage of being a 
> bit slow, since it's coming through ssh.
> 
> Alternatively, you can xhost +, where  is the host 
> where your X app is running.  Then set your DISPLAY variable manually to 
> :display and it'll just go in an unencrypted fashion.  This is 
> normal X forwarding stuff -- you can probably google around for more info on 
> this.
> 
> NOTE: IIRC, xauth is better than xhost these days.  I stopped using X for 
> most things many years ago, so my xhost/xauth information is probably a 
> little dated.  Google around for the most recent / best ways to do this stuff.
> 
> 
> On Dec 6, 2010, at 10:11 PM, Ralph Castain wrote:
> 
> > BTW: you might check to see if the DISPLAY envar is being correctly set on 
> > all procs. Two ways to do it:
> >
> > 1. launch "env" as your app to print out the envars - can be messy on the 
> > output end, though you could use the mpirun options to tag and/or split the 
> > output from the procs
> >
> > 2. in your app, just do a getenv and print the display envar
> >
> > Would help tell us if there is an OMPI problem, or just a problem in how 
> > you setup X11
> >
> >
> > On Dec 6, 2010, at 9:18 PM, Ralph Castain wrote:
> >
> >> Hmmm...yes, the code does seem to handle that '=' being in there. Forgot 
> >> it was there.
> >>
> >> Depending on the version you are using, mpirun could just open the display 
> >> for you. There is an mpirun option that tells us to please start each app 
> >> in its own xterm.
> >>
> >> You shouldn't need forwarding if you are going to see it on a local 
> >> display (i.e., one physically attached to the node), assuming you are 
> >> logged into those nodes (otherwise you don't own the display).
> >>
> >> If you are trying to view it on your own local display, then you do need 
> >> forwarding setup.
> >>
> >>
> >> On Dec 6, 2010, at 8:36 PM, brad baker wrote:
> >>
> >>> Without including the -x DISPLAY, glut doesn't know what display to open. 
> >>>  For instance, without the -x DISPLAY parameter glut returns an error 
> >>> from each process stating that it could not find display "" (empty 
> >>> string).  This strategy is briefly described in the openmpi   FAQs for 
> >>> launching gui applications with openmpi.
> >>>
> >>> I'm assuming that by setting the DISPLAY envar to :0.0, each process will 
> >>> render to their local display, which is my intention, and as I previously 
> >>> stated works for up to 2 processes. So I believe it to be necessary.
> >>>
> >>> But I'm thinking I may have to configure some kind of  X11 forwarding.  
> >>> I'm not sure...
> >>>
> >>> Thanks for your reply! Any more ideas?
> >>> Brad
> >>>
> >>>
> >>> On Mon, Dec 6, 2010 at 6:31 PM, Ralph Castain  wrote:
> >>> Guess I'm not entirely sure I understand how this is supposed to work. 
> >>> All the -x does is tell us to pickup an envar of the given name and 
> >>> forward its value to the remote apps. You can't set the envar's value on 
> >>> the cmd line. So you told mpirun to pickup the value of an envar called 
> >>> "DISPLAY=:0.0".
> >>>
> >>> So yes - I would expect this would be behaving strangely.
> >>>
> >>> If you tell us -x DISPLAY, we'll pickup the local value of DISPLAY and 
> >>> forwa

Re: [OMPI users] MPI_Send doesn't work if the data >= 2GB

[Bowen Zhou]

On 12/05/2010 10:13 PM,

hi,

I met a question recently when I tested the MPI_send and MPI_Recv
functions.  When I run the following codes, the  processes hanged and I
found there was not data transmission in my network at all.

BTW: I finished this test on two X86-64 computers with 16GB memory and
installed Linux.

  1 #include 
  2 #include 
  3 #include 
  4 #include 
  5
  6
  7 int main(int argc, char** argv)
  8 {
  9   int localID;
  10 int numOfPros;
  11 size_t Gsize = (size_t)2 * 1024 * 1024 * 1024;
  12
  13 char* g = (char*)malloc(Gsize);
  14
  15 MPI_Init(&argc, &argv);
  16 MPI_Comm_size(MPI_COMM_WORLD, &numOfPros);
  17 MPI_Comm_rank(MPI_COMM_WORLD, &localID);
  18
  19 MPI_Datatype MPI_Type_lkchar;
  20 MPI_Type_contiguous(2048, MPI_BYTE, &MPI_Type_lkchar);
  21 MPI_Type_commit(&MPI_Type_lkchar);
  22
  23 if (localID == 0)
  24 {
  25 MPI_Send(g, 1024*1024, MPI_Type_lkchar, 1, 1, MPI_COMM_WORLD);
  26 }
  27
  28 if (localID != 0)
  29 {
  30 MPI_Status status;
  31 MPI_Recv(g, 1024*1024, MPI_Type_lkchar, 0, 1, \
  32 MPI_COMM_WORLD, &status);
  33 }
  34
  35 MPI_Finalize();
  36
  37 return 0;
  38 }

Thanks
Xianjun



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Hi,

It might be a problem with the Linux TCP stack. The similar issue 
happens with MPICH2 too. Can you try with message size (2G-1) bytes?

http://trac.mcs.anl.gov/projects/mpich2/ticket/1080

Best,
Bowen Zhou

Re: [OMPI users] glut display 'occasionally' opens

[brad baker]

Thanks for your responses!  I'm at home today so I can't actually do any
tests to 'see' if anything works. But I logged in remotely and I did as
Ralph suggested and ran env as my app.  No process returned a value for
DISPLAY.  Then I made a small program that calls getenv("DISPLAY") to run
with mpi, and each process returns NULL.

I did some googling and found in the mpirun man
page
:

"*Exported Environment Variables*
The *-x* option to *mpirun* can be used to export specific environment
variables to the new processes. While the syntax of the *-x* option allows
the definition of new variables, note that the* parser for this option is
currently not very sophisticated* - it does not even understand quoted
values. Users are advised to* set variables in the environment and use **-x*
* to export them; not to define them*."
So it looks like I need to manually set them, possible how Jeff suggested.
 I'll do some more research on this and get back after I've tried a few
things in the lab.

Thanks again!
Brad


On Tue, Dec 7, 2010 at 10:26 AM, Jeff Squyres  wrote:

> Are you using ssh to launch OMPI between your nodes?  (i.e., is mpirun
> using ssh under the covers to launch on remote nodes)
>
> If so, you might want to just set OMPI to use "ssh -X", which sets up SSH
> tunneled X forwarding, and therefore it sets DISPLAY for you properly on all
> the remote nodes automatically.  But it does have the disadvantage of being
> a bit slow, since it's coming through ssh.
>
> Alternatively, you can xhost +, where  is the
> host where your X app is running.  Then set your DISPLAY variable manually
> to :display and it'll just go in an unencrypted fashion.  This
> is normal X forwarding stuff -- you can probably google around for more info
> on this.
>
> NOTE: IIRC, xauth is better than xhost these days.  I stopped using X for
> most things many years ago, so my xhost/xauth information is probably a
> little dated.  Google around for the most recent / best ways to do this
> stuff.
>
>
> On Dec 6, 2010, at 10:11 PM, Ralph Castain wrote:
>
> > BTW: you might check to see if the DISPLAY envar is being correctly set
> on all procs. Two ways to do it:
> >
> > 1. launch "env" as your app to print out the envars - can be messy on the
> output end, though you could use the mpirun options to tag and/or split the
> output from the procs
> >
> > 2. in your app, just do a getenv and print the display envar
> >
> > Would help tell us if there is an OMPI problem, or just a problem in how
> you setup X11
> >
> >
> > On Dec 6, 2010, at 9:18 PM, Ralph Castain wrote:
> >
> >> Hmmm...yes, the code does seem to handle that '=' being in there. Forgot
> it was there.
> >>
> >> Depending on the version you are using, mpirun could just open the
> display for you. There is an mpirun option that tells us to please start
> each app in its own xterm.
> >>
> >> You shouldn't need forwarding if you are going to see it on a local
> display (i.e., one physically attached to the node), assuming you are logged
> into those nodes (otherwise you don't own the display).
> >>
> >> If you are trying to view it on your own local display, then you do need
> forwarding setup.
> >>
> >>
> >> On Dec 6, 2010, at 8:36 PM, brad baker wrote:
> >>
> >>> Without including the -x DISPLAY, glut doesn't know what display to
> open.  For instance, without the -x DISPLAY parameter glut returns an error
> from each process stating that it could not find display "" (empty string).
>  This strategy is briefly described in the openmpi   FAQs for launching gui
> applications with openmpi.
> >>>
> >>> I'm assuming that by setting the DISPLAY envar to :0.0, each process
> will render to their local display, which is my intention, and as I
> previously stated works for up to 2 processes. So I believe it to be
> necessary.
> >>>
> >>> But I'm thinking I may have to configure some kind of  X11 forwarding.
>  I'm not sure...
> >>>
> >>> Thanks for your reply! Any more ideas?
> >>> Brad
> >>>
> >>>
> >>> On Mon, Dec 6, 2010 at 6:31 PM, Ralph Castain 
> wrote:
> >>> Guess I'm not entirely sure I understand how this is supposed to work.
> All the -x does is tell us to pickup an envar of the given name and forward
> its value to the remote apps. You can't set the envar's value on the cmd
> line. So you told mpirun to pickup the value of an envar called
> "DISPLAY=:0.0".
> >>>
> >>> So yes - I would expect this would be behaving strangely.
> >>>
> >>> If you tell us -x DISPLAY, we'll pickup the local value of DISPLAY and
> forward it. What that will cause your app to do is, I suppose, up to it.
> >>>
> >>>
> >>> On Dec 6, 2010, at 12:42 PM, brad baker wrote:
> >>>
> >>> > Hello,
> >>> >
> >>> > I'm working on an mpi application that opens a glut display on each
> node of a small cluster for opengl rendering (each node has its own
> display). My current implementation scales great with mpich2, but I'd like
> to use openmpi infiniband, which is giving me trou

Re: [OMPI users] glut display 'occasionally' opens

[Jeff Squyres]

Are you using ssh to launch OMPI between your nodes?  (i.e., is mpirun using 
ssh under the covers to launch on remote nodes)

If so, you might want to just set OMPI to use "ssh -X", which sets up SSH 
tunneled X forwarding, and therefore it sets DISPLAY for you properly on all 
the remote nodes automatically.  But it does have the disadvantage of being a 
bit slow, since it's coming through ssh.

Alternatively, you can xhost +, where  is the host 
where your X app is running.  Then set your DISPLAY variable manually to 
:display and it'll just go in an unencrypted fashion.  This is 
normal X forwarding stuff -- you can probably google around for more info on 
this.

NOTE: IIRC, xauth is better than xhost these days.  I stopped using X for most 
things many years ago, so my xhost/xauth information is probably a little 
dated.  Google around for the most recent / best ways to do this stuff.


On Dec 6, 2010, at 10:11 PM, Ralph Castain wrote:

> BTW: you might check to see if the DISPLAY envar is being correctly set on 
> all procs. Two ways to do it:
> 
> 1. launch "env" as your app to print out the envars - can be messy on the 
> output end, though you could use the mpirun options to tag and/or split the 
> output from the procs
> 
> 2. in your app, just do a getenv and print the display envar
> 
> Would help tell us if there is an OMPI problem, or just a problem in how you 
> setup X11
> 
> 
> On Dec 6, 2010, at 9:18 PM, Ralph Castain wrote:
> 
>> Hmmm...yes, the code does seem to handle that '=' being in there. Forgot it 
>> was there.
>> 
>> Depending on the version you are using, mpirun could just open the display 
>> for you. There is an mpirun option that tells us to please start each app in 
>> its own xterm.
>> 
>> You shouldn't need forwarding if you are going to see it on a local display 
>> (i.e., one physically attached to the node), assuming you are logged into 
>> those nodes (otherwise you don't own the display).
>> 
>> If you are trying to view it on your own local display, then you do need 
>> forwarding setup.
>> 
>> 
>> On Dec 6, 2010, at 8:36 PM, brad baker wrote:
>> 
>>> Without including the -x DISPLAY, glut doesn't know what display to open.  
>>> For instance, without the -x DISPLAY parameter glut returns an error from 
>>> each process stating that it could not find display "" (empty string).  
>>> This strategy is briefly described in the openmpi   FAQs for launching gui 
>>> applications with openmpi. 
>>> 
>>> I'm assuming that by setting the DISPLAY envar to :0.0, each process will 
>>> render to their local display, which is my intention, and as I previously 
>>> stated works for up to 2 processes. So I believe it to be necessary.  
>>> 
>>> But I'm thinking I may have to configure some kind of  X11 forwarding.  I'm 
>>> not sure...
>>> 
>>> Thanks for your reply! Any more ideas?
>>> Brad
>>> 
>>> 
>>> On Mon, Dec 6, 2010 at 6:31 PM, Ralph Castain  wrote:
>>> Guess I'm not entirely sure I understand how this is supposed to work. All 
>>> the -x does is tell us to pickup an envar of the given name and forward its 
>>> value to the remote apps. You can't set the envar's value on the cmd line. 
>>> So you told mpirun to pickup the value of an envar called "DISPLAY=:0.0".
>>> 
>>> So yes - I would expect this would be behaving strangely.
>>> 
>>> If you tell us -x DISPLAY, we'll pickup the local value of DISPLAY and 
>>> forward it. What that will cause your app to do is, I suppose, up to it.
>>> 
>>> 
>>> On Dec 6, 2010, at 12:42 PM, brad baker wrote:
>>> 
>>> > Hello,
>>> >
>>> > I'm working on an mpi application that opens a glut display on each node 
>>> > of a small cluster for opengl rendering (each node has its own display). 
>>> > My current implementation scales great with mpich2, but I'd like to use 
>>> > openmpi infiniband, which is giving me trouble.
>>> >
>>> > I've had some success with the -x DISPLAY=:0.0 parameter to mpirun, which 
>>> > will open the display on up to 2 of my nodes... any 2.  But when I 
>>> > attempt to run the application on 4 nodes, the display is 
>>> > non-deterministic. If any open at all process 0 definately will, and 
>>> > sometimes process 3 along with that.  I haven't observed much behavior 
>>> > from process 1 or 2.
>>> >
>>> > My command:
>>> >
>>> > mpirun -x DISPLAY=:0.0 -np 4 -hostfile ~/openmpi.hosts ./myapp
>>> >
>>> > I've tried adding the -d option with no success.
>>> >
>>> > Does anyone have any suggestions or comments?  I'll certainly provide 
>>> > more information if required.
>>> >
>>> > Thanks,
>>> > Brad
>>> > ___
>>> > users mailing list
>>> > us...@open-mpi.org
>>> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> 
>>> 
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> 
>>> ___
>>> users mailing list
>>> us...@open-mpi

Re: [OMPI users] Scalability issue

[Jeff Squyres]

On Dec 7, 2010, at 8:33 AM, Gus Correa wrote:

> Did I understand you right?
> 
> Are you saying that one can effectively double the counting
> capability (i.e. the "count" parameters in MPI calls) of OpenMPI
> by compiling it with 8-byte integer flags?

Yes and no.

If you increase the size of INTEGER *and* int, then hypothetically yes -- 
although I literally just got a report from someone today that tried a compiler 
flag to increase the size of C int to 8 bytes and something didn't work right 
(I don't think we've ever tried this before, so it's not surprising that there 
are likely some bugs in there).  

We have previously tested the increase-the-sizeof-INTEGER-to-8-bytes compiler 
flags and AFAIK, that's still working fine.

When you call MPI_SEND with an INTEGER count, OMPI will truncate it down to the 
size of a C int (if we had 8 byte C ints working, this might be a different 
story).  But keep in mind that increasing the size of C ints will likely cause 
problems in other areas -- are OS system calls that take int parameters firmly 
sized (i.e., int32 and the like)?  I'm not so sure -- indeed, that might even 
be (one of the) problem(s) with the report that I got earlier today...

> And long as one consistently uses the same flags to compile
> the application, everything would work smoothly?

I always recommend using the same flags for compiling OMPI as compiling your 
application.  Of course, you can vary some flags that don't matter (e.g., 
compiling your app with -g and compiling OMPI with -Ox).  But for "significant" 
behavior changes (like changing the size of INTEGER), they should definitely 
match between your app and OMPI.

> As per several previous discussions here in the list,
> I was persuaded to believe that MPI_INT / MPI_INTEGER is written
> in stone to be 4-bytes (perhaps by MPI standard,
> perhaps the configure script, maybe by both),

Neither, actually.  :-)

The MPI spec is very, very careful not to mandate the size of int or INTEGER at 
all.

> and that "counts" in [Open]MPI would also be restricted to that size
> i.e., effectively up to 2147483647, if I counted right.

*Most* commodity systems (excluding the embedded world) have 4 byte int's these 
days, in part because most systems are this way (i.e., momentum).  
Hence, when we talk about the 2B count limit, we're referring to the fact that 
most systems where MPI is used default to 4 byte int's.

> I may have inadvertently misled Benjamin, if this perception is wrong.
> I will gladly stand corrected, if this is so.
> 
> You are the OpenMPI user's oracle (oops, sorry Cisco),
> so please speak out.

Please buy Cisco stuff!  :-p

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Scalability issue

[Gus Correa]

Hi Jeff

Did I understand you right?

Are you saying that one can effectively double the counting
capability (i.e. the "count" parameters in MPI calls) of OpenMPI
by compiling it with 8-byte integer flags?

And long as one consistently uses the same flags to compile
the application, everything would work smoothly?

As per several previous discussions here in the list,
I was persuaded to believe that MPI_INT / MPI_INTEGER is written
in stone to be 4-bytes (perhaps by MPI standard,
perhaps the configure script, maybe by both),
and that "counts" in [Open]MPI would also be restricted to that size
i.e., effectively up to 2147483647, if I counted right.
I may have inadvertently misled Benjamin, if this perception is wrong.
I will gladly stand corrected, if this is so.

You are the OpenMPI user's oracle (oops, sorry Cisco),
so please speak out.

Cheers,
Gus Correa

Jeff Squyres wrote:
It is always a good idea to have your application's sizeof(INTEGER) 
match the MPI's sizeof(INTEGER).  Having them mismatch is a recipe for trouble.


Meaning: if you're compiling your app with -make-integer-be-8-bytes, 
then you should configure/build Open MPI with that same flag.


I'm thinking that this should *only* affect the back-end behavior 
of MPI_INTEGER; the size of address pointers and whatnot should not be 
affected (unless -make-integer-be-8-bytes also changes 
the sizes of some other types).



On Dec 5, 2010, at 9:01 PM, Gustavo Correa wrote:


Hi Benjamin

I guess you could compile OpenMPI with standard integer and real sizes.
Then compile your application (DRAGON) with the flags to change to 8-byte
integers and 8-byte reals.
We have some programs here that use real8 and are compiled this way,
and run without a problem.
I guess this is what Tim Prince was also telling you in his comments.

You can pass those flags to the MPI compiler wrappers (mpif77 etc),
which will relay them to gfortran when you compile DRAGON.

I am not even sure if those flags would be accepted or ignored by OpenMPI
when you build it.
I guess they will be ignored.
You could check this out by looking at the MPI type sizes in your header
files in the include directory and subdirectories.

Maybe an OpenMPI developer could shed some light here.

Moreover, if I remember right, 
the MPI address type complies with the machine architecture,
i.e., 32 bits if your machine is 32-bit, 64-bits if the machine is 64-bit, 
and you don't need to force it to be 8-bytes with compilation flags.


Unfortunately mixing pointers ("Cray pointers", I suppose) 
with integers is a common source of headaches, if DRAGON does this.

It is yet another possible situation where negative integers could crop in
and lead to segmentation fault.
At least one ocean circulation model we run here had
many problems because of this mix of integers and (Cray) pointers
spread all across the code.

Gus Correa

On Dec 5, 2010, at 7:17 PM, Benjamin Toueg wrote:


Unfortunately DRAGON is old FORTRAN77. Integers have been used instead of 
pointers. If I compile it in 64bits without -f-default-integer-8, the so-called 
pointers will remain in 32bits. Problems could also arise from its data 
structure handlers.

Therefore -f-default-integer-8 is absolutely necessary.

Futhermore MPI_SEND and MPI_RECEIVE are called a dozen times in only one source 
file (used for passing a data structure from one node to another) and it has 
proved to be working in every situtation.

Not knowing which line is causing my segfault is annoying. <323.gif>

Regards,
Benjamin

2010/12/6 Gustavo Correa 
Hi Benjamin

I would just rebuild OpenMPI withOUT the compiler flags that change the standard
sizes of  "int" and "float" (do a "make cleandist" first!), then recompile your 
program,
and see how it goes.
I don't think you are gaining anything by trying to change the standard 
"int/integer" and
"real/float" sizdes, and most likely they are inviting trouble, making things 
more confusing.
Worst scenario, you will at least be sure that the bug is somewhere else, not 
on the mismatch
of basic type sizes.

If you need to pass 8-byte real buffers, use MPI_DOUBLE_PRECISION, or MPI_REAL8
in your (Fortran) MPI calls, and declare them in the Fortran code accordingly
(double precision or real(kind=8)).

If I remember right, there is no  8-byte integer support in the Fortran MPI 
bindings,
only in the C bindings, but some OpenMPI expert could clarify this.
Hence, if you are passing 8-byte integers in your MPI calls this may be also 
problematic.

My two cents,
Gus Correa

On Dec 5, 2010, at 3:04 PM, Benjamin Toueg wrote:


Hi,

First of all thanks for your insight !

Do you get a corefile?
I don't get a core file, but I get a file called _FIL001. It doesn't contain any 
debugging symbols. It's most likely a digested version of the input file given to 
the executable : ./myexec < inputfile.

there's no line numbers printed in the stack trace
I would love to see those, but even if I compile openmpi with -debug -mem-debug 
-mem-profi

Re: [OMPI users] Segmentation fault in mca_pml_ob1.so

[Grzegorz Maj]

I recompiled MPI with -g, but it didn't solve the problem. Two things that
have changed are: buf in PMPI_Recv is no longer of value 0 and backtrace in
gdb shows more functions (eg. mca_pml_ob1_recv_frag_callback_put as #1).

As you recommended, I will try to walk up the stack, but it's not so easy
for me to follow this code.

This is the backtrace I got with -g:
-
Program received signal SIGSEGV, Segmentation fault.
0x7f1f1a11e4eb in mca_pml_ob1_send_request_put (sendreq=0x1437b00,
btl=0xdae850, hdr=0xeb4870) at pml_ob1_sendreq.c:1231
1231 pml_ob1_sendreq.c: No such file or directory.
 in pml_ob1_sendreq.c
(gdb) bt
#0  0x7f1f1a11e4eb in mca_pml_ob1_send_request_put (sendreq=0x1437b00,
btl=0xdae850, hdr=0xeb4870) at pml_ob1_sendreq.c:1231
#1  0x7f1f1a1124de in mca_pml_ob1_recv_frag_callback_put (btl=0xdae850,
tag=72 'H', des=0x7f1f1ff6bb00, cbdata=0x0) at pml_ob1_recvfrag.c:361
#2  0x7f1f19660e0f in mca_btl_tcp_endpoint_recv_handler (sd=24, flags=2,
user=0xe2ab40) at btl_tcp_endpoint.c:718
#3  0x7f1f1d74aa5b in event_process_active (base=0xd82af0) at
event.c:651
#4  0x7f1f1d74b087 in opal_event_base_loop (base=0xd82af0, flags=2) at
event.c:823
#5  0x7f1f1d74ac76 in opal_event_loop (flags=2) at event.c:730
#6  0x7f1f1d73a360 in opal_progress () at runtime/opal_progress.c:189
#7  0x7f1f1a10c0af in opal_condition_wait (c=0x7f1f1df3a5c0,
m=0x7f1f1df3a620) at ../../../../opal/threads/condition.h:99
#8  0x7f1f1a10bef1 in ompi_request_wait_completion (req=0xe1eb00) at
../../../../ompi/request/request.h:375
#9  0x7f1f1a10bdb5 in mca_pml_ob1_recv (addr=0x7f1f1a083080, count=1,
datatype=0xeb3da0, src=-1, tag=0, comm=0xeb0cd0, status=0xe43f00) at
pml_ob1_irecv.c:104
#10 0x7f1f1dc9e324 in PMPI_Recv (buf=0x7f1f1a083080, count=1,
type=0xeb3da0, source=-1, tag=0, comm=0xeb0cd0, status=0xe43f00) at
precv.c:75
#11 0x0049cc43 in BI_Srecv ()
#12 0x0049c555 in BI_IdringBR ()
#13 0x00495ba1 in ilp64_Cdgebr2d ()
#14 0x0047ffa0 in Cdgebr2d ()
#15 0x7f1f1f99c8e1 in PB_CInV2 () from
/home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so
#16 0x7f1f1f9c489c in PB_CpgemmAB () from
/home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so
#17 0x7f1f1fa748fd in pdgemm_ () from
/home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so
-

Thanks,
Grzegorz Maj



2010/12/7 Terry Dontje 

>  I am not sure this has anything to do with your problem but if you look at
> the stack entry for PMPI_Recv I noticed the buf has a value of 0.  Shouldn't
> that be an address?
>
> Does your code fail if the MPI library is built with -g?  If it does fail
> the same way, the next step I would do would be to walk up the stack and try
> and figure out where the sendreq address is coming from because supposedly
> it is that address that is not mapped according to the original stack.
>
> --td
>
>
> On 12/07/2010 08:29 AM, Grzegorz Maj wrote:
>
> Some update on this issue. I've attached gdb to the crashing
> application and I got:
>
> -
> Program received signal SIGSEGV, Segmentation fault.
> mca_pml_ob1_send_request_put (sendreq=0x130c480, btl=0xc49850,
> hdr=0xd10e60) at pml_ob1_sendreq.c:1231
> 1231  pml_ob1_sendreq.c: No such file or directory.
>   in pml_ob1_sendreq.c
> (gdb) bt
> #0  mca_pml_ob1_send_request_put (sendreq=0x130c480, btl=0xc49850,
> hdr=0xd10e60) at pml_ob1_sendreq.c:1231
> #1  0x7fc55bf31693 in mca_btl_tcp_endpoint_recv_handler (sd= optimized out>, flags=, user= out>) at btl_tcp_endpoint.c:718
> #2  0x7fc55fff7de4 in event_process_active (base=0xc1daf0,
> flags=2) at event.c:651
> #3  opal_event_base_loop (base=0xc1daf0, flags=2) at event.c:823
> #4  0x7fc55ffe9ff1 in opal_progress () at runtime/opal_progress.c:189
> #5  0x7fc55c9d7115 in opal_condition_wait (addr= out>, count=, datatype=,
> src=, tag=,
> comm=, status=0xcc6100) at
> ../../../../opal/threads/condition.h:99
> #6  ompi_request_wait_completion (addr=,
> count=, datatype=,
> src=, tag=,
> comm=, status=0xcc6100) at
> ../../../../ompi/request/request.h:375
> #7  mca_pml_ob1_recv (addr=, count= optimized out>, datatype=, src= out>, tag=, comm=,
> status=0xcc6100) at pml_ob1_irecv.c:104
> #8  0x7fc560511260 in PMPI_Recv (buf=0x0, count=12884048,
> type=0xd10410, source=-1, tag=0, comm=0xd0daa0, status=0xcc6100) at
> precv.c:75
> #9  0x0049cc43 in BI_Srecv ()
> #10 0x0049c555 in BI_IdringBR ()
> #11 0x00495ba1 in ilp64_Cdgebr2d ()
> #12 0x0047ffa0 in Cdgebr2d ()
> #13 0x7fc5621da8e1 in PB_CInV2 () from
> /home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so
> #14 0x7fc56220289c in PB_CpgemmAB () from
> /home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so
> #15 0x7fc5622b28fd in pdgemm_ () from
> /home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so
> -
>
> So this looks like the line responsible for segmentation fa

Re: [OMPI users] Scalability issue

[Jeff Squyres]

It is always a good idea to have your application's sizeof(INTEGER) match the 
MPI's sizeof(INTEGER).  Having them mismatch is a recipe for trouble.

Meaning: if you're compiling your app with -make-integer-be-8-bytes, then you 
should configure/build Open MPI with that same flag.

I'm thinking that this should *only* affect the back-end behavior of 
MPI_INTEGER; the size of address pointers and whatnot should not be affected 
(unless -make-integer-be-8-bytes also changes the sizes of some other types).


On Dec 5, 2010, at 9:01 PM, Gustavo Correa wrote:

> Hi Benjamin
> 
> I guess you could compile OpenMPI with standard integer and real sizes.
> Then compile your application (DRAGON) with the flags to change to 8-byte
> integers and 8-byte reals.
> We have some programs here that use real8 and are compiled this way,
> and run without a problem.
> I guess this is what Tim Prince was also telling you in his comments.
> 
> You can pass those flags to the MPI compiler wrappers (mpif77 etc),
> which will relay them to gfortran when you compile DRAGON.
> 
> I am not even sure if those flags would be accepted or ignored by OpenMPI
> when you build it.
> I guess they will be ignored.
> You could check this out by looking at the MPI type sizes in your header
> files in the include directory and subdirectories.
> 
> Maybe an OpenMPI developer could shed some light here.
> 
> Moreover, if I remember right, 
> the MPI address type complies with the machine architecture,
> i.e., 32 bits if your machine is 32-bit, 64-bits if the machine is 64-bit, 
> and you don't need to force it to be 8-bytes with compilation flags.
> 
> Unfortunately mixing pointers ("Cray pointers", I suppose) 
> with integers is a common source of headaches, if DRAGON does this.
> It is yet another possible situation where negative integers could crop in
> and lead to segmentation fault.
> At least one ocean circulation model we run here had
> many problems because of this mix of integers and (Cray) pointers
> spread all across the code.
> 
> Gus Correa
> 
> On Dec 5, 2010, at 7:17 PM, Benjamin Toueg wrote:
> 
>> Unfortunately DRAGON is old FORTRAN77. Integers have been used instead of 
>> pointers. If I compile it in 64bits without -f-default-integer-8, the 
>> so-called pointers will remain in 32bits. Problems could also arise from its 
>> data structure handlers.
>> 
>> Therefore -f-default-integer-8 is absolutely necessary.
>> 
>> Futhermore MPI_SEND and MPI_RECEIVE are called a dozen times in only one 
>> source file (used for passing a data structure from one node to another) and 
>> it has proved to be working in every situtation.
>> 
>> Not knowing which line is causing my segfault is annoying. <323.gif>
>> 
>> Regards,
>> Benjamin
>> 
>> 2010/12/6 Gustavo Correa 
>> Hi Benjamin
>> 
>> I would just rebuild OpenMPI withOUT the compiler flags that change the 
>> standard
>> sizes of  "int" and "float" (do a "make cleandist" first!), then recompile 
>> your program,
>> and see how it goes.
>> I don't think you are gaining anything by trying to change the standard 
>> "int/integer" and
>> "real/float" sizdes, and most likely they are inviting trouble, making 
>> things more confusing.
>> Worst scenario, you will at least be sure that the bug is somewhere else, 
>> not on the mismatch
>> of basic type sizes.
>> 
>> If you need to pass 8-byte real buffers, use MPI_DOUBLE_PRECISION, or 
>> MPI_REAL8
>> in your (Fortran) MPI calls, and declare them in the Fortran code accordingly
>> (double precision or real(kind=8)).
>> 
>> If I remember right, there is no  8-byte integer support in the Fortran MPI 
>> bindings,
>> only in the C bindings, but some OpenMPI expert could clarify this.
>> Hence, if you are passing 8-byte integers in your MPI calls this may be also 
>> problematic.
>> 
>> My two cents,
>> Gus Correa
>> 
>> On Dec 5, 2010, at 3:04 PM, Benjamin Toueg wrote:
>> 
>>> Hi,
>>> 
>>> First of all thanks for your insight !
>>> 
>>> Do you get a corefile?
>>> I don't get a core file, but I get a file called _FIL001. It doesn't 
>>> contain any debugging symbols. It's most likely a digested version of the 
>>> input file given to the executable : ./myexec < inputfile.
>>> 
>>> there's no line numbers printed in the stack trace
>>> I would love to see those, but even if I compile openmpi with -debug 
>>> -mem-debug -mem-profile, they don't show up. I recompiled my sources to be 
>>> sure to properly link them to the newly debugged version of openmpi. I 
>>> assumed I didn't need to compile my own sources with -g option since it 
>>> crashes in openmpi itself ? I didn't try to run mpiexec via gdb either, I 
>>> guess it wont help since I already get the trace.
>>> 
>>> the -fdefault-integer-8 options ought to be highly dangerous
>>> Thanks for noting. Indeed I had some issues with this option. For instance 
>>> I have to declare some arguments as INTEGER*4 like RANK,SIZE,IERR in :
>>> CALL MPI_COMM_RANK(MPI_COMM_WORLD,RANK,IERR)
>>> CALL MPI_COMM_

Re: [OMPI users] Interaction with boost::asio

[Ralph Castain]

You might want to ask the boost people - we wouldn't have any idea what asio is 
or does.

On Dec 7, 2010, at 6:06 AM, Hannes Brandstätter-Müller wrote:

> Hello!
> 
> I am using OpenMPI in combination with the boost libraries (especially 
> boost::asio) and came across a weird interaction. When I use asio to send a 
> message via TCP, some messages do not arrive at the server.
> 
> The effect is exhibited when I send a message from the tcp client to the 
> server after calling MPI::Init();
> If I remove this line, it works as intended.
> 
> Is this interaction known? If so, is there a workaround or something I can do 
> to use this?
> 
> Thanks,
> Hannes
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Segmentation fault in mca_pml_ob1.so

[Terry Dontje]

I am not sure this has anything to do with your problem but if you look 
at the stack entry for PMPI_Recv I noticed the buf has a value of 0.  
Shouldn't that be an address?


Does your code fail if the MPI library is built with -g?  If it does 
fail the same way, the next step I would do would be to walk up the 
stack and try and figure out where the sendreq address is coming from 
because supposedly it is that address that is not mapped according to 
the original stack.


--td

On 12/07/2010 08:29 AM, Grzegorz Maj wrote:

Some update on this issue. I've attached gdb to the crashing
application and I got:

-
Program received signal SIGSEGV, Segmentation fault.
mca_pml_ob1_send_request_put (sendreq=0x130c480, btl=0xc49850,
hdr=0xd10e60) at pml_ob1_sendreq.c:1231
1231pml_ob1_sendreq.c: No such file or directory.
in pml_ob1_sendreq.c
(gdb) bt
#0  mca_pml_ob1_send_request_put (sendreq=0x130c480, btl=0xc49850,
hdr=0xd10e60) at pml_ob1_sendreq.c:1231
#1  0x7fc55bf31693 in mca_btl_tcp_endpoint_recv_handler (sd=, flags=, user=) at btl_tcp_endpoint.c:718
#2  0x7fc55fff7de4 in event_process_active (base=0xc1daf0,
flags=2) at event.c:651
#3  opal_event_base_loop (base=0xc1daf0, flags=2) at event.c:823
#4  0x7fc55ffe9ff1 in opal_progress () at runtime/opal_progress.c:189
#5  0x7fc55c9d7115 in opal_condition_wait (addr=, count=, datatype=,
src=, tag=,
 comm=, status=0xcc6100) at
../../../../opal/threads/condition.h:99
#6  ompi_request_wait_completion (addr=,
count=, datatype=,
src=, tag=,
 comm=, status=0xcc6100) at
../../../../ompi/request/request.h:375
#7  mca_pml_ob1_recv (addr=, count=, datatype=, src=, tag=, comm=,
 status=0xcc6100) at pml_ob1_irecv.c:104
#8  0x7fc560511260 in PMPI_Recv (buf=0x0, count=12884048,
type=0xd10410, source=-1, tag=0, comm=0xd0daa0, status=0xcc6100) at
precv.c:75
#9  0x0049cc43 in BI_Srecv ()
#10 0x0049c555 in BI_IdringBR ()
#11 0x00495ba1 in ilp64_Cdgebr2d ()
#12 0x0047ffa0 in Cdgebr2d ()
#13 0x7fc5621da8e1 in PB_CInV2 () from
/home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so
#14 0x7fc56220289c in PB_CpgemmAB () from
/home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so
#15 0x7fc5622b28fd in pdgemm_ () from
/home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so
-

So this looks like the line responsible for segmentation fault is:
mca_bml_base_endpoint_t *bml_endpoint = sendreq->req_endpoint;

I repeated it several times: always crashes in the same line.

I have no idea what to do with this. Again, any help would be appreciated.

Thanks,
Grzegorz Maj



2010/12/6 Grzegorz Maj:

Hi,
I'm using mkl scalapack in my project. Recently, I was trying to run
my application on new set of nodes. Unfortunately, when I try to
execute more than about 20 processes, I get segmentation fault.

[compn7:03552] *** Process received signal ***
[compn7:03552] Signal: Segmentation fault (11)
[compn7:03552] Signal code: Address not mapped (1)
[compn7:03552] Failing at address: 0x20b2e68
[compn7:03552] [ 0] /lib64/libpthread.so.0(+0xf3c0) [0x7f46e0fc33c0]
[compn7:03552] [ 1]
/home/gmaj/lib/openmpi/lib/openmpi/mca_pml_ob1.so(+0xd577)
[0x7f46dd093577]
[compn7:03552] [ 2]
/home/gmaj/lib/openmpi/lib/openmpi/mca_btl_tcp.so(+0x5b4c)
[0x7f46dc5edb4c]
[compn7:03552] [ 3]
/home/gmaj/lib/openmpi/lib/libopen-pal.so.0(+0x1dbe8) [0x7f46e0679be8]
[compn7:03552] [ 4]
(home/gmaj/lib/openmpi/lib/libopen-pal.so.0(opal_progress+0xa1)
[0x7f46e066dbf1]
[compn7:03552] [ 5]
/home/gmaj/lib/openmpi/lib/openmpi/mca_pml_ob1.so(+0x5945)
[0x7f46dd08b945]
[compn7:03552] [ 6]
/home/gmaj/lib/openmpi/lib/libmpi.so.0(MPI_Send+0x6a) [0x7f46e0b4f10a]
[compn7:03552] [ 7] /home/gmaj/matrix/matrix(BI_Ssend+0x21) [0x49cc11]
[compn7:03552] [ 8] /home/gmaj/matrix/matrix(BI_IdringBR+0x79) [0x49c579]
[compn7:03552] [ 9] /home/gmaj/matrix/matrix(ilp64_Cdgebr2d+0x221) [0x495bb1]
[compn7:03552] [10] /home/gmaj/matrix/matrix(Cdgebr2d+0xd0) [0x47ffb0]
[compn7:03552] [11]
/home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so(PB_CInV2+0x1304)
[0x7f46e27f5124]
[compn7:03552] *** End of error message ***

This error appears during some scalapack computation. My processes do
some mpi communication before this error appears.

I found out, that by modifying btl_tcp_eager_limit and
btl_tcp_max_send_size parameters, I can run more processes - the
smaller those values are, the more processes I can run. Unfortunately,
by this method I've succeeded to run up to 30 processes, which is
still far to small.

Some clue may be what valgrind says:

==3894== Syscall param writev(vector[...]) points to uninitialised byte(s)
==3894==at 0x82D009B: writev (in /lib64/libc-2.12.90.so)
==3894==by 0xBA2136D: mca_btl_tcp_frag_send (in
/home/gmaj/lib/openmpi/lib/openmpi/mca_btl_tcp.so)
==3894==by 0xBA203D0: mca_btl_tcp_endpoint_send (in
/home/gmaj/lib/openmpi/lib/openmpi/mca_btl_tcp.so)
==3894==by 0xB003583: mca_pml_

Re: [OMPI users] Segmentation fault in mca_pml_ob1.so

[Grzegorz Maj]

Some update on this issue. I've attached gdb to the crashing
application and I got:

-
Program received signal SIGSEGV, Segmentation fault.
mca_pml_ob1_send_request_put (sendreq=0x130c480, btl=0xc49850,
hdr=0xd10e60) at pml_ob1_sendreq.c:1231
1231pml_ob1_sendreq.c: No such file or directory.
in pml_ob1_sendreq.c
(gdb) bt
#0  mca_pml_ob1_send_request_put (sendreq=0x130c480, btl=0xc49850,
hdr=0xd10e60) at pml_ob1_sendreq.c:1231
#1  0x7fc55bf31693 in mca_btl_tcp_endpoint_recv_handler (sd=, flags=, user=) at btl_tcp_endpoint.c:718
#2  0x7fc55fff7de4 in event_process_active (base=0xc1daf0,
flags=2) at event.c:651
#3  opal_event_base_loop (base=0xc1daf0, flags=2) at event.c:823
#4  0x7fc55ffe9ff1 in opal_progress () at runtime/opal_progress.c:189
#5  0x7fc55c9d7115 in opal_condition_wait (addr=, count=, datatype=,
src=, tag=,
comm=, status=0xcc6100) at
../../../../opal/threads/condition.h:99
#6  ompi_request_wait_completion (addr=,
count=, datatype=,
src=, tag=,
comm=, status=0xcc6100) at
../../../../ompi/request/request.h:375
#7  mca_pml_ob1_recv (addr=, count=, datatype=, src=, tag=, comm=,
status=0xcc6100) at pml_ob1_irecv.c:104
#8  0x7fc560511260 in PMPI_Recv (buf=0x0, count=12884048,
type=0xd10410, source=-1, tag=0, comm=0xd0daa0, status=0xcc6100) at
precv.c:75
#9  0x0049cc43 in BI_Srecv ()
#10 0x0049c555 in BI_IdringBR ()
#11 0x00495ba1 in ilp64_Cdgebr2d ()
#12 0x0047ffa0 in Cdgebr2d ()
#13 0x7fc5621da8e1 in PB_CInV2 () from
/home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so
#14 0x7fc56220289c in PB_CpgemmAB () from
/home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so
#15 0x7fc5622b28fd in pdgemm_ () from
/home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so
-

So this looks like the line responsible for segmentation fault is:
mca_bml_base_endpoint_t *bml_endpoint = sendreq->req_endpoint;

I repeated it several times: always crashes in the same line.

I have no idea what to do with this. Again, any help would be appreciated.

Thanks,
Grzegorz Maj



2010/12/6 Grzegorz Maj :
> Hi,
> I'm using mkl scalapack in my project. Recently, I was trying to run
> my application on new set of nodes. Unfortunately, when I try to
> execute more than about 20 processes, I get segmentation fault.
>
> [compn7:03552] *** Process received signal ***
> [compn7:03552] Signal: Segmentation fault (11)
> [compn7:03552] Signal code: Address not mapped (1)
> [compn7:03552] Failing at address: 0x20b2e68
> [compn7:03552] [ 0] /lib64/libpthread.so.0(+0xf3c0) [0x7f46e0fc33c0]
> [compn7:03552] [ 1]
> /home/gmaj/lib/openmpi/lib/openmpi/mca_pml_ob1.so(+0xd577)
> [0x7f46dd093577]
> [compn7:03552] [ 2]
> /home/gmaj/lib/openmpi/lib/openmpi/mca_btl_tcp.so(+0x5b4c)
> [0x7f46dc5edb4c]
> [compn7:03552] [ 3]
> /home/gmaj/lib/openmpi/lib/libopen-pal.so.0(+0x1dbe8) [0x7f46e0679be8]
> [compn7:03552] [ 4]
> (home/gmaj/lib/openmpi/lib/libopen-pal.so.0(opal_progress+0xa1)
> [0x7f46e066dbf1]
> [compn7:03552] [ 5]
> /home/gmaj/lib/openmpi/lib/openmpi/mca_pml_ob1.so(+0x5945)
> [0x7f46dd08b945]
> [compn7:03552] [ 6]
> /home/gmaj/lib/openmpi/lib/libmpi.so.0(MPI_Send+0x6a) [0x7f46e0b4f10a]
> [compn7:03552] [ 7] /home/gmaj/matrix/matrix(BI_Ssend+0x21) [0x49cc11]
> [compn7:03552] [ 8] /home/gmaj/matrix/matrix(BI_IdringBR+0x79) [0x49c579]
> [compn7:03552] [ 9] /home/gmaj/matrix/matrix(ilp64_Cdgebr2d+0x221) [0x495bb1]
> [compn7:03552] [10] /home/gmaj/matrix/matrix(Cdgebr2d+0xd0) [0x47ffb0]
> [compn7:03552] [11]
> /home/gmaj/lib/intel_mkl/current/lib/em64t/libmkl_scalapack_ilp64.so(PB_CInV2+0x1304)
> [0x7f46e27f5124]
> [compn7:03552] *** End of error message ***
>
> This error appears during some scalapack computation. My processes do
> some mpi communication before this error appears.
>
> I found out, that by modifying btl_tcp_eager_limit and
> btl_tcp_max_send_size parameters, I can run more processes - the
> smaller those values are, the more processes I can run. Unfortunately,
> by this method I've succeeded to run up to 30 processes, which is
> still far to small.
>
> Some clue may be what valgrind says:
>
> ==3894== Syscall param writev(vector[...]) points to uninitialised byte(s)
> ==3894==    at 0x82D009B: writev (in /lib64/libc-2.12.90.so)
> ==3894==    by 0xBA2136D: mca_btl_tcp_frag_send (in
> /home/gmaj/lib/openmpi/lib/openmpi/mca_btl_tcp.so)
> ==3894==    by 0xBA203D0: mca_btl_tcp_endpoint_send (in
> /home/gmaj/lib/openmpi/lib/openmpi/mca_btl_tcp.so)
> ==3894==    by 0xB003583: mca_pml_ob1_send_request_start_rdma (in
> /home/gmaj/lib/openmpi/lib/openmpi/mca_pml_ob1.so)
> ==3894==    by 0xAFFA7C9: mca_pml_ob1_send (in
> /home/gmaj/lib/openmpi/lib/openmpi/mca_pml_ob1.so)
> ==3894==    by 0x6D4B109: PMPI_Send (in 
> /home/gmaj/lib/openmpi/lib/libmpi.so.0)
> ==3894==    by 0x49CC10: BI_Ssend (in /home/gmaj/matrix/matrix)
> ==3894==    by 0x49C578: BI_IdringBR (in /home/gmaj/matrix/matrix)
> ==3894==    by 0x495BB0

[OMPI users] Interaction with boost::asio

[Hannes Brandstätter-Müller]

Hello!

I am using OpenMPI in combination with the boost libraries (especially
boost::asio) and came across a weird interaction. When I use asio to send a
message via TCP, some messages do not arrive at the server.

The effect is exhibited when I send a message from the tcp client to the
server after calling MPI::Init();
If I remove this line, it works as intended.

Is this interaction known? If so, is there a workaround or something I can
do to use this?

Thanks,
Hannes

Re: [OMPI users] glut display 'occasionally' opens

[Ralph Castain]

BTW: you might check to see if the DISPLAY envar is being correctly set on all 
procs. Two ways to do it:

1. launch "env" as your app to print out the envars - can be messy on the 
output end, though you could use the mpirun options to tag and/or split the 
output from the procs

2. in your app, just do a getenv and print the display envar

Would help tell us if there is an OMPI problem, or just a problem in how you 
setup X11


On Dec 6, 2010, at 9:18 PM, Ralph Castain wrote:

> Hmmm...yes, the code does seem to handle that '=' being in there. Forgot it 
> was there.
> 
> Depending on the version you are using, mpirun could just open the display 
> for you. There is an mpirun option that tells us to please start each app in 
> its own xterm.
> 
> You shouldn't need forwarding if you are going to see it on a local display 
> (i.e., one physically attached to the node), assuming you are logged into 
> those nodes (otherwise you don't own the display).
> 
> If you are trying to view it on your own local display, then you do need 
> forwarding setup.
> 
> 
> On Dec 6, 2010, at 8:36 PM, brad baker wrote:
> 
>> Without including the -x DISPLAY, glut doesn't know what display to open.  
>> For instance, without the -x DISPLAY parameter glut returns an error from 
>> each process stating that it could not find display "" (empty string).  This 
>> strategy is briefly described in the openmpi   FAQs for launching gui 
>> applications with openmpi. 
>> 
>> I'm assuming that by setting the DISPLAY envar to :0.0, each process will 
>> render to their local display, which is my intention, and as I previously 
>> stated works for up to 2 processes. So I believe it to be necessary.  
>> 
>> But I'm thinking I may have to configure some kind of  X11 forwarding.  I'm 
>> not sure...
>> 
>> Thanks for your reply! Any more ideas?
>> Brad
>> 
>> 
>> On Mon, Dec 6, 2010 at 6:31 PM, Ralph Castain  wrote:
>> Guess I'm not entirely sure I understand how this is supposed to work. All 
>> the -x does is tell us to pickup an envar of the given name and forward its 
>> value to the remote apps. You can't set the envar's value on the cmd line. 
>> So you told mpirun to pickup the value of an envar called "DISPLAY=:0.0".
>> 
>> So yes - I would expect this would be behaving strangely.
>> 
>> If you tell us -x DISPLAY, we'll pickup the local value of DISPLAY and 
>> forward it. What that will cause your app to do is, I suppose, up to it.
>> 
>> 
>> On Dec 6, 2010, at 12:42 PM, brad baker wrote:
>> 
>> > Hello,
>> >
>> > I'm working on an mpi application that opens a glut display on each node 
>> > of a small cluster for opengl rendering (each node has its own display). 
>> > My current implementation scales great with mpich2, but I'd like to use 
>> > openmpi infiniband, which is giving me trouble.
>> >
>> > I've had some success with the -x DISPLAY=:0.0 parameter to mpirun, which 
>> > will open the display on up to 2 of my nodes... any 2.  But when I attempt 
>> > to run the application on 4 nodes, the display is non-deterministic. If 
>> > any open at all process 0 definately will, and sometimes process 3 along 
>> > with that.  I haven't observed much behavior from process 1 or 2.
>> >
>> > My command:
>> >
>> > mpirun -x DISPLAY=:0.0 -np 4 -hostfile ~/openmpi.hosts ./myapp
>> >
>> > I've tried adding the -d option with no success.
>> >
>> > Does anyone have any suggestions or comments?  I'll certainly provide more 
>> > information if required.
>> >
>> > Thanks,
>> > Brad
>> > ___
>> > users mailing list
>> > us...@open-mpi.org
>> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 

Re: [OMPI users] MPI_Send doesn't work if the data >= 2GB

[孟宪军]

Hi Gus Correa

First of all, thanks for your suggestions.

1) The malloc function do return a non_NULL pointer.

2) I didn't tried the MPI_Isend function, actually, The really function I
need to use is MPI_Allgatherv(). When I used it, I found this function
didn't work when the the data >= 2GB, then I debugged it and found this
function finally call the MPI_Send.

3) I have a large number of data need to train. so transfer the message >=
2GB is neccerary. Although I can divided the data into smaller, but I think
the effciency will become lower too.


Regards
Xianjun Meng

2010/12/7 Gus Correa 

> Hi Xianjun
>
> Suggestions/Questions:
>
> 1) Did you check if malloc returns a non-NULL pointer?
> Your program is assuming this, but it may not be true,
> and in this case the problem is not with MPI.
> You can print a message and call MPI_Abort if it doesn't.
>
> 2) Have you tried MPI_Isend/MPI_Irecv?
> Or perhaps the buffered cousin MPI_Ibsend?
>
> 3) Why do you want to send these huge messages?
> Wouldn't it be less of a trouble to send several
> smaller messages?
>
> I hope it helps,
> Gus Correa
>
> Xianjun wrote:
>
>>
>> Hi
>>
>> Are you running on two processes (mpiexec -n 2)?
>> Yes
>>
>> Have you tried to print Gsize?
>> Yes, I had checked my codes several times, and I thought the errors came
>> from the OpenMpi. :)
>>
>> The command line I used:
>> "mpirun -hostfile ./Serverlist -np 2 ./test". The "Serverlist" file
>> include several computers in my network.
>>
>> The command line that I used to build the openmpi-1.4.1:
>> ./configure --enable-debug --prefix=/usr/work/openmpi ; make all install;
>>
>> What interconnect do you use?
>> It is normal TCP/IP interconnect with 1GB network card. when I debugged my
>> codes(and the openmpi codes), I found the openMpi do call the
>> "mca_pml_ob1_send_request_start_rdma(...)" function, but I was not quite
>> sure which protocal was used when transfer 2BG data. Do you have any
>> opinions? Thanks
>>
>> Best Regards
>> Xianjun Meng
>>
>> 2010/12/7 Gus Correa mailto:g...@ldeo.columbia.edu
>> >>
>>
>>
>>Hi Xianjun
>>
>>Are you running on two processes (mpiexec -n 2)?
>>I think this code will deadlock for more than two processes.
>>The MPI_Recv won't have a matching send for rank>1.
>>
>>Also, this is C, not MPI,
>>but you may be wrapping into the negative numbers.
>>Have you tried to print Gsize?
>>It is probably -2147483648 in 32bit and 64bit machines.
>>
>>My two cents.
>>Gus Correa
>>
>>Mike Dubman wrote:
>>
>>Hi,
>>What interconnect and command line do you use? For InfiniBand
>>openib component there is a known issue with large transfers (2GB)
>>
>>https://svn.open-mpi.org/trac/ompi/ticket/2623
>>
>>try disabling memory pinning:
>>
>> http://www.open-mpi.org/faq/?category=openfabrics#large-message-leave-pinned
>>
>>
>>regards
>>M
>>
>>
>>2010/12/6 mailto:xjun.m...@gmail.com>
>>>>
>>
>>
>>
>>   hi,
>>
>>   In my computers(X86-64), the sizeof(int)=4, but the
>>   sizeof(long)=sizeof(double)=sizeof(size_t)=8. when I checked my
>>   mpi.h file, I found that the definition about the sizeof(int) is
>>   correct. meanwhile, I think the mpi.h file was generated
>>according
>>   to my compute environment when I compiled the Openmpi. So, my
>>codes
>>   still don't work. :(
>>
>>   Further, I found when I called the collective routines(such as,
>>   MPI_Allgatherv(...)) which are implemented by the Point 2 Point
>>   don't work either when the data > 2GB.
>>
>>   Thanks
>>   Xianjun
>>
>>   2010/12/6 Tim Prince mailto:n...@aol.com>
>>>>
>>
>>
>>
>>   On 12/5/2010 7:13 PM, Xianjun wrote:
>>
>>   hi,
>>
>>   I met a question recently when I tested the MPI_send and
>>   MPI_Recv
>>   functions. When I run the following codes, the processes
>>   hanged and I
>>   found there was not data transmission in my network
>>at all.
>>
>>   BTW: I finished this test on two X86-64 computers
>>with 16GB
>>   memory and
>>   installed Linux.
>>
>>   1 #include 
>>   2 #include 
>>   3 #include 
>>   4 #include 
>>   5
>>   6
>>   7 int main(int argc, char** argv)
>>   8 {
>>   9 int localID;
>>   10 int numOfPros;
>>   11 size_t Gsize = (size_t)2 * 1024 * 1024 * 1024;
>>   12
>>   13 char* g = (char*)malloc(Gsize);
>>   14
>>   15 MPI_Init(&argc, &a