Re: [OMPI users] OpenMPI how large its buffer size ?

[Jeff Squyres]

+1 on all that has been said.

As Eugene stated: this is not an internal Open MPI bug.  Your application is 
calling some form of an MPI receive with a buffer that is too small.  The MPI 
specification defines this as a truncation error; hence, Open MPI gives you an 
ERR_TRUNCATE.  You can fix the error by calling that MPI receive with a bigger 
buffer.

I believe that the issue is getting further complicated because you are not 
calling MPI directly -- you are calling boost, and so the actual MPI calls that 
are being made are being obscured.  You might need to dive into the boost.mpi 
documentation a bit more to understand exactly how you are posting a receive 
that is too small for an incoming message.  

I'm guessing that you're either accidentally posting a receive that is too 
small, or your worker nodes are sending multiple different kinds of messages to 
the master on the same tag, and the messages after the first one are larger 
than 72 bytes (i.e., it's a timing issue that some peer process' 2nd message is 
reaching the master at a "bad" time -- if this is the case, I'd suggest using 
different tags to separate the different kinds of messages that are being sent).

-

The details of how Open MPI moves messages across the network is somewhat 
irrelevant to this issue.  But if you care how that actually works, check out 
these FAQ items (they're specific to the OpenFabrics transport, but the same 
general method is used in many of Open MPI's transports):

http://www.open-mpi.org/faq/?category=openfabrics#large-message-tuning-1.3
http://www.open-mpi.org/faq/?category=openfabrics#large-message-leave-pinned



On Jul 12, 2010, at 6:31 AM, jody wrote:

> Hi
> > mpi_irecv(workerNodeID, messageTag, bufferVector[row][column])
> OpenMPI contains no function of this form.
> There is MPI_Irecv, but it takes a different number of arguments.
> 
> Or is this a boost method?
> If yes, i guess you have to make sure that the
> bufferVector[row][column] is large enough...
> Perhaps there is a boost forum you can check out if the problem persists
> 
> Jody
> 
> 
> On Sun, Jul 11, 2010 at 10:13 AM, Jack Bryan <dtustud...@hotmail.com> wrote:
> > thanks for your reply.
> > The message size is 72 bytes.
> > The master sends out the message package to each 51 nodes.
> > Then, after doing their local work, the worker node send back the same-size
> > message to the master.
> > Master use vector.push_back(new messageType) to receive each message from
> > workers.
> > Master use the
> > mpi_irecv(workerNodeID, messageTag, bufferVector[row][column])
> > to receive the worker message.
> > the row is the rankID of each worker, the column is index for  message from
> > worker.
> > Each worker may send multiple messages to master.
> > when the worker node size is large, i got MPI_ERR_TRUNCATE error.
> > Any help is appreciated.
> > JACK
> > July 10  2010
> >
> > ____________
> > Date: Sat, 10 Jul 2010 23:12:49 -0700
> > From: eugene@oracle.com
> > To: us...@open-mpi.org
> > Subject: Re: [OMPI users] OpenMPI how large its buffer size ?
> >
> > Jack Bryan wrote:
> >
> > The master node can receive message ( the same size)  from 50 worker nodes.
> > But, it cannot receive message from 51 nodes. It caused "truncate error".
> >
> > How big was the buffer that the program specified in the receive call?  How
> > big was the message that was sent?
> >
> > MPI_ERR_TRUNCATE means that you posted a receive with an application buffer
> > that turned out to be too small to hold the message that was received.  It's
> > a user application error that has nothing to do with MPI's internal
> > buffers.  MPI's internal buffers don't need to be big enough to hold that
> > message.  MPI could require the sender and receiver to coordinate so that
> > only part of the message is moved at a time.
> >
> > I used the same buffer to get the message in 50 node case.
> > About ""rendezvous" protocol", what is the meaning of "the sender sends a
> > short portion "?
> > What is the "short portion", is it a small mart of the message of the sender
> > ?
> >
> > It's at least the message header (communicator, tag, etc.) so that the
> > receiver can figure out if this is the expected message or not.  In
> > practice, there is probably also some data in there as well.  The amount of
> > that portion depends on the MPI implementation and, in practice, the
> > interconnect the message traveled over, MPI-implementation-dependent
> > environment variables set by the user, etc.  E.g., with OMPI over shared

Re: [OMPI users] OpenMPI how large its buffer size ?

[jody]

Hi
> mpi_irecv(workerNodeID, messageTag, bufferVector[row][column])
OpenMPI contains no function of this form.
There is MPI_Irecv, but it takes a different number of arguments.

Or is this a boost method?
If yes, i guess you have to make sure that the
bufferVector[row][column] is large enough...
Perhaps there is a boost forum you can check out if the problem persists

Jody


On Sun, Jul 11, 2010 at 10:13 AM, Jack Bryan <dtustud...@hotmail.com> wrote:
> thanks for your reply.
> The message size is 72 bytes.
> The master sends out the message package to each 51 nodes.
> Then, after doing their local work, the worker node send back the same-size
> message to the master.
> Master use vector.push_back(new messageType) to receive each message from
> workers.
> Master use the
> mpi_irecv(workerNodeID, messageTag, bufferVector[row][column])
> to receive the worker message.
> the row is the rankID of each worker, the column is index for  message from
> worker.
> Each worker may send multiple messages to master.
> when the worker node size is large, i got MPI_ERR_TRUNCATE error.
> Any help is appreciated.
> JACK
> July 10  2010
>
> 
> Date: Sat, 10 Jul 2010 23:12:49 -0700
> From: eugene@oracle.com
> To: us...@open-mpi.org
> Subject: Re: [OMPI users] OpenMPI how large its buffer size ?
>
> Jack Bryan wrote:
>
> The master node can receive message ( the same size)  from 50 worker nodes.
> But, it cannot receive message from 51 nodes. It caused "truncate error".
>
> How big was the buffer that the program specified in the receive call?  How
> big was the message that was sent?
>
> MPI_ERR_TRUNCATE means that you posted a receive with an application buffer
> that turned out to be too small to hold the message that was received.  It's
> a user application error that has nothing to do with MPI's internal
> buffers.  MPI's internal buffers don't need to be big enough to hold that
> message.  MPI could require the sender and receiver to coordinate so that
> only part of the message is moved at a time.
>
> I used the same buffer to get the message in 50 node case.
> About ""rendezvous" protocol", what is the meaning of "the sender sends a
> short portion "?
> What is the "short portion", is it a small mart of the message of the sender
> ?
>
> It's at least the message header (communicator, tag, etc.) so that the
> receiver can figure out if this is the expected message or not.  In
> practice, there is probably also some data in there as well.  The amount of
> that portion depends on the MPI implementation and, in practice, the
> interconnect the message traveled over, MPI-implementation-dependent
> environment variables set by the user, etc.  E.g., with OMPI over shared
> memory by default it's about 4Kbytes (if I remember correctly).
>
> This "rendezvous" protocol" can work automatically in background without
> programmer
> indicates in his program ?
>
> Right.  MPI actually allows you to force such synchronization with
> MPI_Ssend, but typically MPI implementations use it automatically for
> "plain" long sends as well even if the user didn't not use MPI_Ssend.
>
> The "acknowledgement " can be generated by the receiver only when the
> corresponding mpi_irecv is posted by the receiver ?
>
> Right.
>
> 
> The New Busy think 9 to 5 is a cute idea. Combine multiple calendars with
> Hotmail. Get busy.
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] OpenMPI how large its buffer size ?

[Jack Bryan]

thanks for your reply. 
The message size is 72 bytes. 
The master sends out the message package to each 51 nodes. 
Then, after doing their local work, the worker node send back the same-size 
message to the master. 
Master use vector.push_back(new messageType) to receive each message from 
workers. 
Master use thempi_irecv(workerNodeID, messageTag, bufferVector[row][column])
to receive the worker message. 
the row is the rankID of each worker, the column is index for  message from 
worker.Each worker may send multiple messages to master. 
when the worker node size is large, i got MPI_ERR_TRUNCATE error.
Any help is appreciated. 
JACK
July 10  2010

List-Post: users@lists.open-mpi.org
Date: Sat, 10 Jul 2010 23:12:49 -0700
From: eugene@oracle.com
To: us...@open-mpi.org
Subject: Re: [OMPI users] OpenMPI how large its buffer size ?






  
  


Jack Bryan wrote:

  
  The master node can receive message ( the same size)  from 50
worker nodes. 
  But, it cannot receive message from 51 nodes. It caused
"truncate error".

How big was the buffer that the program specified in the receive call? 
How big was the message that was sent?



MPI_ERR_TRUNCATE means that you posted a receive with an application
buffer that turned out to be too small to hold the message that was
received.  It's a user application error that has nothing to do with
MPI's internal buffers.  MPI's internal buffers don't need to be big
enough to hold that message.  MPI could require the sender and receiver
to coordinate so that only part of the message is moved at a time.


  

  
  I used the same buffer to get the message in 50 node case. 
  

  
  About ""rendezvous" protocol", what is the meaning of "the
sender sends a short portion "?
  What is the "short portion", is it a small mart of the message
of the sender ?

It's at least the message header (communicator, tag, etc.) so that the
receiver can figure out if this is the expected message or not.  In
practice, there is probably also some data in there as well.  The
amount of that portion depends on the MPI implementation and, in
practice, the interconnect the message traveled over,
MPI-implementation-dependent environment variables set by the user,
etc.  E.g., with OMPI over shared memory by default it's about 4Kbytes
(if I remember correctly).


  This "rendezvous" protocol" can work automatically in background
without programmer
  indicates in his program ?

Right.  MPI actually allows you to force such synchronization with
MPI_Ssend, but typically MPI implementations use it automatically for
"plain" long sends as well even if the user didn't not use MPI_Ssend.


  The "acknowledgement " can be generated by the receiver only
when the
  corresponding mpi_irecv is posted by the receiver ? 

Right.
  
_
The New Busy think 9 to 5 is a cute idea. Combine multiple calendars with 
Hotmail. 
http://www.windowslive.com/campaign/thenewbusy?tile=multicalendar=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_5

Re: [OMPI users] OpenMPI how large its buffer size ?

[Eugene Loh]

Jack Bryan wrote:

  
  The master node can receive message ( the same size)  from 50
worker nodes. 
  But, it cannot receive message from 51 nodes. It caused
"truncate error".

How big was the buffer that the program specified in the receive call? 
How big was the message that was sent?

MPI_ERR_TRUNCATE means that you posted a receive with an application
buffer that turned out to be too small to hold the message that was
received.  It's a user application error that has nothing to do with
MPI's internal buffers.  MPI's internal buffers don't need to be big
enough to hold that message.  MPI could require the sender and receiver
to coordinate so that only part of the message is moved at a time.

  
  
  I used the same buffer to get the message in 50 node case. 
  
  
  About ""rendezvous" protocol", what is the meaning of "the
sender sends a short portion "?
  What is the "short portion", is it a small mart of the message
of the sender ?

It's at least the message header (communicator, tag, etc.) so that the
receiver can figure out if this is the expected message or not.  In
practice, there is probably also some data in there as well.  The
amount of that portion depends on the MPI implementation and, in
practice, the interconnect the message traveled over,
MPI-implementation-dependent environment variables set by the user,
etc.  E.g., with OMPI over shared memory by default it's about 4Kbytes
(if I remember correctly).

  This "rendezvous" protocol" can work automatically in background
without programmer
  indicates in his program ?

Right.  MPI actually allows you to force such synchronization with
MPI_Ssend, but typically MPI implementations use it automatically for
"plain" long sends as well even if the user didn't not use MPI_Ssend.

  The "acknowledgement " can be generated by the receiver only
when the
  corresponding mpi_irecv is posted by the receiver ? 

Right.

Re: [OMPI users] OpenMPI how large its buffer size ?

[Jack Bryan]

Hi, 
thanks for all your replies. 
The master node can receive message ( the same size)  from 50 worker nodes. 
But, it cannot receive message from 51 nodes. It caused "truncate error". 
I used the same buffer to get the message in 50 node case. 
About ""rendezvous" protocol", what is the meaning of "the sender sends a short 
portion "?
What is the "short portion", is it a small mart of the message of the sender 
?This "rendezvous" protocol" can work automatically in background without 
programmerindicates in his program ? 
The "acknowledgement " can be generated by the receiver only when 
thecorresponding mpi_irecv is posted by the receiver ? 
Any help is appreciated. 
Jack
July 10  2010  
List-Post: users@lists.open-mpi.org
Date: Sat, 10 Jul 2010 20:41:26 -0700
From: eugene....@oracle.com
To: us...@open-mpi.org
Subject: Re: [OMPI users] OpenMPI how large its buffer size ?






  
  


I hope I understand the question properly.



The "truncate error" means that the receive buffer provided by the user
was too small to receive the designated message.  That's an error in
the user code.



You're asking about some buffering sizes within the MPI
implementation.  We can talk about that, but it probably first makes
sense to clarify what MPI is doing.  If a sender posts a large send and
the receiver has not posted a reply, the MPI implementation is not
required to move any data.  In particular, most MPI implementations
will use a "rendezvous" protocol in which the sender sends a short
portion and then waits for an acknowledgement from the receiver that it
is ready to receive the message (and knows into which user buffer to
place the received data).  This protocol is used so that the MPI
implementation does not have to buffer internally arbitrarily large
messages.



So, if you post a large send but no receive, the MPI implementation is
probably buffering very little data.  The message won't advance until
the receive has been posted.  This means that a blocking MPI_Send will
wait and a nonblocking MPI_Isend will return without having done much.



Jack Bryan wrote:

  Hi, 
  

  
  thanks for the program from Jody. 
  

  
  David indicated the question that I want to ask. 
  

  
  But, Jody's approach is ok when the MPI built-in buffer size is
large enough to hold the 
  message such as 100kB in the buffer. 
  

  
  In
asynchronous communication, when the sender posts a mpi_isend, the
message is put in 
  a
buffer provided by the MPI. 
  

  
  At
this point, the receiver may still not post its corresponding
mpi_irecv. So, the buffer size is 
  important
here. 
  

  
  Without
knowing the buffer size, I may get " truncate error " on Open MPI. 
  

  
  How to know the size of the buffer automatically
created by Open MPI in the background ?
  

  
  Any
help is appreciated. 
  

Jack,
  

  
  July 10 2010
  

  From: solarbik...@gmail.com

List-Post: users@lists.open-mpi.org
Date: Sat, 10 Jul 2010 16:46:12 -0700

To: us...@open-mpi.org

Subject: Re: [OMPI users] OpenMPI how large its buffer size ?

  

I believe his question is regarding when under non-blocking send/recv,
how does MPI know how much memory to allocate to receive the message,
since the size is determined AFTER the irecv is posted.  So if the send
post isend, but the receiver hasn't post irecv, what would the MPI do
with the message.

  

I believe MPI would automatically create a buffer in the background to
store the message.

  

  On Sat, Jul 10, 2010 at 1:55 PM, jody <jody@gmail.com>
wrote:

  Perhaps
i misunderstand your question...

Generally, it is the user's job to provide the buffers both to send and
receive.

If you call MPI_Recv, you must pass a buffer that is large enough to

hold the data sent by the

corresponding MPI_Send. I.e., if you know your sender will send

messages of 100kB,

then you must provide a buffer of size 100kB to the receiver.

If the message size is unknown at compile time, you may have to send

two messages:

first an integer which tells the receiver how large a buffer it has to

allocate, and then

the actual message (which then nicely fits into the freshly allocated
buffer)



#include 

#include 



#include 





#include "mpi.h"



#define SENDER 1

#define RECEIVER   0

#define TAG_LEN   77

#define TAG_DATA  78

#define MAX_MESSAGE 16



int main(int argc, char *argv[]) {



   int num_procs;

   int rank;

   int *send_buf;

   int *recv_buf;

   int send_message_size;

   int recv_message_size;

   MPI_Status st;

   int i;



   /* initialize random numbers */

   srand(time(NULL));

   MPI_Init(, );

   MPI_Comm_size(MPI_COMM_WORLD, _procs);

   MPI_Comm_rank(MPI_COMM_WORLD, );



   if (rank == RECEIVER) {

   /* the receiver */

   /* wait for message length */

   MPI_Recv(_message_size, 1, MPI_INT, SENDER, TAG_LEN,

Re: [OMPI users] OpenMPI how large its buffer size ?

[Eugene Loh]

I hope I understand the question properly.

The "truncate error" means that the receive buffer provided by the user
was too small to receive the designated message.  That's an error in
the user code.

You're asking about some buffering sizes within the MPI
implementation.  We can talk about that, but it probably first makes
sense to clarify what MPI is doing.  If a sender posts a large send and
the receiver has not posted a reply, the MPI implementation is not
required to move any data.  In particular, most MPI implementations
will use a "rendezvous" protocol in which the sender sends a short
portion and then waits for an acknowledgement from the receiver that it
is ready to receive the message (and knows into which user buffer to
place the received data).  This protocol is used so that the MPI
implementation does not have to buffer internally arbitrarily large
messages.

So, if you post a large send but no receive, the MPI implementation is
probably buffering very little data.  The message won't advance until
the receive has been posted.  This means that a blocking MPI_Send will
wait and a nonblocking MPI_Isend will return without having done much.

Jack Bryan wrote:

  Hi, 
  
  
  thanks for the program from Jody. 
  
  
  David indicated the question that I want to ask. 
  
  
  But, Jody's approach is ok when the MPI built-in buffer size is
large enough to hold the 
  message such as 100kB in the buffer. 
  
  
  In
asynchronous communication, when the sender posts a mpi_isend, the
message is put in 
  a
buffer provided by the MPI. 
  
  
  At
this point, the receiver may still not post its corresponding
mpi_irecv. So, the buffer size is 
  important
here. 
  
  
  Without
knowing the buffer size, I may get " truncate error " on Open MPI. 
  
  
  How to know the size of the buffer automatically
created by Open MPI in the background ?
  
  
  Any
help is appreciated. 
  
Jack,
  
  
  July 10 2010
  
  From: solarbik...@gmail.com
List-Post: users@lists.open-mpi.org
Date: Sat, 10 Jul 2010 16:46:12 -0700
To: us...@open-mpi.org
Subject: Re: [OMPI users] OpenMPI how large its buffer size ?
  
I believe his question is regarding when under non-blocking send/recv,
how does MPI know how much memory to allocate to receive the message,
since the size is determined AFTER the irecv is posted.  So if the send
post isend, but the receiver hasn't post irecv, what would the MPI do
with the message.
  
I believe MPI would automatically create a buffer in the background to
store the message.
  
  On Sat, Jul 10, 2010 at 1:55 PM, jody <jody@gmail.com>
wrote:
  Perhaps
i misunderstand your question...
Generally, it is the user's job to provide the buffers both to send and
receive.
If you call MPI_Recv, you must pass a buffer that is large enough to
hold the data sent by the
corresponding MPI_Send. I.e., if you know your sender will send
messages of 100kB,
then you must provide a buffer of size 100kB to the receiver.
If the message size is unknown at compile time, you may have to send
two messages:
first an integer which tells the receiver how large a buffer it has to
allocate, and then
the actual message (which then nicely fits into the freshly allocated
buffer)

#include 
#include 

#include 


#include "mpi.h"

#define SENDER     1
#define RECEIVER   0
#define TAG_LEN   77
#define TAG_DATA  78
#define MAX_MESSAGE 16

int main(int argc, char *argv[]) {

   int num_procs;
   int rank;
   int *send_buf;
   int *recv_buf;
   int send_message_size;
   int recv_message_size;
   MPI_Status st;
   int i;

   /* initialize random numbers */
   srand(time(NULL));
   MPI_Init(, );
   MPI_Comm_size(MPI_COMM_WORLD, _procs);
   MPI_Comm_rank(MPI_COMM_WORLD, );

   if (rank == RECEIVER) {
       /* the receiver */
       /* wait for message length */
       MPI_Recv(_message_size, 1, MPI_INT, SENDER, TAG_LEN,
MPI_COMM_WORLD, );
       /* create a buffer of the required size */
       recv_buf = (int*) malloc(recv_message_size*sizeof(int));
       /* get data */
       MPI_Recv(recv_buf, recv_message_size, MPI_INT, SENDER,
TAG_DATA, MPI_COMM_WORLD, );

       printf("Receiver got %d integers:", recv_message_size);
       for (i = 0; i < recv_message_size; i++) {
           printf(" %d", recv_buf[i]);
       }
       printf("\n");

       /* clean up */
       free(recv_buf);

   } else if (rank == SENDER) {
       /* the sender */
       /* random message size */
       send_message_size =
(int)((1.0*MAX_MESSAGE*rand())/(1.0*RAND_MAX));
       /* create a buffer of the required size */
       send_buf = (int*) malloc(send_message_size*sizeof(int));
       /* create random message */
       for (i = 0; i < send_message_size; i++) {
           send_buf[i] = rand();
       }

       printf("Sender has %d integers:", send_message_size);
       for (i = 0; i < send_message_size; 

Re: [OMPI users] OpenMPI how large its buffer size ?

[David Zhang]

I believe what you are eluding to is " what is the maximum temperary
holding buffer size created automatically by MPI". For that I don't
know the answer to, althogh I have sent data array of mpi_real8 of
size over 15000 with isend and irecv without problems.

On 7/10/10, Jack Bryan <dtustud...@hotmail.com> wrote:
>
> Hi,
> thanks for the program from Jody.
> David indicated the question that I want to ask.
> But, Jody's approach is ok when the MPI built-in buffer size is large enough
> to hold the message such as 100kB in the buffer.
> In asynchronous communication, when the sender posts a mpi_isend, the
> message is put in a buffer provided by the MPI.
> At this point, the receiver may still not post its corresponding mpi_irecv.
> So, the buffer size is important here.
> Without knowing the buffer size, I may get " truncate error " on Open MPI.
> How to know the size of the buffer automatically created by Open MPI in the
> background ?
> Any help is appreciated.
> Jack,
> July 10 2010
> From: solarbik...@gmail.com
> Date: Sat, 10 Jul 2010 16:46:12 -0700
> To: us...@open-mpi.org
> Subject: Re: [OMPI users] OpenMPI how large its buffer size ?
>
> I believe his question is regarding when under non-blocking send/recv, how
> does MPI know how much memory to allocate to receive the message, since the
> size is determined AFTER the irecv is posted.  So if the send post isend,
> but the receiver hasn't post irecv, what would the MPI do with the message.
>
>
>
> I believe MPI would automatically create a buffer in the background to store
> the message.
>
> On Sat, Jul 10, 2010 at 1:55 PM, jody <jody@gmail.com> wrote:
>
>
> Perhaps i misunderstand your question...
>
> Generally, it is the user's job to provide the buffers both to send and
> receive.
>
> If you call MPI_Recv, you must pass a buffer that is large enough to
>
> hold the data sent by the
>
> corresponding MPI_Send. I.e., if you know your sender will send
>
> messages of 100kB,
>
> then you must provide a buffer of size 100kB to the receiver.
>
> If the message size is unknown at compile time, you may have to send
>
> two messages:
>
> first an integer which tells the receiver how large a buffer it has to
>
> allocate, and then
>
> the actual message (which then nicely fits into the freshly allocated
> buffer)
>
>
>
> #include 
>
> #include 
>
>
>
> #include 
>
>
>
>
>
> #include "mpi.h"
>
>
>
> #define SENDER 1
>
> #define RECEIVER   0
>
> #define TAG_LEN   77
>
> #define TAG_DATA  78
>
> #define MAX_MESSAGE 16
>
>
>
> int main(int argc, char *argv[]) {
>
>
>
> int num_procs;
>
> int rank;
>
> int *send_buf;
>
> int *recv_buf;
>
> int send_message_size;
>
> int recv_message_size;
>
> MPI_Status st;
>
> int i;
>
>
>
> /* initialize random numbers */
>
> srand(time(NULL));
>
> MPI_Init(, );
>
> MPI_Comm_size(MPI_COMM_WORLD, _procs);
>
> MPI_Comm_rank(MPI_COMM_WORLD, );
>
>
>
> if (rank == RECEIVER) {
>
> /* the receiver */
>
> /* wait for message length */
>
> MPI_Recv(_message_size, 1, MPI_INT, SENDER, TAG_LEN,
>
> MPI_COMM_WORLD, );
>
> /* create a buffer of the required size */
>
> recv_buf = (int*) malloc(recv_message_size*sizeof(int));
>
> /* get data */
>
> MPI_Recv(recv_buf, recv_message_size, MPI_INT, SENDER,
>
> TAG_DATA, MPI_COMM_WORLD, );
>
>
>
> printf("Receiver got %d integers:", recv_message_size);
>
> for (i = 0; i < recv_message_size; i++) {
>
> printf(" %d", recv_buf[i]);
>
> }
>
> printf("\n");
>
>
>
> /* clean up */
>
> free(recv_buf);
>
>
>
> } else if (rank == SENDER) {
>
> /* the sender */
>
> /* random message size */
>
> send_message_size = (int)((1.0*MAX_MESSAGE*rand())/(1.0*RAND_MAX));
>
> /* create a buffer of the required size */
>
> send_buf = (int*) malloc(send_message_size*sizeof(int));
>
> /* create random message */
>
> for (i = 0; i < send_message_size; i++) {
>
> send_buf[i] = rand();
>
> }
>
>
>
> printf("Sender has %d integers:", send_message_size);
>
> for (i = 0; i < send_message_size; i++) {
>
> printf(" %d", send_buf[i]);
>
> }
>
> pri

Re: [OMPI users] OpenMPI how large its buffer size ?

[Jack Bryan]

Hi, 
thanks for the program from Jody. 
David indicated the question that I want to ask. 
But, Jody's approach is ok when the MPI built-in buffer size is large enough to 
hold the message such as 100kB in the buffer. 
In asynchronous communication, when the sender posts a mpi_isend, the message 
is put in a buffer provided by the MPI. 
At this point, the receiver may still not post its corresponding mpi_irecv. So, 
the buffer size is important here. 
Without knowing the buffer size, I may get " truncate error " on Open MPI. 
How to know the size of the buffer automatically created by Open MPI in the 
background ?
Any help is appreciated. 
Jack,
July 10 2010
From: solarbik...@gmail.com
List-Post: users@lists.open-mpi.org
Date: Sat, 10 Jul 2010 16:46:12 -0700
To: us...@open-mpi.org
Subject: Re: [OMPI users] OpenMPI how large its buffer size ?

I believe his question is regarding when under non-blocking send/recv, how does 
MPI know how much memory to allocate to receive the message, since the size is 
determined AFTER the irecv is posted.  So if the send post isend, but the 
receiver hasn't post irecv, what would the MPI do with the message.



I believe MPI would automatically create a buffer in the background to store 
the message.

On Sat, Jul 10, 2010 at 1:55 PM, jody <jody@gmail.com> wrote:


Perhaps i misunderstand your question...

Generally, it is the user's job to provide the buffers both to send and receive.

If you call MPI_Recv, you must pass a buffer that is large enough to

hold the data sent by the

corresponding MPI_Send. I.e., if you know your sender will send

messages of 100kB,

then you must provide a buffer of size 100kB to the receiver.

If the message size is unknown at compile time, you may have to send

two messages:

first an integer which tells the receiver how large a buffer it has to

allocate, and then

the actual message (which then nicely fits into the freshly allocated buffer)



#include 

#include 



#include 





#include "mpi.h"



#define SENDER 1

#define RECEIVER   0

#define TAG_LEN   77

#define TAG_DATA  78

#define MAX_MESSAGE 16



int main(int argc, char *argv[]) {



int num_procs;

int rank;

int *send_buf;

int *recv_buf;

int send_message_size;

int recv_message_size;

MPI_Status st;

int i;



/* initialize random numbers */

srand(time(NULL));

MPI_Init(, );

MPI_Comm_size(MPI_COMM_WORLD, _procs);

MPI_Comm_rank(MPI_COMM_WORLD, );



if (rank == RECEIVER) {

/* the receiver */

/* wait for message length */

MPI_Recv(_message_size, 1, MPI_INT, SENDER, TAG_LEN,

MPI_COMM_WORLD, );

/* create a buffer of the required size */

recv_buf = (int*) malloc(recv_message_size*sizeof(int));

/* get data */

MPI_Recv(recv_buf, recv_message_size, MPI_INT, SENDER,

TAG_DATA, MPI_COMM_WORLD, );



printf("Receiver got %d integers:", recv_message_size);

for (i = 0; i < recv_message_size; i++) {

printf(" %d", recv_buf[i]);

}

printf("\n");



/* clean up */

free(recv_buf);



} else if (rank == SENDER) {

/* the sender */

/* random message size */

send_message_size = (int)((1.0*MAX_MESSAGE*rand())/(1.0*RAND_MAX));

/* create a buffer of the required size */

send_buf = (int*) malloc(send_message_size*sizeof(int));

/* create random message */

for (i = 0; i < send_message_size; i++) {

send_buf[i] = rand();

}



printf("Sender has %d integers:", send_message_size);

for (i = 0; i < send_message_size; i++) {

printf(" %d", send_buf[i]);

}

printf("\n");



/* send message size to receiver */

MPI_Send(_message_size,  1, MPI_INT, RECEIVER, TAG_LEN,

MPI_COMM_WORLD);

/* now send messagge */

MPI_Send(send_buf, send_message_size, MPI_INT, RECEIVER,

TAG_DATA, MPI_COMM_WORLD);



/* clean up */

free(send_buf);



}



MPI_Finalize();

}



I hope this helps

  Jody





On Sat, Jul 10, 2010 at 7:12 AM, Jack Bryan <dtustud...@hotmail.com> wrote:

> Dear All:

> How to find the buffer size of OpenMPI ?

> I need to transfer large data between nodes on a cluster with OpenMPI 1.3.4.

> Many nodes need to send data to the same node .

> Workers use mpi_isend, the receiver node use  mpi_irecv.

> because they are non-blocking, the messages are stored in buffers of

> senders.

> And then, the receiver collect messages from its buffer.

> If the receiver's buffer is too small, there will be truncate error.

> Any help is appreciated.

> Jack

> July 9  2010

>

> 

> Hotmail is redefining busy with tools for the New Busy. Get more from your

&

Re: [OMPI users] OpenMPI how large its buffer size ?

[David Zhang]

I believe his question is regarding when under non-blocking send/recv, how
does MPI know how much memory to allocate to receive the message, since the
size is determined AFTER the irecv is posted.  So if the send post isend,
but the receiver hasn't post irecv, what would the MPI do with the message.

I believe MPI would automatically create a buffer in the background to store
the message.

On Sat, Jul 10, 2010 at 1:55 PM, jody  wrote:

> Perhaps i misunderstand your question...
> Generally, it is the user's job to provide the buffers both to send and
> receive.
> If you call MPI_Recv, you must pass a buffer that is large enough to
> hold the data sent by the
> corresponding MPI_Send. I.e., if you know your sender will send
> messages of 100kB,
> then you must provide a buffer of size 100kB to the receiver.
> If the message size is unknown at compile time, you may have to send
> two messages:
> first an integer which tells the receiver how large a buffer it has to
> allocate, and then
> the actual message (which then nicely fits into the freshly allocated
> buffer)
>
> #include 
> #include 
>
> #include 
>
>
> #include "mpi.h"
>
> #define SENDER 1
> #define RECEIVER   0
> #define TAG_LEN   77
> #define TAG_DATA  78
> #define MAX_MESSAGE 16
>
> int main(int argc, char *argv[]) {
>
>int num_procs;
>int rank;
>int *send_buf;
>int *recv_buf;
>int send_message_size;
>int recv_message_size;
>MPI_Status st;
>int i;
>
>/* initialize random numbers */
>srand(time(NULL));
>MPI_Init(, );
>MPI_Comm_size(MPI_COMM_WORLD, _procs);
>MPI_Comm_rank(MPI_COMM_WORLD, );
>
>if (rank == RECEIVER) {
>/* the receiver */
>/* wait for message length */
>MPI_Recv(_message_size, 1, MPI_INT, SENDER, TAG_LEN,
> MPI_COMM_WORLD, );
>/* create a buffer of the required size */
>recv_buf = (int*) malloc(recv_message_size*sizeof(int));
>/* get data */
>MPI_Recv(recv_buf, recv_message_size, MPI_INT, SENDER,
> TAG_DATA, MPI_COMM_WORLD, );
>
>printf("Receiver got %d integers:", recv_message_size);
>for (i = 0; i < recv_message_size; i++) {
>printf(" %d", recv_buf[i]);
>}
>printf("\n");
>
>/* clean up */
>free(recv_buf);
>
>} else if (rank == SENDER) {
>/* the sender */
>/* random message size */
>send_message_size = (int)((1.0*MAX_MESSAGE*rand())/(1.0*RAND_MAX));
>/* create a buffer of the required size */
>send_buf = (int*) malloc(send_message_size*sizeof(int));
>/* create random message */
>for (i = 0; i < send_message_size; i++) {
>send_buf[i] = rand();
>}
>
>printf("Sender has %d integers:", send_message_size);
>for (i = 0; i < send_message_size; i++) {
>printf(" %d", send_buf[i]);
>}
>printf("\n");
>
>/* send message size to receiver */
>MPI_Send(_message_size,  1, MPI_INT, RECEIVER, TAG_LEN,
> MPI_COMM_WORLD);
>/* now send messagge */
>MPI_Send(send_buf, send_message_size, MPI_INT, RECEIVER,
> TAG_DATA, MPI_COMM_WORLD);
>
>/* clean up */
>free(send_buf);
>
>}
>
>MPI_Finalize();
> }
>
> I hope this helps
>  Jody
>
>
> On Sat, Jul 10, 2010 at 7:12 AM, Jack Bryan 
> wrote:
> > Dear All:
> > How to find the buffer size of OpenMPI ?
> > I need to transfer large data between nodes on a cluster with OpenMPI
> 1.3.4.
> > Many nodes need to send data to the same node .
> > Workers use mpi_isend, the receiver node use  mpi_irecv.
> > because they are non-blocking, the messages are stored in buffers of
> > senders.
> > And then, the receiver collect messages from its buffer.
> > If the receiver's buffer is too small, there will be truncate error.
> > Any help is appreciated.
> > Jack
> > July 9  2010
> >
> > 
> > Hotmail is redefining busy with tools for the New Busy. Get more from
> your
> > inbox. See how.
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>



-- 
David Zhang
University of California, San Diego

Re: [OMPI users] OpenMPI how large its buffer size ?

[jody]

Perhaps i misunderstand your question...
Generally, it is the user's job to provide the buffers both to send and receive.
If you call MPI_Recv, you must pass a buffer that is large enough to
hold the data sent by the
corresponding MPI_Send. I.e., if you know your sender will send
messages of 100kB,
then you must provide a buffer of size 100kB to the receiver.
If the message size is unknown at compile time, you may have to send
two messages:
first an integer which tells the receiver how large a buffer it has to
allocate, and then
the actual message (which then nicely fits into the freshly allocated buffer)

#include 
#include 

#include 


#include "mpi.h"

#define SENDER 1
#define RECEIVER   0
#define TAG_LEN   77
#define TAG_DATA  78
#define MAX_MESSAGE 16

int main(int argc, char *argv[]) {

int num_procs;
int rank;
int *send_buf;
int *recv_buf;
int send_message_size;
int recv_message_size;
MPI_Status st;
int i;

/* initialize random numbers */
srand(time(NULL));
MPI_Init(, );
MPI_Comm_size(MPI_COMM_WORLD, _procs);
MPI_Comm_rank(MPI_COMM_WORLD, );

if (rank == RECEIVER) {
/* the receiver */
/* wait for message length */
MPI_Recv(_message_size, 1, MPI_INT, SENDER, TAG_LEN,
MPI_COMM_WORLD, );
/* create a buffer of the required size */
recv_buf = (int*) malloc(recv_message_size*sizeof(int));
/* get data */
MPI_Recv(recv_buf, recv_message_size, MPI_INT, SENDER,
TAG_DATA, MPI_COMM_WORLD, );

printf("Receiver got %d integers:", recv_message_size);
for (i = 0; i < recv_message_size; i++) {
printf(" %d", recv_buf[i]);
}
printf("\n");

/* clean up */
free(recv_buf);

} else if (rank == SENDER) {
/* the sender */
/* random message size */
send_message_size = (int)((1.0*MAX_MESSAGE*rand())/(1.0*RAND_MAX));
/* create a buffer of the required size */
send_buf = (int*) malloc(send_message_size*sizeof(int));
/* create random message */
for (i = 0; i < send_message_size; i++) {
send_buf[i] = rand();
}

printf("Sender has %d integers:", send_message_size);
for (i = 0; i < send_message_size; i++) {
printf(" %d", send_buf[i]);
}
printf("\n");

/* send message size to receiver */
MPI_Send(_message_size,  1, MPI_INT, RECEIVER, TAG_LEN,
MPI_COMM_WORLD);
/* now send messagge */
MPI_Send(send_buf, send_message_size, MPI_INT, RECEIVER,
TAG_DATA, MPI_COMM_WORLD);

/* clean up */
free(send_buf);

}

MPI_Finalize();
}

I hope this helps
  Jody


On Sat, Jul 10, 2010 at 7:12 AM, Jack Bryan  wrote:
> Dear All:
> How to find the buffer size of OpenMPI ?
> I need to transfer large data between nodes on a cluster with OpenMPI 1.3.4.
> Many nodes need to send data to the same node .
> Workers use mpi_isend, the receiver node use  mpi_irecv.
> because they are non-blocking, the messages are stored in buffers of
> senders.
> And then, the receiver collect messages from its buffer.
> If the receiver's buffer is too small, there will be truncate error.
> Any help is appreciated.
> Jack
> July 9  2010
>
> 
> Hotmail is redefining busy with tools for the New Busy. Get more from your
> inbox. See how.
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>