Hello,
thanks for the quick answer. I am sorry that I forgot to mention this: I
did compile OpenMPI with MPI_THREAD_MULTIPLE support and test if
required == provided after the MPI_Thread_init call.
I do not see any mechanism for protecting the accesses to the requests to a
single thread? What is the thread model you're using?
Again I am sorry that this was not clear: In the pseudo code below I
wanted to indicate the access-protection I do by thread-id dependent
calls if(0 == thread-id) and by using the trylock(...) (using
pthread-mutexes). In the code all accesses concerning one MPI_Request
(which are pthread-global-pointers in my case) are protected and called
in sequential order, i.e. MPI_Isend/recv is returns before any thread is
allowed to call the corresponding MPI_Test and no-one can call MPI_Test
any more when a thread is allowed to call MPI_Wait.
I did this in the same manner before with other MPI implementations, but
also on the same machine with the same (untouched) OpenMPI
implementation, also using pthreads and MPI in combination, but I used
MPI_Request req;
instead of
MPI_Request* req;
(and later)
req = (MPI_Request*)malloc(sizeof(MPI_Request));
In my recent (problem) code, I also tried not using pointers, but got
the same problem. Also, as I described in the first mail, I tried
everything concerning the memory allocation of the MPI_Request objects.
I tried not calling malloc. This I guessed wouldn't work, but the
OpenMPI documentation says this:
" Nonblocking calls allocate a communication request object and
associate it with the request handle the argument request). "
[http://www.open-mpi.org/doc/v1.4/man3/MPI_Isend.3.php] and
" [...] if the communication object was created by a nonblocking send or
receive, then it is deallocated and the request handle is set to
MPI_REQUEST_NULL."
[http://www.open-mpi.org/doc/v1.4/man3/MPI_Test.3.php] and (in slightly
different words) [http://www.open-mpi.org/doc/v1.4/man3/MPI_Wait.3.php]
So I thought that it might do some kind of optimized memory stuff
internally.
I also tried allocating req (for each used MPI_Request) once before the
first use and deallocation after the last use (which I thought was the
way it was supposed to work), but that crashes also.
I tried replacing the pointers through global variables
MPI_Request req;
which didn't do the job...
The only thing that seems to work is what I mentioned below: Allocate
every time I am going to need it in the MPI_Isend/recv, use it in
MPI_Test/Wait and after that deallocate it by hand each time.
I don't think that this is supposed to be like this since I have to do a
call to malloc and free so often (for multiple MPI_Request objects in
each iteration) that it will most likely limit performance...
Anyway I still have the same problem and am still unclear on what kind
of memory allocation I should be doing for the MPI_Requests. Is there
anything else (besides MPI_THREAD_MULTIPLE support, thread access
control, sequential order of MPI_Isend/recv, MPI_Test and MPI_Wait for
one MPI_Request object) I need to take care of? If not, what could I do
to find the source of my problem?
Thanks again for any kind of help!
Kind regards,
David
> From an implementation perspective, your code is correct only if you
initialize the MPI library with MPI_THREAD_MULTIPLE and if the library accepts.
Otherwise, there is an assumption that the application is single threaded, or that
the MPI behavior is implementation dependent. Please read the MPI standard
regarding to MPI_Init_thread for more details.
Regards,
george.
On May 19, 2011, at 02:34 , David Büttner wrote:
Hello,
I am working on a hybrid MPI (OpenMPI 1.4.3) and Pthread code. I am using
MPI_Isend and MPI_Irecv for communication and MPI_Test/MPI_Wait to check if it
is done. I do this repeatedly in the outer loop of my code. The MPI_Test is
used in the inner loop to check if some function can be called which depends on
the received data.
The program regularly crashed (only when not using printf...) and after
debugging it I figured out the following problem:
In MPI_Isend I have an invalid read of memory. I fixed the problem with not
re-using a
MPI_Request req_s, req_r;
but by using
MPI_Request* req_s;
MPI_Request* req_r
and re-allocating them before the MPI_Isend/recv.
The documentation says, that in MPI_Wait and MPI_Test (if successful) the
request-objects are deallocated and set to MPI_REQUEST_NULL.
It also says, that in MPI_Isend and MPI_Irecv, it allocates the Objects and
associates it with the request object.
As I understand this, this either means I can use a pointer to MPI_Request
which I don't have to initialize for this (it doesn't work but crashes), or
that I can use a MPI_Request pointer which I have initialized with
malloc(sizeof(MPI_REQUEST)) (or passing the address of a MPI_Request req),
which is set and unset in the functions. But this version crashes, too.
What works is using a pointer, which I allocate before the MPI_Isend/recv and
which I free after MPI_Wait in every iteration. In other words: It only uses if
I don't reuse any kind of MPI_Request. Only if I recreate one every time.
Is this, what is should be like? I believe that a reuse of the memory would be
a lot more efficient (less calls to malloc...). Am I missing something here? Or
am I doing something wrong?
Let me provide some more detailed information about my problem:
I am running the program on a 30 node infiniband cluster. Each node has 4 single
core Opteron CPUs. I am running 1 MPI Rank per node and 4 threads per rank (->
one thread per core).
I am compiling with mpicc of OpenMPI using gcc below.
Some pseudo-code of the program can be found at the end of this e-mail.
I was able to reproduce the problem using different amount of nodes and even
using one node only. The problem does not arise when I put printf-debugging
information into the code. This pointed me into the direction that I have some
memory problem, where some write accesses some memory it is not supposed to.
I ran the tests using valgrind with --leak-check=full and --show-reachable=yes,
which pointed me either to MPI_Isend or MPI_Wait depending on whether I had the
threads spin in a loop for MPI_Test to return success or used MPI_Wait
respectively.
I would appreciate your help with this. Am I missing something important here?
Is there a way to re-use the request in the different iterations other than I
thought it should work?
Or is there a way to re-initialize the allocated memory before the
MPI_Isend/recv so that I at least don't have to call free and malloc each time?
Thank you very much for your help!
Kind regards,
David Büttner
_____________________
Pseudo-Code of program:
MPI_Request* req_s;
MPI_Request* req_w;
OUTER-LOOP
if(0 == threadid)
{
req_s = malloc(sizeof(MPI_Request));
req_r = malloc(sizeof(MPI_Request));
MPI_Isend(..., req_s)
MPI_Irecv(..., req_r)
}
pthread_barrier
INNER-LOOP (while NOT_DONE or RET)
if(TRYLOCK&& NOT_DONE)
{
if(MPI_TEST(req_r))
{
Call_Function_A;
NOT_DONE = 0;
}
}
RET = Call_Function_B;
}
pthread_barrier_wait
if(0 == threadid)
{
MPI_WAIT(req_s)
MPI_WAIT(req_r)
free(req_s);
free(req_r);
}
_____________
--
David Büttner, Informatik, Technische Universität München
TUM I-10 - FMI 01.06.059 - Tel. 089 / 289-17676
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
"To preserve the freedom of the human mind then and freedom of the press, every
spirit should be ready to devote itself to martyrdom; for as long as we may think as we
will, and speak as we think, the condition of man will proceed in improvement."
-- Thomas Jefferson, 1799
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
David Büttner, Informatik, Technische Universität München
TUM I-10 - FMI 01.06.059 - Tel. 089 / 289-17676
