As far as I have been able to understand while looking at the code, it
very much seems that Joshua pointed out the exact cause for the issue.
munmap'ing a virtual address space region does not evict it from
mpool_grdma->pool->lru_list . If a later mmap happens to return the
same address (a priori tied to different physical location), the
userspace believes this segment is already registered, and eventually
the transfer is directed to a bogus location.
This also seems to match this old discussion:
http://lists.openfabrics.org/pipermail/general/2009-April/058786.html
although I didn't read the whole discussion there, it very much seems
that the proposal for moving the pinning/caching logic to the kernel
did not make it, unfortunately.
So are we here in the situation where this "munmap should be
intercepted" logic actually proves too fragile ? (in that it's not
intercepted in my case). The memory MCA in my configuration is:
MCA memory: linux (MCA v2.0, API v2.0, Component v1.8.3)
I traced a bit what happens at the mmap call, it seems to go straight
to the libc, not via openmpi first.
For the time being, I think I'll have to consider any mmap()/munmap()
rather unsafe to play with in an openmpi application.
E.
P.S: a last version of the test case is attached.
Le 11 nov. 2014 19:48, "Emmanuel Thomé" <emmanuel.th...@gmail.com> a écrit :
>
> Thanks a lot for your analysis. This seems consistent with what I can
> obtain by playing around with my different test cases.
>
> It seems that munmap() does *not* unregister the memory chunk from the
> cache. I suppose this is the reason for the bug.
>
> In fact using mmap(..., MAP_ANONYMOUS | MAP_PRIVATE) and munmap() as
> substitutes for malloc()/free() trigger the same problem.
>
> It looks to me that there is an oversight in the OPAL hooks around the
> memory functions, then. Do you agree ?
>
> E.
>
> On Tue, Nov 11, 2014 at 3:17 PM, Joshua Ladd <jladd.m...@gmail.com> wrote:
> > I was able to reproduce your issue and I think I understand the problem a
> > bit better at least. This demonstrates exactly what I was pointing to:
> >
> > It looks like when the test switches over from eager RDMA (I'll explain in a
> > second), to doing a rendezvous protocol working entirely in user buffer
> > space things go bad.
> >
> > If you're input is smaller than some threshold, the eager RDMA limit, then
> > the contents of your user buffer are copied into OMPI/OpenIB BTL scratch
> > buffers called "eager fragments". This pool of resources is preregistered,
> > pinned, and have had their rkeys exchanged. So, in the eager protocol, your
> > data is copied into these "locked and loaded" RDMA frags and the put/get is
> > handled internally. When the data is received, it's copied back out into
> > your buffer. In your setup, this always works.
> >
> > $mpirun -np 2 --map-by node --bind-to core -mca pml ob1 -mca
> > btl_openib_if_include mlx4_0:1 -mca btl_openib_use_eager_rdma 1 -mca
> > btl_openib_eager_limit 512 -mca btl openib,self ./ibtest -s 56
> > per-node buffer has size 448 bytes
> > node 0 iteration 0, lead word received from peer is 0x00000401 [ok]
> > node 0 iteration 1, lead word received from peer is 0x00000801 [ok]
> > node 0 iteration 2, lead word received from peer is 0x00000c01 [ok]
> > node 0 iteration 3, lead word received from peer is 0x00001001 [ok]
> >
> > When you exceed the eager threshold, this always fails on the second
> > iteration. To understand this, you need to understand that there is a
> > protocol switch where now your user buffer is used for the transfer. Hence,
> > the user buffer is registered with the HCA. This operation is an inherently
> > high latency operation and is one of the primary motives for doing
> > copy-in/copy-out into preregistered buffers for small, latency sensitive
> > ops. For bandwidth bound transfers, the cost to register can be amortized
> > over the whole transfer, but it still affects the total bandwidth. In the
> > case of a rendezvous protocol where the user buffer is registered, there is
> > an optimization mostly used to help improve the numbers in a bandwidth
> > benchmark called a registration cache. With registration caching the user
> > buffer is registered once and the mkey put into a cache and the memory is
> > kept pinned until the system provides some notification via either memory
> > hooks in p2p malloc, or ummunotify that the buffer has been freed and this
> > signals that the mkey can be evicted from the cache. On subsequent
> > send/recv operations from the same user buffer address, OpenIB BTL will find
> > the address in the registration cache and take the cached mkey and avoid
> > paying the cost of the memory registration the memory registration and start
> > the data transfer.
> >
> > What I noticed is when the rendezvous protocol kicks in, it always fails on
> > the second iteration.
> >
> > $mpirun -np 2 --map-by node --bind-to core -mca pml ob1 -mca
> > btl_openib_if_include mlx4_0:1 -mca btl_openib_use_eager_rdma 1 -mca
> > btl_openib_eager_limit 128 -mca btl openib,self ./ibtest -s 56
> > per-node buffer has size 448 bytes
> > node 0 iteration 0, lead word received from peer is 0x00000401 [ok]
> > node 0 iteration 1, lead word received from peer is 0x00000000 [NOK]
> > --------------------------------------------------------------------------
> >
> > So, I suspected it has something to do with the way the virtual address is
> > being handled in this case. To test that theory, I just completely disabled
> > the registration cache by setting -mca mpi_leave_pinned 0 and things start
> > to work:
> >
> > $mpirun -np 2 --map-by node --bind-to core -mca pml ob1 -mca
> > btl_openib_if_include mlx4_0:1 -mca btl_openib_use_eager_rdma 1 -mca
> > btl_openib_eager_limit 128 -mca mpi_leave_pinned 0 -mca btl openib,self
> > ./ibtest -s 56
> > per-node buffer has size 448 bytes
> > node 0 iteration 0, lead word received from peer is 0x00000401 [ok]
> > node 0 iteration 1, lead word received from peer is 0x00000801 [ok]
> > node 0 iteration 2, lead word received from peer is 0x00000c01 [ok]
> > node 0 iteration 3, lead word received from peer is 0x00001001 [ok]
> >
> > I don't know enough about memory hooks or the registration cache
> > implementation to speak with any authority, but it looks like this is where
> > the issue resides. As a workaround, can you try your original experiment
> > with -mca mpi_leave_pinned 0 and see if you get consistent results.
> >
> >
> > Josh
> >
> >
> >
> >
> >
> > On Tue, Nov 11, 2014 at 7:07 AM, Emmanuel Thomé <emmanuel.th...@gmail.com>
> > wrote:
> >>
> >> Hi again,
> >>
> >> I've been able to simplify my test case significantly. It now runs
> >> with 2 nodes, and only a single MPI_Send / MPI_Recv pair is used.
> >>
> >> The pattern is as follows.
> >>
> >> * - ranks 0 and 1 both own a local buffer.
> >> * - each fills it with (deterministically known) data.
> >> * - rank 0 collects the data from rank 1's local buffer
> >> * (whose contents should be no mystery), and writes this to a
> >> * file-backed mmaped area.
> >> * - rank 0 compares what it receives with what it knows it *should
> >> * have* received.
> >>
> >> The test fails if:
> >>
> >> * - the openib btl is used among the 2 nodes
> >> * - a file-backed mmaped area is used for receiving the data.
> >> * - the write is done to a newly created file.
> >> * - per-node buffer is large enough.
> >>
> >> For a per-node buffer size above 12kb (12240 bytes to be exact), my
> >> program fails, since the MPI_Recv does not receive the correct data
> >> chunk (it just gets zeroes).
> >>
> >> I attach the simplified test case. I hope someone will be able to
> >> reproduce the problem.
> >>
> >> Best regards,
> >>
> >> E.
> >>
> >>
> >> On Mon, Nov 10, 2014 at 5:48 PM, Emmanuel Thomé
> >> <emmanuel.th...@gmail.com> wrote:
> >> > Thanks for your answer.
> >> >
> >> > On Mon, Nov 10, 2014 at 4:31 PM, Joshua Ladd <jladd.m...@gmail.com>
> >> > wrote:
> >> >> Just really quick off the top of my head, mmaping relies on the virtual
> >> >> memory subsystem, whereas IB RDMA operations rely on physical memory
> >> >> being
> >> >> pinned (unswappable.)
> >> >
> >> > Yes. Does that mean that the result of computations should be
> >> > undefined if I happen to give a user buffer which corresponds to a
> >> > file ? That would be surprising.
> >> >
> >> >> For a large message transfer, the OpenIB BTL will
> >> >> register the user buffer, which will pin the pages and make them
> >> >> unswappable.
> >> >
> >> > Yes. But what are the semantics of pinning the VM area pointed to by
> >> > ptr if ptr happens to be mmaped from a file ?
> >> >
> >> >> If the data being transfered is small, you'll copy-in/out to
> >> >> internal bounce buffers and you shouldn't have issues.
> >> >
> >> > Are you saying that the openib layer does have provision in this case
> >> > for letting the RDMA happen with a pinned physical memory range, and
> >> > later perform the copy to the file-backed mmaped range ? That would
> >> > make perfect sense indeed, although I don't have enough familiarity
> >> > with the OMPI code to see where it happens, and more importantly
> >> > whether the completion properly waits for this post-RDMA copy to
> >> > complete.
> >> >
> >> >
> >> >> 1.If you try to just bcast a few kilobytes of data using this
> >> >> technique, do
> >> >> you run into issues?
> >> >
> >> > No. All "simpler" attempts were successful, unfortunately. Can you be
> >> > a little bit more precise about what scenario you imagine ? The
> >> > setting "all ranks mmap a local file, and rank 0 broadcasts there" is
> >> > successful.
> >> >
> >> >> 2. How large is the data in the collective (input and output), is
> >> >> in_place
> >> >> used? I'm guess it's large enough that the BTL tries to work with the
> >> >> user
> >> >> buffer.
> >> >
> >> > MPI_IN_PLACE is used in reduce_scatter and allgather in the code.
> >> > Collectives are with communicators of 2 nodes, and we're talking (for
> >> > the smallest failing run) 8kb per node (i.e. 16kb total for an
> >> > allgather).
> >> >
> >> > E.
> >> >
> >> >> On Mon, Nov 10, 2014 at 9:29 AM, Emmanuel Thomé
> >> >> <emmanuel.th...@gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> Hi,
> >> >>>
> >> >>> I'm stumbling on a problem related to the openib btl in
> >> >>> openmpi-1.[78].*, and the (I think legitimate) use of file-backed
> >> >>> mmaped areas for receiving data through MPI collective calls.
> >> >>>
> >> >>> A test case is attached. I've tried to make is reasonably small,
> >> >>> although I recognize that it's not extra thin. The test case is a
> >> >>> trimmed down version of what I witness in the context of a rather
> >> >>> large program, so there is no claim of relevance of the test case
> >> >>> itself. It's here just to trigger the desired misbehaviour. The test
> >> >>> case contains some detailed information on what is done, and the
> >> >>> experiments I did.
> >> >>>
> >> >>> In a nutshell, the problem is as follows.
> >> >>>
> >> >>> - I do a computation, which involves MPI_Reduce_scatter and
> >> >>> MPI_Allgather.
> >> >>> - I save the result to a file (collective operation).
> >> >>>
> >> >>> *If* I save the file using something such as:
> >> >>> fd = open("blah", ...
> >> >>> area = mmap(..., fd, )
> >> >>> MPI_Gather(..., area, ...)
> >> >>> *AND* the MPI_Reduce_scatter is done with an alternative
> >> >>> implementation (which I believe is correct)
> >> >>> *AND* communication is done through the openib btl,
> >> >>>
> >> >>> then the file which gets saved is inconsistent with what is obtained
> >> >>> with the normal MPI_Reduce_scatter (alghough memory areas do coincide
> >> >>> before the save).
> >> >>>
> >> >>> I tried to dig a bit in the openib internals, but all I've been able
> >> >>> to witness was beyond my expertise (an RDMA read not transferring the
> >> >>> expected data, but I'm too uncomfortable with this layer to say
> >> >>> anything I'm sure about).
> >> >>>
> >> >>> Tests have been done with several openmpi versions including 1.8.3, on
> >> >>> a debian wheezy (7.5) + OFED 2.3 cluster.
> >> >>>
> >> >>> It would be great if someone could tell me if he is able to reproduce
> >> >>> the bug, or tell me whether something which is done in this test case
> >> >>> is illegal in any respect. I'd be glad to provide further information
> >> >>> which could be of any help.
> >> >>>
> >> >>> Best regards,
> >> >>>
> >> >>> E. Thomé.
> >> >>>
> >> >>> _______________________________________________
> >> >>> users mailing list
> >> >>> us...@open-mpi.org
> >> >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> >> >>> Link to this post:
> >> >>> http://www.open-mpi.org/community/lists/users/2014/11/25730.php
> >> >>
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> users mailing list
> >> >> us...@open-mpi.org
> >> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> >> >> Link to this post:
> >> >> http://www.open-mpi.org/community/lists/users/2014/11/25732.php
> >>
> >> _______________________________________________
> >> users mailing list
> >> us...@open-mpi.org
> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> >> Link to this post:
> >> http://www.open-mpi.org/community/lists/users/2014/11/25740.php
> >
> >
> >
> > _______________________________________________
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post:
> > http://www.open-mpi.org/community/lists/users/2014/11/25743.php
#define _GNU_SOURCE
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <assert.h>
#include <mpi.h>
#include <sys/mman.h>
/* This test file illustrates how in certain circumstances, an mmap area
* cannot correctly receive data sent from an MPI_Send call.
*
* This program wants to run on 2 distinct nodes connected with
* infiniband.
*
* Normal behaviour of the program consists in printing output similar
* to:
node 0 iteration 0, lead word received from peer is 0x00000401 [ok]
node 0 iteration 1, lead word received from peer is 0x00000801 [ok]
node 0 iteration 2, lead word received from peer is 0x00000c01 [ok]
node 0 iteration 3, lead word received from peer is 0x00001001 [ok]
*
* Abnormal behaviour is when the job ends with MPI_Abort after printing
* a line such as:
node 0 iteration 1, lead word received from peer is 0x00000000 [NOK]
*
* Each iteration of the main loop does the same thing.
* - rank 0 allocates a buffer with mmap
* - rank 1 sends data there with MPI_Send
* - rank 0 verifies that the data has been correctly received.
* - rank 0 frees the buffer with munmap
*
* The final check performed by rank 0 fails if the following conditions
* are met:
*
* - the openib btl is used among the 2 nodes
* - allocation is done via mmap/munmap (not via malloc/free)
* - the send is large enough.
*
* The first condition is controlled by the btl mca.
* The size of the transfer is controlled by the -s command line
* argument */
/* For compiling, one may do:
MPI=$HOME/Packages/openmpi-1.8.3
$MPI/bin/mpicc -W -Wall -std=c99 -O0 -g prog5.c
* For running, assuming /tmp/hosts contains the list of 2 nodes, and
* $SSH is used to connect to these:
SSH_AUTH_SOCK= DISPLAY= $MPI/bin/mpiexec -machinefile /tmp/hosts --mca plm_rsh_agent $SSH --mca rmaps_base_mapping_policy node -n 2 ./a.out -s 2048
*/
/*
* Tested (FAIL means that setting USE_MMAP_FOR_FILE_IO above yields to a
* program failure, while we succeed if it is unset).
*
* IB boards MCX353A-FCBT, fw rev 2.32.5100, MLNX_OFED_LINUX-2.3-1.0.1-debian7.5-x86_64
* openmpi-1.8.4rc1 FAIL (ok with --mca btl ^openib)
* openmpi-1.8.3 FAIL (ok with --mca btl ^openib)
*
* A previous, longer test case also failed with IB boards MHGH29-XTC.
*/
/* Passing --mca mpi_leave_pinned 0 eliminates the bug */
int main(int argc, char * argv[])
{
MPI_Init(&argc, &argv);
int size;
int rank;
int eitems = 1530; /* eitems >= 1530 seem to fail on my cluster */
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
if (size != 2) abort();
int use_mmap = 1;
for(argc--, argv++; argc ; ) {
if (argc >= 2 && strcmp(argv[0], "-s") == 0) {
eitems = atoi(argv[1]);
argc -= 2;
argv += 2;
continue;
}
if (strcmp(argv[0], "-malloc") == 0) {
use_mmap = 0;
argc--, argv++;
continue;
}
fprintf(stderr, "Unexpected: %s\n", argv[0]);
exit(EXIT_FAILURE);
}
size_t chunksize = eitems * sizeof(unsigned long);
size_t wsiz = ((chunksize - 1) | (sysconf (_SC_PAGESIZE)-1)) + 1;
unsigned long * localbuf = malloc(chunksize);
for(int iter = 0 ; iter < 4 ; iter++) {
unsigned long magic = (1 + iter) << 10;
int ok = 1;
if (rank == 1) {
for(int item = 0 ; item < eitems ; item++) {
localbuf[item] = magic + rank;
}
MPI_Send(localbuf, eitems, MPI_UNSIGNED_LONG, 0, 0, MPI_COMM_WORLD);
} else {
unsigned long * recvbuf;
if (use_mmap) {
recvbuf = mmap(NULL, wsiz, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
} else {
recvbuf = malloc(wsiz);
}
MPI_Recv(recvbuf, eitems, MPI_UNSIGNED_LONG, !rank, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
ok = (*recvbuf == magic + !rank);
fprintf(stderr, "node %d iteration %d, lead word received from peer is 0x%08lx [%s]\n", rank, iter, *recvbuf, ok?"ok":"NOK");
if (use_mmap) {
munmap(recvbuf, wsiz);
} else {
free(recvbuf);
}
}
/* only rank 0 has performed a new check */
MPI_Bcast(&ok, 1, MPI_INT, 0, MPI_COMM_WORLD);
if (!ok) MPI_Abort(MPI_COMM_WORLD, 1);
}
free(localbuf);
MPI_Finalize();
}
