[OMPI users] One-sided datatype errors
Hi, I'm getting strange behavior using datatypes in a one-sided MPI_Accumulate operation. The attached example performs an accumulate into a patch of a shared 2d matrix. It uses indexed datatypes and can be built with displacement or absolute indices (hindexed) - both cases fail. I'm seeing data validation errors, hanging, or other erroneous behavior under OpenMPI 1.5 on Infiniband. The example works correctly under the current release of MVAPICH on IB and under MPICH on shared memory. Any help would be greatly appreciated. Best, ~Jim. /* One-Sided MPI 2-D Strided Accumulate Test * * Author: James Dinan * Date : December, 2010 * * This code performs N accumulates into a 2d patch of a shared array. The * array has dimensions [X, Y] and the subarray has dimensions [SUB_X, SUB_Y] * and begins at index [0, 0]. The input and output buffers are specified * using an MPI indexed type. */ #include #include #include #define XDIM 16 #define YDIM 16 #define SUB_XDIM 8 #define SUB_YDIM 8 #define ITERATIONS 1 int main(int argc, char **argv) { int i, j, rank, nranks, peer, bufsize, errors; double *win_buf, *src_buf; MPI_Win buf_win; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &nranks); bufsize = XDIM * YDIM * sizeof(double); MPI_Alloc_mem(bufsize, MPI_INFO_NULL, &win_buf); MPI_Alloc_mem(bufsize, MPI_INFO_NULL, &src_buf); if (rank == 0) printf("MPI RMA Strided Accumulate Test:\n"); for (i = 0; i < XDIM*YDIM; i++) { *(win_buf + i) = 1.0 + rank; *(src_buf + i) = 1.0 + rank; } MPI_Win_create(win_buf, bufsize, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &buf_win); peer = (rank+1) % nranks; // Perform ITERATIONS strided accumulate operations for (i = 0; i < ITERATIONS; i++) { MPI_Aint idx_loc[SUB_YDIM]; int idx_rem[SUB_YDIM]; int blk_len[SUB_YDIM]; MPI_Datatype src_type, dst_type; for (i = 0; i < SUB_YDIM; i++) { MPI_Get_address(&src_buf[i*XDIM], &idx_loc[i]); idx_rem[i] = i*XDIM; blk_len[i] = SUB_XDIM; } #ifdef ABSOLUTE MPI_Type_hindexed(SUB_YDIM, blk_len, idx_loc, MPI_DOUBLE, &src_type); #else MPI_Type_indexed(SUB_YDIM, blk_len, idx_rem, MPI_DOUBLE, &src_type); #endif MPI_Type_indexed(SUB_YDIM, blk_len, idx_rem, MPI_DOUBLE, &dst_type); MPI_Type_commit(&src_type); MPI_Type_commit(&dst_type); MPI_Win_lock(MPI_LOCK_EXCLUSIVE, peer, 0, buf_win); #ifdef ABSOLUTE MPI_Accumulate(MPI_BOTTOM, 1, src_type, peer, 0, 1, dst_type, MPI_SUM, buf_win); #else MPI_Accumulate(src_buf, 1, src_type, peer, 0, 1, dst_type, MPI_SUM, buf_win); #endif MPI_Win_unlock(peer, buf_win); MPI_Type_free(&src_type); MPI_Type_free(&dst_type); } MPI_Barrier(MPI_COMM_WORLD); // Verify that the results are correct MPI_Win_lock(MPI_LOCK_EXCLUSIVE, rank, 0, buf_win); errors = 0; for (i = 0; i < SUB_XDIM; i++) { for (j = 0; j < SUB_YDIM; j++) { const double actual = *(win_buf + i + j*XDIM); const double expected = (1.0 + rank) + (1.0 + ((rank+nranks-1)%nranks)) * (ITERATIONS); if (actual - expected > 1e-10) { printf("%d: Data validation failed at [%d, %d] expected=%f actual=%f\n", rank, j, i, expected, actual); errors++; fflush(stdout); } } } for (i = SUB_XDIM; i < XDIM; i++) { for (j = 0; j < SUB_YDIM; j++) { const double actual = *(win_buf + i + j*XDIM); const double expected = 1.0 + rank; if (actual - expected > 1e-10) { printf("%d: Data validation failed at [%d, %d] expected=%f actual=%f\n", rank, j, i, expected, actual); errors++; fflush(stdout); } } } for (i = 0; i < XDIM; i++) { for (j = SUB_YDIM; j < YDIM; j++) { const double actual = *(win_buf + i + j*XDIM); const double expected = 1.0 + rank; if (actual - expected > 1e-10) { printf("%d: Data validation failed at [%d, %d] expected=%f actual=%f\n", rank, j, i, expected, actual); errors++; fflush(stdout); } } } MPI_Win_unlock(rank, buf_win); MPI_Win_free(&buf_win); MPI_Free_mem(win_buf); MPI_Free_mem(src_buf); MPI_Finalize(); if (errors == 0) { printf("%d: Success\n", rank); return 0; } else { printf("%d: Fail\n", rank); return 1; } }
Re: [OMPI users] One-sided datatype errors
Hi James: I can reproduce the problem on a single node with Open MPI 1.5 and the trunk. I have submitted a ticket with the information. https://svn.open-mpi.org/trac/ompi/ticket/2656 Rolf On 12/13/10 18:44, James Dinan wrote: Hi, I'm getting strange behavior using datatypes in a one-sided MPI_Accumulate operation. The attached example performs an accumulate into a patch of a shared 2d matrix. It uses indexed datatypes and can be built with displacement or absolute indices (hindexed) - both cases fail. I'm seeing data validation errors, hanging, or other erroneous behavior under OpenMPI 1.5 on Infiniband. The example works correctly under the current release of MVAPICH on IB and under MPICH on shared memory. Any help would be greatly appreciated. Best, ~Jim. ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] One-sided datatype errors
Hi Rolf, Thanks for your help. I also noticed trouble with subarray data types. I attached the same test again, but with subarray rather than indexed types. It works correctly with MVAPICH on IB, but fails with OpenMPI 1.5 with the following message: $ mpiexec -n 2 ./a.out MPI RMA Strided Accumulate Test: [f3:1747] *** An error occurred in MPI_Accumlate [f3:1747] *** on win 3 [f3:1747] *** MPI_ERR_TYPE: invalid datatype [f3:1747] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) Thanks, ~Jim. On 12/14/2010 09:05 AM, Rolf vandeVaart wrote: Hi James: I can reproduce the problem on a single node with Open MPI 1.5 and the trunk. I have submitted a ticket with the information. https://svn.open-mpi.org/trac/ompi/ticket/2656 Rolf On 12/13/10 18:44, James Dinan wrote: Hi, I'm getting strange behavior using datatypes in a one-sided MPI_Accumulate operation. The attached example performs an accumulate into a patch of a shared 2d matrix. It uses indexed datatypes and can be built with displacement or absolute indices (hindexed) - both cases fail. I'm seeing data validation errors, hanging, or other erroneous behavior under OpenMPI 1.5 on Infiniband. The example works correctly under the current release of MVAPICH on IB and under MPICH on shared memory. Any help would be greatly appreciated. Best, ~Jim. ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users /* One-Sided MPI 2-D Strided Accumulate Test * * Author: James Dinan * Date : December, 2010 * * This code performs N accumulates into a 2d patch of a shared array. The * array has dimensions [X, Y] and the subarray has dimensions [SUB_X, SUB_Y] * and begins at index [0, 0]. The input and output buffers are specified * using an MPI subarray type. */ #include #include #include #define XDIM 1024 #define YDIM 1024 #define SUB_XDIM 512 #define SUB_YDIM 512 #define ITERATIONS 10 int main(int argc, char **argv) { int i, j, rank, nranks, peer, bufsize, errors; double *win_buf, *src_buf; MPI_Win buf_win; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &nranks); bufsize = XDIM * YDIM * sizeof(double); MPI_Alloc_mem(bufsize, MPI_INFO_NULL, &win_buf); MPI_Alloc_mem(bufsize, MPI_INFO_NULL, &src_buf); if (rank == 0) printf("MPI RMA Strided Accumulate Test:\n"); for (i = 0; i < XDIM*YDIM; i++) { *(win_buf + i) = 1.0 + rank; *(src_buf + i) = 1.0 + rank; } MPI_Win_create(win_buf, bufsize, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &buf_win); peer = (rank+1) % nranks; // Perform ITERATIONS strided accumulate operations for (i = 0; i < ITERATIONS; i++) { int ndims = 2; int src_arr_sizes[2]= { XDIM, YDIM }; int src_arr_subsizes[2] = { SUB_XDIM, SUB_YDIM }; int src_arr_starts[2] = {0,0 }; int dst_arr_sizes[2]= { XDIM, YDIM }; int dst_arr_subsizes[2] = { SUB_XDIM, SUB_YDIM }; int dst_arr_starts[2] = {0,0 }; MPI_Datatype src_type, dst_type; MPI_Type_create_subarray(ndims, src_arr_sizes, src_arr_subsizes, src_arr_starts, MPI_ORDER_C, MPI_DOUBLE, &src_type); MPI_Type_create_subarray(ndims, dst_arr_sizes, dst_arr_subsizes, dst_arr_starts, MPI_ORDER_C, MPI_DOUBLE, &dst_type); MPI_Type_commit(&src_type); MPI_Type_commit(&dst_type); MPI_Win_lock(MPI_LOCK_EXCLUSIVE, peer, 0, buf_win); MPI_Accumulate(src_buf, 1, src_type, peer, 0, 1, dst_type, MPI_SUM, buf_win); MPI_Win_unlock(peer, buf_win); MPI_Type_free(&src_type); MPI_Type_free(&dst_type); } MPI_Barrier(MPI_COMM_WORLD); // Verify that the results are correct MPI_Win_lock(MPI_LOCK_EXCLUSIVE, rank, 0, buf_win); errors = 0; for (i = 0; i < SUB_XDIM; i++) { for (j = 0; j < SUB_YDIM; j++) { const double actual = *(win_buf + i + j*XDIM); const double expected = (1.0 + rank) + (1.0 + ((rank+nranks-1)%nranks)) * (ITERATIONS); if (actual - expected > 1e-10) { printf("%d: Data validation failed at [%d, %d] expected=%f actual=%f\n", rank, j, i, expected, actual); errors++; fflush(stdout); } } } for (i = SUB_XDIM; i < XDIM; i++) { for (j = 0; j < SUB_YDIM; j++) { const double actual = *(win_buf + i + j*XDIM); const double expected = 1.0 + rank; if (actual - expected > 1e-10) { printf("%d: Data validation failed at [%d, %d] expected=%f actual=%f\n", rank, j, i, expected, actual); errors++; fflush(stdout);