[OMPI users] unexpected behavior when combining MPI_Gather and MPI_Type_vector

[Jonas Thies via users]

Dear OpenMPI community,

Here's a little puzzle for the Christmas holidays (although I would 
really appreciate a quick solution!).

I'm stuck with the following relatively basic problem: given a local 
nloc x m matrix X_p in column-major ordering on each MPI process p, 
perform a single MPI_Gather operation to construct the matrix
X_0
X_1
...

X_nproc

again, in col-major ordering. My approach is to use MPI_Type_vector to 
define an stype and an rtype, where stype has stride nloc, and rtype has 
stride nproc*nloc. The observation is that there is an unexpected 
displacement of (m-1)*n*p in the result array for the part arriving from 
process p.

The MFE code is attached, and I use OpenMPI 4.0.5 with GCC 11.2 
(although other versions and even distributions seem to display the same 
behavior). Example (nloc=3, nproc=3, m=2, with some additional columns 
printed for the sake of demonstration):


> mpicxx -o matrix_gather matrix_gather.cpp
mpirun -np 3 ./matrix_gather

v_loc on P0: 3x2
0 9
1 10
2 11

v_loc on P1: 3x2
3 12
4 13
5 14

v_loc on P2: 3x2
6 15
7 16
8 17

v_glob on P0: 9x4
0 9 0 0
1 10 0 0
2 11 0 0
0 3 12 0
0 4 13 0
0 5 14 0
0 0 6 15
0 0 7 16
0 0 8 17

Any ideas?

Thanks,

Jonas


--
*J. Thies*
Assistant Professor

TU Delft
Faculty Electrical Engineering, Mathematics and Computer Science
Institute of Applied Mathematics and High Performance Computing Center
Mekelweg 4
2628 CD Delft

T +31 15 27 XXXX
*j.th...@tudelft.nl*
#include <mpi.h>
#include <iostream>
#include <sstream>
#include <string>

void print(std::string label, int rank, int nloc, int m, int* array)
{
  std::ostringstream oss;
  oss << label << " on P"<<rank<<": "<< nloc << "x" << m << std::endl;

  for (int i=0; i<nloc; i++)
  {
    for (int j=0; j<m; j++)
    {
      oss << array[j*nloc+i] << " ";
    }
    oss << std::endl;
  }
  std::cout << oss.str()<<std::endl;
}

int main(int argc, char** argv){

MPI_Init(&argc,&argv);
int rank, nproc;
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
MPI_Comm_size(MPI_COMM_WORLD,&nproc);

int nloc=3;
int n=nloc*nproc;
int m=2;

int *v_loc = new int[nloc*m+1000];
int *v_glob = new int[n*m+1000];

for (int j=0; j<m; j++)
for (int i=0; i<nloc; i++)
v_loc[j*nloc+i]=j*n + rank*nloc + i;

for (int r=0; r<nproc; r++)
{
  if (rank==r) print("v_loc", rank, nloc, m, v_loc);
  std::cout << std::flush;
  MPI_Barrier(MPI_COMM_WORLD);
}

MPI_Datatype stype, rtype;

// this data typre represents the local nloc x m matrix,
// which is column-major and has stride nloc.
MPI_Type_vector(m,nloc,nloc,MPI_INT,&stype);
MPI_Type_commit(&stype);

// this represents a block of size nloc x m within a col-major
// matrix of size n x m, hence the stride is n.
MPI_Type_vector(m,nloc,n,MPI_INT,&rtype);
MPI_Type_commit(&rtype);

// these two result in the same thing:
//MPI_Allgather(v_loc,nloc*m,MPI_INT,v_glob,1,rtype,MPI_COMM_WORLD);
MPI_Allgather(v_loc,1,stype,v_glob,1,rtype,MPI_COMM_WORLD);

// note: print an additional column to show the displacement error we get:
if (!rank) print("v_glob", rank, n, m+nproc-1, v_glob);

MPI_Type_free(&stype);
MPI_Type_free(&rtype);

delete [] v_loc;
delete [] v_glob;

MPI_Finalize();
}