> On Aug 19, 2020, at 9:39 PM, Matthew Knepley <knep...@gmail.com> wrote:
>
> Jed is more knowledgeable about the communication, but I have a simple
> question about the FEM method. Normally, the way
> we divide unknowns is that the only unknowns which might have entries
> computed off-process are those on the partition boundary.
> However, it sounds like you have a huge number of communicated values. Is it
> possible that the division of rows in your matrix does
> not match the division of the cells you compute element matrices for?
I hope that is not the case. I am using libMesh to manage the mesh and creation
of sparsity pattern, which uses Parmetis to create the partitions.
libMesh ensures that off-process entries are only at the partition boundary
(unless an extra set of DoFs are marked for coupling.
I also printed and looked at the n_nz and n_oz values on each rank and it does
not seem to raise any flags.
I will try to dig in a bit further to make sure everything checks out.
Looking at the screenshots I had shared yesterday, all processes are in this
function:
PetscErrorCode MatAssemblyEnd_MPIAIJ(Mat mat,MatAssemblyType mode)
{
Mat_MPIAIJ *aij = (Mat_MPIAIJ*)mat->data;
Mat_SeqAIJ *a = (Mat_SeqAIJ*)aij->A->data;
PetscErrorCode ierr;
PetscMPIInt n;
PetscInt i,j,rstart,ncols,flg;
PetscInt *row,*col;
PetscBool other_disassembled;
PetscScalar *val;
/* do not use 'b = (Mat_SeqAIJ*)aij->B->data' as B can be reset in
disassembly */
PetscFunctionBegin;
if (!aij->donotstash && !mat->nooffprocentries) {
while (1) {
ierr =
MatStashScatterGetMesg_Private(&mat->stash,&n,&row,&col,&val,&flg);CHKERRQ(ierr);
if (!flg) break;
for (i=0; i<n; ) {
/* Now identify the consecutive vals belonging to the same row */
for (j=i,rstart=row[j]; j<n; j++) {
if (row[j] != rstart) break;
}
if (j < n) ncols = j-i;
else ncols = n-i;
/* Now assemble all these values with a single function call */
ierr =
MatSetValues_MPIAIJ(mat,1,row+i,ncols,col+i,val+i,mat->insertmode);CHKERRQ(ierr);
i = j;
}
}
ierr = MatStashScatterEnd_Private(&mat->stash);CHKERRQ(ierr);
}
ierr = MatAssemblyBegin(aij->A,mode);CHKERRQ(ierr);
ierr = MatAssemblyEnd(aij->A,mode);CHKERRQ(ierr);
/* determine if any processor has disassembled, if so we must
also disassemble ourselfs, in order that we may reassemble. */
/*
if nonzero structure of submatrix B cannot change then we know that
no processor disassembled thus we can skip this stuff
*/
if (!((Mat_SeqAIJ*)aij->B->data)->nonew) {
ierr =
MPIU_Allreduce(&mat->was_assembled,&other_disassembled,1,MPIU_BOOL,MPI_PROD,PetscObjectComm((PetscObject)mat));CHKERRQ(ierr);
if (mat->was_assembled && !other_disassembled) {
ierr = MatDisAssemble_MPIAIJ(mat);CHKERRQ(ierr);
}
}
if (!mat->was_assembled && mode == MAT_FINAL_ASSEMBLY) {
ierr = MatSetUpMultiply_MPIAIJ(mat);CHKERRQ(ierr);
}
ierr = MatSetOption(aij->B,MAT_USE_INODES,PETSC_FALSE);CHKERRQ(ierr);
ierr = MatAssemblyBegin(aij->B,mode);CHKERRQ(ierr);
ierr = MatAssemblyEnd(aij->B,mode);CHKERRQ(ierr);
ierr = PetscFree2(aij->rowvalues,aij->rowindices);CHKERRQ(ierr);
aij->rowvalues = 0;
ierr = VecDestroy(&aij->diag);CHKERRQ(ierr);
if (a->inode.size) mat->ops->multdiagonalblock = MatMultDiagonalBlock_MPIAIJ;
/* if no new nonzero locations are allowed in matrix then only set the matrix
state the first time through */
if ((!mat->was_assembled && mode == MAT_FINAL_ASSEMBLY) ||
!((Mat_SeqAIJ*)(aij->A->data))->nonew) {
PetscObjectState state = aij->A->nonzerostate + aij->B->nonzerostate;
ierr =
MPIU_Allreduce(&state,&mat->nonzerostate,1,MPIU_INT64,MPI_SUM,PetscObjectComm((PetscObject)mat));CHKERRQ(ierr);
}
PetscFunctionReturn(0);
}
I noticed that of the 8 MPI processes, 2 were stuck at
ierr =
MatStashScatterGetMesg_Private(&mat->stash,&n,&row,&col,&val,&flg);CHKERRQ(ierr);
Other two were stuck at
ierr = MatStashScatterEnd_Private(&mat->stash);CHKERRQ(ierr);
And remaining four were under
ierr = MatSetUpMultiply_MPIAIJ(mat);CHKERRQ(ierr);
Is it expected for processes to be at different stages in this function?
-Manav
