Great observation Lawrence. https://www.slideshare.net/jsquyres/friends-dont-let-friends-leak-mpirequests
You can add the following option to --download-mpich --download-mpich-configure-arguments="--enable-error-messages=all --enable-g" then MPICH will report all MPI resources that have not been freed during the run. This helps catching missing waits, etc. We have a nightly test that utilizes this for the PETSc libraries. Barry > On May 30, 2019, at 6:48 AM, Lawrence Mitchell <we...@gmx.li> wrote: > > Hi Sanjay, > >> On 30 May 2019, at 08:58, Sanjay Govindjee via petsc-users >> <petsc-users@mcs.anl.gov> wrote: >> >> The problem seems to persist but with a different signature. Graphs >> attached as before. >> >> Totals with MPICH (NB: single run) >> >> For the CG/Jacobi data_exchange_total = 41,385,984; kspsolve_total >> = 38,289,408 >> For the GMRES/BJACOBI data_exchange_total = 41,324,544; kspsolve_total >> = 41,324,544 >> >> Just reading the MPI docs I am wondering if I need some sort of >> MPI_Wait/MPI_Waitall before my MPI_Barrier in the data exchange routine? >> I would have thought that with the blocking receives and the MPI_Barrier >> that everything will have fully completed and cleaned up before >> all processes exited the routine, but perhaps I am wrong on that. > > > Skimming the fortran code you sent you do: > > for i in ...: > call MPI_Isend(..., req, ierr) > > for i in ...: > call MPI_Recv(..., ierr) > > But you never call MPI_Wait on the request you got back from the Isend. So > the MPI library will never free the data structures it created. > > The usual pattern for these non-blocking communications is to allocate an > array for the requests of length nsend+nrecv and then do: > > for i in nsend: > call MPI_Isend(..., req[i], ierr) > for j in nrecv: > call MPI_Irecv(..., req[nsend+j], ierr) > > call MPI_Waitall(req, ..., ierr) > > I note also there's no need for the Barrier at the end of the routine, this > kind of communication does neighbourwise synchronisation, no need to add > (unnecessary) global synchronisation too. > > As an aside, is there a reason you don't use PETSc's VecScatter to manage > this global to local exchange? > > Cheers, > > Lawrence