1) Correct: Placing a WaitAll before the MPI_Barrier solve the problem
in our send-get routine for OPENMPI
2) Correct: The problem persists with KSPSolve
3) Correct: WaitAll did not fix the problem in our send-get nor in
KSPSolve when using MPICH
Also correct. Commenting out the call to KSPSolve results in zero
memory growth on OPENMPI.
On 5/30/19 11:59 AM, Smith, Barry F. wrote:
Thanks for the update. So the current conclusions are that using the
Waitall in your code
1) solves the memory issue with OpenMPI in your code
2) does not solve the memory issue with PETSc KSPSolve
3) MPICH has memory issues both for your code and PETSc KSPSolve (despite) the
wait all fix?
If you literately just comment out the call to KSPSolve() with OpenMPI is there
no growth in memory usage?
Both 2 and 3 are concerning, indicate possible memory leak bugs in MPICH and
not freeing all MPI resources in KSPSolve()
Junchao, can you please investigate 2 and 3 with, for example, a TS example
that uses the linear solver (like with -ts_type beuler)? Thanks
Barry
On May 30, 2019, at 1:47 PM, Sanjay Govindjee <s...@berkeley.edu> wrote:
Lawrence,
Thanks for taking a look! This is what I had been wondering about -- my
knowledge of MPI is pretty minimal and
this origins of the routine were from a programmer we hired a decade+ back from
NERSC. I'll have to look into
VecScatter. It will be great to dispense with our roll-your-own routines (we
even have our own reduceALL scattered around the code).
Interestingly, the MPI_WaitALL has solved the problem when using OpenMPI but it
still persists with MPICH. Graphs attached.
I'm going to run with openmpi for now (but I guess I really still need to
figure out what is wrong with MPICH and WaitALL;
I'll try Barry's suggestion of
--download-mpich-configure-arguments="--enable-error-messages=all --enable-g"
later today and report back).
Regarding MPI_Barrier, it was put in due a problem that some processes were
finishing up sending and receiving and exiting the subroutine
before the receiving processes had completed (which resulted in data loss as
the buffers are freed after the call to the routine). MPI_Barrier was the
solution proposed
to us. I don't think I can dispense with it, but will think about some more.
I'm not so sure about using MPI_IRecv as it will require a bit of rewriting
since right now I process the received
data sequentially after each blocking MPI_Recv -- clearly slower but easier to
code.
Thanks again for the help.
-sanjay
On 5/30/19 4:48 AM, Lawrence Mitchell wrote:
Hi Sanjay,
On 30 May 2019, at 08:58, Sanjay Govindjee via petsc-users
<petsc-users@mcs.anl.gov> wrote:
The problem seems to persist but with a different signature. Graphs attached
as before.
Totals with MPICH (NB: single run)
For the CG/Jacobi data_exchange_total = 41,385,984; kspsolve_total =
38,289,408
For the GMRES/BJACOBI data_exchange_total = 41,324,544; kspsolve_total =
41,324,544
Just reading the MPI docs I am wondering if I need some sort of
MPI_Wait/MPI_Waitall before my MPI_Barrier in the data exchange routine?
I would have thought that with the blocking receives and the MPI_Barrier that
everything will have fully completed and cleaned up before
all processes exited the routine, but perhaps I am wrong on that.
Skimming the fortran code you sent you do:
for i in ...:
call MPI_Isend(..., req, ierr)
for i in ...:
call MPI_Recv(..., ierr)
But you never call MPI_Wait on the request you got back from the Isend. So the
MPI library will never free the data structures it created.
The usual pattern for these non-blocking communications is to allocate an array
for the requests of length nsend+nrecv and then do:
for i in nsend:
call MPI_Isend(..., req[i], ierr)
for j in nrecv:
call MPI_Irecv(..., req[nsend+j], ierr)
call MPI_Waitall(req, ..., ierr)
I note also there's no need for the Barrier at the end of the routine, this
kind of communication does neighbourwise synchronisation, no need to add
(unnecessary) global synchronisation too.
As an aside, is there a reason you don't use PETSc's VecScatter to manage this
global to local exchange?
Cheers,
Lawrence
<cg_mpichwall.png><cg_wall.png><gmres_mpichwall.png><gmres_wall.png>
