Randy, Just email me the PetscMallocDump output file directly.
Barry > On Mar 6, 2015, at 12:10 PM, Randall Mackie <rlmackie...@gmail.com> wrote: > > Hi Barry, > > I've followed your suggestion to use PetscMallocDump after the call to > KSPSolve and I've looked at the output. > > What I find is that for a smaller sized model that I can run on my laptop, I > don't see any increase in the memory after it is allocated the first pass > through. > > However, with this other much larger model, I do clearly see an increase in > memory after each call to that particular KSPSolve: > > [0]Total space allocated 1885177504 bytes > [0]Total space allocated 2034781200 bytes > [0]Total space allocated 2184383136 bytes > [0]Total space allocated 2333985072 bytes > [0]Total space allocated 2483587008 bytes > > > That model is too large to run on just 1 process, but I've got it down to 8 > processes, and each process shows a similar increase in memory. > > Just to confirm that it was happening in that KSPSolve call, I put an > additional PetscMallocDump before and after, and I can confirm that it is > clearly increasing the memory after the call to KSPSolve. > > I've tried sorting through the output of that malloc dump, but the file is 4 > Mbytes in size for process 0, so I've tried grepping for various things, like > GAMG, etc, and I don't yet see what is causing the increase in memory. > > Do you guys have any sophisticated scripts to parse the output with? > > If you'd like the output file, I can put it somewhere, I just don't want to > drop a 4 Mbyte file in everyone's inbox. > > As another datapoint, I did save the matrix and vector to a binary file using > Vec/Mat View, then wrote a little test program that read them in (Vec/Mat > Load) and did the KSPSolve. I did not see any memory issues with that. > > So, I am unsure what to do at this point, or how to proceed to track this > down. > > Any suggestions appreciated. > > > Randy > > >> On Mar 5, 2015, at 2:41 PM, Barry Smith <bsm...@mcs.anl.gov> wrote: >> >> >> Randy, >> >> I've not been able to reproduce this; let us know if get to the point of >> having something we can run and debug. >> >> Barry >> >>> On Mar 4, 2015, at 7:45 PM, Randall Mackie <rlmackie...@gmail.com> wrote: >>> >>> In my application, I am repeatedly calling KSPSolve with the following >>> options: >>> >>> -ksp_type gmres \ >>> -pc_type gamg \ >>> -pc_gamg_type agg \ >>> -pc_gamg_agg_nsmooths 1\ >>> >>> >>> each call is after the matrix and right hand side have been updated. >>> >>> This works well in the sense that it solves the system in a reasonable >>> number of steps, however, I have noticed that the memory footprint of the >>> application increases by about 500 Mbytes after each call to KSPSolve (this >>> is a big problem), and after several calls, I've maxed out the memory. >>> >>> Is this expected behavior? >>> >>> I've combed through my code looking to make sure I don't have any memory >>> leaks, and so far I haven't found any (doesn't mean there are not there). >>> >>> However, when I use another PC, like jacobi, just to compare, I don't see >>> this memory issue, or if I comment out that call to KSPSolve (there is a >>> lot of other stuff going on in the code besides this call), I don't see >>> this issue. >>> >>> I've tried to destroy the KSP after each solve and recreate it each time, >>> but there still seems to be some memory getting added. >>> >>> I've tried to distill this down to a smaller problem and test program, but >>> so far I have been unsuccessful. >>> >>> >>> Is there a way to completely release the memory associated with the GAMG >>> preconditioner after a call to KSPSolve? >>> >>> >>> Any other suggestions for tracking this down? I've run out of ideas. >>> >>> >>> Thanks in advance, >>> >>> Randy >>> >>> >>> >>> >> >