Dear Jonathan, Daniel, Thank you for your responses. Just yesterday, I discovered and solved this problem (another remains). It wasn't a result of calls to .sweep. The time-varying boundary condition for one of the PDEs was being re-defined within the time-stepping loop using the PDE.faceGrad.constrain() method. This led to a net creation of objects with every timestep, irrespective of garbage collection call frequency, and that in turn caused the slow-down of the simulation.
The solution was to instead update the value of the boundary condition by the .setValue method within the time-stepping loop, as below. # Outside the loop, declare a FaceVariable for the value of the BC: species_flux_neg_particle_surf = FaceVariable(mesh=p2d_mesh) # Next, apply that value to the BC at the top of that mesh: Cs_p2d.faceGrad.constrain(species_flux_neg_particle_surf, where=p2d_mesh.facesTop) ..... # Within the time-stepping loop, update the boundary condition using .setValue as follows: species_flux_neg_particle_surf.setValue(my_new_BC_value) # Enjoy not creating new objects I haven't yet had time to finish producing a new vprof memory consumption plot for comparison. However, it's clear from Pympler’s SummaryTracker().print_diff() function that this change to the way the BC is updated solved the memory leak issue. For comparison, here are the number of objects and CPU time per timestep plotted against three seconds of simulation time, firstly using the .faceGrad.constrian() method and, secondly, using the .setValue() method for updating. The now-stable number of objects illustrates the fixed leak. faceGrad.constrain, leaking: https://goo.gl/3LqSm7 setValue, memory leak fixed: https://goo.gl/6kQMjH There is a new issue which also slows the simulation to an unusable level, described below. With the memory leak solved, I was able to run the simulation well beyond three seconds, and discovered that the number of sweeps required per timestep begins to exponentially increase after around 120s of simulation time. It seems that this in turn pulls up the CPU time required per timestep. The plot at the following link illustrates the problem: https://goo.gl/G9DD5r I do not know why this is. It's clear that a memory leak is no longer the cause - the number of objects is relatively constant (varying only slightly between garbage collector cycles). Plotting, at 1 & 150 seconds into the simulation, the residuals returned by the .sweep function for each of the six PDEs being solved in the time-stepping loop provides some insight into the stability of convergence. Each subplot in a figure is for one of the six PDEs: https://goo.gl/Nnm7Si At 150s, the residuals are still decreasing with sweeping, but at a much slower rate, towards the tolerance (1e-4). Do you know why this might be happening? With best regards, - Ian -----Original Message----- From: fipy-boun...@nist.gov [mailto:fipy-boun...@nist.gov] On Behalf Of Guyer, Jonathan E. Dr. (Fed) Sent: 11 October 2016 16:37 To: FIPY <FIPY@nist.gov> Subject: Re: Memory Leakage & Object Build-up with FiPy Sweeps I have access to their code. Ian, please provide an explicit recipe for demonstrating the leak with the code in your github repo. - Jon > On Oct 11, 2016, at 11:15 AM, Daniel Wheeler <daniel.wheel...@gmail.com> > wrote: > > Hi Ian, > > Could you possible post your code or a version of the code that demonstrates > the problem? Also, do you have the same issue with different solver suites? > > Cheers, > > Daniel > > > > On Fri, Sep 30, 2016 at 12:41 PM, Campbell, Ian <i.campbel...@imperial.ac.uk> > wrote: > Hi All, > > > > We are sweeping six PDEs in a time-stepping loop. We’ve noticed that as CPU > time progresses, the duration of each time-step increases, although the sweep > count remains constant. This is illustrated in the Excel file of data logged > from the simulation, which is available at the first hyperlink below. > > > > Hence, we suspected a memory leak may be occurring. After conducting > memory-focused line-profiling with the vprof tool, we observed a linear > increase in total memory consumption at a rate of approximately 3 MB per > timestep loop. This is evident in the graph at the second link below, which > illustrates the memory increase over three seconds of simulation. > > > > As a further step, we used Pympler to investigate the source of RAM > consumption increase for each timestep. The table below is an output from > Pympler’s SummaryTracker().print_diff(), which describe the additional > objects created within every time-step. Clearly, there are ~3.2 MB of > additional data being generated with every loop – this correlates perfectly > with the total rate of increase of memory consumption reported by vprof. > Although we are not yet sure, we suspect that the increasing time spent per > loop is the result of this apparent memory leak. > > > > We suspect this is the result of the calls to .sweep, since we are not > explicitly creating these objects. Can the origin of these objects be traced, > and furthermore, is there a way to avoid re-creating them and consuming more > memory with every loop? Without some method of unloading or preventing this > object build-up, it isn’t feasible to run our simulation for long durations. > > > dict > > 2684 > > 927.95 > > KB > > type > > 1716 > > 757.45 > > KB > > tuple > > 9504 > > 351.31 > > KB > > list > > 4781 > > 227.09 > > KB > > str > > 2582 > > 210.7 > > KB > > numpy.ndarray > > 396 > > 146.78 > > KB > > cell > > 3916 > > 107.08 > > KB > > property > > 2288 > > 98.31 > > KB > > weakref > > 2287 > > 98.27 > > KB > > function (getName) > > 1144 > > 67.03 > > KB > > function (getRank) > > 1144 > > 67.03 > > KB > > function (_calcValue_) > > 1144 > > 67.03 > > KB > > function (__init__) > > 1144 > > 67.03 > > KB > > function (_getRepresentation) > > 1012 > > 59.3 > > KB > > function (__setitem__) > > 572 > > 33.52 > > KB > > SUM > > 3285.88 > > KB > > > > > > https://imperialcollegelondon.box.com/s/zp9jj67du3mxdcfgbc4el8cqpxwnv0y4 > > > > https://imperialcollegelondon.box.com/s/ict9tnswqk9z57ovx8r3ll5po5ccrib9 > > > > With best regards, > > > > - Ian & Krishna > > > > P.S. Daniel, thank you very much for the excellent example solution you > provided in response to our question on obtaining the sharp discontinuity. > > > > Ian Campbell | PhD Candidate > > Electrochemical Science & Engineering Group > > Imperial College London, SW7 2AZ, United Kingdom > > > > > _______________________________________________ > fipy mailing list > fipy@nist.gov > http://www.ctcms.nist.gov/fipy > [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ] > > > > > -- > Daniel Wheeler > _______________________________________________ > fipy mailing list > fipy@nist.gov > http://www.ctcms.nist.gov/fipy > [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ] _______________________________________________ fipy mailing list fipy@nist.gov http://www.ctcms.nist.gov/fipy [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ] _______________________________________________ fipy mailing list fipy@nist.gov http://www.ctcms.nist.gov/fipy [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ]
