RE: Memory Leakage & Object Build-up with FiPy Sweeps
Dear All, Here is an update to my e-mail from yesterday which described the exponential slow-down of my simulation with increasing CPU time. I have a applied a higher level of under-relaxation (i.e. smaller values for the underRelaxation term in the .sweep method), and by 200 seconds of simulation time, the number of sweeps and CPU time per sweep is remaining relatively constant. I’m running a longer simulation now to verify this, although I am hesitant to say the issue is solved until I investigate further. Generally, the stability and rate of convergence are highly sensitive to the fixed under-relaxation factors used for each of the six .sweep methods within each time-step. Varying some relaxation factors by more than 0.01 can cause the simulation to either not converge at all, or to fail immediately with a StagnatedSolverWarning. Best regards, - Ian -Original Message- From: fipy-boun...@nist.gov [mailto:fipy-boun...@nist.gov] On Behalf Of Campbell, Ian Sent: 11 October 2016 18:21 To: fipy@nist.gov Subject: RE: Memory Leakage & Object Build-up with FiPy Sweeps Dear Jonathan, Daniel, Thank you for your responses. Just yesterday, I discovered and solved this problem (another remains). It wasn't a result of calls to .sweep. The time-varying boundary condition for one of the PDEs was being re-defined within the time-stepping loop using the PDE.faceGrad.constrain() method. This led to a net creation of objects with every timestep, irrespective of garbage collection call frequency, and that in turn caused the slow-down of the simulation. The solution was to instead update the value of the boundary condition by the .setValue method within the time-stepping loop, as below. # Outside the loop, declare a FaceVariable for the value of the BC: species_flux_neg_particle_surf = FaceVariable(mesh=p2d_mesh) # Next, apply that value to the BC at the top of that mesh: Cs_p2d.faceGrad.constrain(species_flux_neg_particle_surf, where=p2d_mesh.facesTop) . # Within the time-stepping loop, update the boundary condition using .setValue as follows: species_flux_neg_particle_surf.setValue(my_new_BC_value) # Enjoy not creating new objects I haven't yet had time to finish producing a new vprof memory consumption plot for comparison. However, it's clear from Pympler’s SummaryTracker().print_diff() function that this change to the way the BC is updated solved the memory leak issue. For comparison, here are the number of objects and CPU time per timestep plotted against three seconds of simulation time, firstly using the .faceGrad.constrian() method and, secondly, using the .setValue() method for updating. The now-stable number of objects illustrates the fixed leak. faceGrad.constrain, leaking: https://goo.gl/3LqSm7 setValue, memory leak fixed: https://goo.gl/6kQMjH There is a new issue which also slows the simulation to an unusable level, described below. With the memory leak solved, I was able to run the simulation well beyond three seconds, and discovered that the number of sweeps required per timestep begins to exponentially increase after around 120s of simulation time. It seems that this in turn pulls up the CPU time required per timestep. The plot at the following link illustrates the problem: https://goo.gl/G9DD5r I do not know why this is. It's clear that a memory leak is no longer the cause - the number of objects is relatively constant (varying only slightly between garbage collector cycles). Plotting, at 1 & 150 seconds into the simulation, the residuals returned by the .sweep function for each of the six PDEs being solved in the time-stepping loop provides some insight into the stability of convergence. Each subplot in a figure is for one of the six PDEs: https://goo.gl/Nnm7Si At 150s, the residuals are still decreasing with sweeping, but at a much slower rate, towards the tolerance (1e-4). Do you know why this might be happening? With best regards, - Ian -Original Message- From: fipy-boun...@nist.gov [mailto:fipy-boun...@nist.gov] On Behalf Of Guyer, Jonathan E. Dr. (Fed) Sent: 11 October 2016 16:37 To: FIPY Subject: Re: Memory Leakage & Object Build-up with FiPy Sweeps I have access to their code. Ian, please provide an explicit recipe for demonstrating the leak with the code in your github repo. - Jon > On Oct 11, 2016, at 11:15 AM, Daniel Wheeler > wrote: > > Hi Ian, > > Could you possible post your code or a version of the code that demonstrates > the problem? Also, do you have the same issue with different solver suites? > > Cheers, > > Daniel > > > > On Fri, Sep 30, 2016 at 12:41 PM, Campbell, Ian > wrote: > Hi All, > > > > We are sweeping six PDEs in a time-stepping loop. We’ve noticed that as CPU > time progresses, the duration of each time-step increases, although the sweep > count remains constant. T
RE: Memory Leakage & Object Build-up with FiPy Sweeps
Dear Jonathan, Daniel, Thank you for your responses. Just yesterday, I discovered and solved this problem (another remains). It wasn't a result of calls to .sweep. The time-varying boundary condition for one of the PDEs was being re-defined within the time-stepping loop using the PDE.faceGrad.constrain() method. This led to a net creation of objects with every timestep, irrespective of garbage collection call frequency, and that in turn caused the slow-down of the simulation. The solution was to instead update the value of the boundary condition by the .setValue method within the time-stepping loop, as below. # Outside the loop, declare a FaceVariable for the value of the BC: species_flux_neg_particle_surf = FaceVariable(mesh=p2d_mesh) # Next, apply that value to the BC at the top of that mesh: Cs_p2d.faceGrad.constrain(species_flux_neg_particle_surf, where=p2d_mesh.facesTop) . # Within the time-stepping loop, update the boundary condition using .setValue as follows: species_flux_neg_particle_surf.setValue(my_new_BC_value) # Enjoy not creating new objects I haven't yet had time to finish producing a new vprof memory consumption plot for comparison. However, it's clear from Pympler’s SummaryTracker().print_diff() function that this change to the way the BC is updated solved the memory leak issue. For comparison, here are the number of objects and CPU time per timestep plotted against three seconds of simulation time, firstly using the .faceGrad.constrian() method and, secondly, using the .setValue() method for updating. The now-stable number of objects illustrates the fixed leak. faceGrad.constrain, leaking: https://goo.gl/3LqSm7 setValue, memory leak fixed: https://goo.gl/6kQMjH There is a new issue which also slows the simulation to an unusable level, described below. With the memory leak solved, I was able to run the simulation well beyond three seconds, and discovered that the number of sweeps required per timestep begins to exponentially increase after around 120s of simulation time. It seems that this in turn pulls up the CPU time required per timestep. The plot at the following link illustrates the problem: https://goo.gl/G9DD5r I do not know why this is. It's clear that a memory leak is no longer the cause - the number of objects is relatively constant (varying only slightly between garbage collector cycles). Plotting, at 1 & 150 seconds into the simulation, the residuals returned by the .sweep function for each of the six PDEs being solved in the time-stepping loop provides some insight into the stability of convergence. Each subplot in a figure is for one of the six PDEs: https://goo.gl/Nnm7Si At 150s, the residuals are still decreasing with sweeping, but at a much slower rate, towards the tolerance (1e-4). Do you know why this might be happening? With best regards, - Ian -Original Message- From: fipy-boun...@nist.gov [mailto:fipy-boun...@nist.gov] On Behalf Of Guyer, Jonathan E. Dr. (Fed) Sent: 11 October 2016 16:37 To: FIPY Subject: Re: Memory Leakage & Object Build-up with FiPy Sweeps I have access to their code. Ian, please provide an explicit recipe for demonstrating the leak with the code in your github repo. - Jon > On Oct 11, 2016, at 11:15 AM, Daniel Wheeler > wrote: > > Hi Ian, > > Could you possible post your code or a version of the code that demonstrates > the problem? Also, do you have the same issue with different solver suites? > > Cheers, > > Daniel > > > > On Fri, Sep 30, 2016 at 12:41 PM, Campbell, Ian > wrote: > Hi All, > > > > We are sweeping six PDEs in a time-stepping loop. We’ve noticed that as CPU > time progresses, the duration of each time-step increases, although the sweep > count remains constant. This is illustrated in the Excel file of data logged > from the simulation, which is available at the first hyperlink below. > > > > Hence, we suspected a memory leak may be occurring. After conducting > memory-focused line-profiling with the vprof tool, we observed a linear > increase in total memory consumption at a rate of approximately 3 MB per > timestep loop. This is evident in the graph at the second link below, which > illustrates the memory increase over three seconds of simulation. > > > > As a further step, we used Pympler to investigate the source of RAM > consumption increase for each timestep. The table below is an output from > Pympler’s SummaryTracker().print_diff(), which describe the additional > objects created within every time-step. Clearly, there are ~3.2 MB of > additional data being generated with every loop – this correlates perfectly > with the total rate of increase of memory consumption reported by vprof. > Although we are not yet sure, we suspect that the increasing time spent per > loop is the
Re: Memory Leakage & Object Build-up with FiPy Sweeps
Hi all, Could Ian's issue be related to the issues I came across at the end of March? Trevor Keller seems to have isolated my memory leak to versions of Trilinos newer than 12.0. Trevor, It seems that I never got around to testing an older version of Trilinos, I'll do that now. -Mike On 10/11/2016 10:36 AM, Guyer, Jonathan E. Dr. (Fed) wrote: > I have access to their code. Ian, please provide an explicit recipe for > demonstrating the leak with the code in your github repo. > > - Jon > >> On Oct 11, 2016, at 11:15 AM, Daniel Wheeler >> wrote: >> >> Hi Ian, >> >> Could you possible post your code or a version of the code that demonstrates >> the problem? Also, do you have the same issue with different solver suites? >> >> Cheers, >> >> Daniel >> >> >> >> On Fri, Sep 30, 2016 at 12:41 PM, Campbell, Ian >> wrote: >> Hi All, >> >> >> >> We are sweeping six PDEs in a time-stepping loop. We’ve noticed that as CPU >> time progresses, the duration of each time-step increases, although the >> sweep count remains constant. This is illustrated in the Excel file of data >> logged from the simulation, which is available at the first hyperlink below. >> >> >> >> Hence, we suspected a memory leak may be occurring. After conducting >> memory-focused line-profiling with the vprof tool, we observed a linear >> increase in total memory consumption at a rate of approximately 3 MB per >> timestep loop. This is evident in the graph at the second link below, which >> illustrates the memory increase over three seconds of simulation. >> >> >> >> As a further step, we used Pympler to investigate the source of RAM >> consumption increase for each timestep. The table below is an output from >> Pympler’s SummaryTracker().print_diff(), which describe the additional >> objects created within every time-step. Clearly, there are ~3.2 MB of >> additional data being generated with every loop – this correlates perfectly >> with the total rate of increase of memory consumption reported by vprof. >> Although we are not yet sure, we suspect that the increasing time spent per >> loop is the result of this apparent memory leak. >> >> >> >> We suspect this is the result of the calls to .sweep, since we are not >> explicitly creating these objects. Can the origin of these objects be >> traced, and furthermore, is there a way to avoid re-creating them and >> consuming more memory with every loop? Without some method of unloading or >> preventing this object build-up, it isn’t feasible to run our simulation for >> long durations. >> >> >> dict >> >> 2684 >> >> 927.95 >> >> KB >> >> type >> >> 1716 >> >> 757.45 >> >> KB >> >> tuple >> >> 9504 >> >> 351.31 >> >> KB >> >> list >> >> 4781 >> >> 227.09 >> >> KB >> >> str >> >> 2582 >> >> 210.7 >> >> KB >> >> numpy.ndarray >> >> 396 >> >> 146.78 >> >> KB >> >> cell >> >> 3916 >> >> 107.08 >> >> KB >> >> property >> >> 2288 >> >> 98.31 >> >> KB >> >> weakref >> >> 2287 >> >> 98.27 >> >> KB >> >> function (getName) >> >> 1144 >> >> 67.03 >> >> KB >> >> function (getRank) >> >> 1144 >> >> 67.03 >> >> KB >> >> function (_calcValue_) >> >> 1144 >> >> 67.03 >> >> KB >> >> function (__init__) >> >> 1144 >> >> 67.03 >> >> KB >> >> function (_getRepresentation) >> >> 1012 >> >> 59.3 >> >> KB >> >> function (__setitem__) >> >> 572 >> >> 33.52 >> >> KB >> >> SUM >> >> 3285.88 >> >> KB >> >> >> >> >> >> https://imperialcollegelondon.box.com/s/zp9jj67du3mxdcfgbc4el8cqpxwnv0y4 >> >> >> >> https://imperialcollegelondon.box.com/s/ict9tnswqk9z57ovx8r3ll5po5ccrib9 >> >> >> >> With best regards, >> >> >> >> - Ian & Krishna >> >> >> >> P.S. Daniel, thank you very much for the excellent example solution you >> provided in response to our question on obtaining the sharp discontinuity. >> >> >> >> Ian Campbell | PhD Candidate >> >> Electrochemical Science & Engineering Group >> >> Imperial College London, SW7 2AZ, United Kingdom >> >> >> >> >> ___ >> fipy mailing list >> fipy@nist.gov >> http://www.ctcms.nist.gov/fipy >>[ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ] >> >> >> >> >> -- >> Daniel Wheeler >> ___ >> fipy mailing list >> fipy@nist.gov >> http://www.ctcms.nist.gov/fipy >> [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ] > > ___ > fipy mailing list > fipy@nist.gov > http://www.ctcms.nist.gov/fipy >[ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ] ___ fipy mailing list fipy@nist.gov http://www.ctcms.nist.gov/fipy [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ]
Re: Memory Leakage & Object Build-up with FiPy Sweeps
I have access to their code. Ian, please provide an explicit recipe for demonstrating the leak with the code in your github repo. - Jon > On Oct 11, 2016, at 11:15 AM, Daniel Wheeler > wrote: > > Hi Ian, > > Could you possible post your code or a version of the code that demonstrates > the problem? Also, do you have the same issue with different solver suites? > > Cheers, > > Daniel > > > > On Fri, Sep 30, 2016 at 12:41 PM, Campbell, Ian > wrote: > Hi All, > > > > We are sweeping six PDEs in a time-stepping loop. We’ve noticed that as CPU > time progresses, the duration of each time-step increases, although the sweep > count remains constant. This is illustrated in the Excel file of data logged > from the simulation, which is available at the first hyperlink below. > > > > Hence, we suspected a memory leak may be occurring. After conducting > memory-focused line-profiling with the vprof tool, we observed a linear > increase in total memory consumption at a rate of approximately 3 MB per > timestep loop. This is evident in the graph at the second link below, which > illustrates the memory increase over three seconds of simulation. > > > > As a further step, we used Pympler to investigate the source of RAM > consumption increase for each timestep. The table below is an output from > Pympler’s SummaryTracker().print_diff(), which describe the additional > objects created within every time-step. Clearly, there are ~3.2 MB of > additional data being generated with every loop – this correlates perfectly > with the total rate of increase of memory consumption reported by vprof. > Although we are not yet sure, we suspect that the increasing time spent per > loop is the result of this apparent memory leak. > > > > We suspect this is the result of the calls to .sweep, since we are not > explicitly creating these objects. Can the origin of these objects be traced, > and furthermore, is there a way to avoid re-creating them and consuming more > memory with every loop? Without some method of unloading or preventing this > object build-up, it isn’t feasible to run our simulation for long durations. > > > dict > > 2684 > > 927.95 > > KB > > type > > 1716 > > 757.45 > > KB > > tuple > > 9504 > > 351.31 > > KB > > list > > 4781 > > 227.09 > > KB > > str > > 2582 > > 210.7 > > KB > > numpy.ndarray > > 396 > > 146.78 > > KB > > cell > > 3916 > > 107.08 > > KB > > property > > 2288 > > 98.31 > > KB > > weakref > > 2287 > > 98.27 > > KB > > function (getName) > > 1144 > > 67.03 > > KB > > function (getRank) > > 1144 > > 67.03 > > KB > > function (_calcValue_) > > 1144 > > 67.03 > > KB > > function (__init__) > > 1144 > > 67.03 > > KB > > function (_getRepresentation) > > 1012 > > 59.3 > > KB > > function (__setitem__) > > 572 > > 33.52 > > KB > > SUM > > 3285.88 > > KB > > > > > > https://imperialcollegelondon.box.com/s/zp9jj67du3mxdcfgbc4el8cqpxwnv0y4 > > > > https://imperialcollegelondon.box.com/s/ict9tnswqk9z57ovx8r3ll5po5ccrib9 > > > > With best regards, > > > > - Ian & Krishna > > > > P.S. Daniel, thank you very much for the excellent example solution you > provided in response to our question on obtaining the sharp discontinuity. > > > > Ian Campbell | PhD Candidate > > Electrochemical Science & Engineering Group > > Imperial College London, SW7 2AZ, United Kingdom > > > > > ___ > fipy mailing list > fipy@nist.gov > http://www.ctcms.nist.gov/fipy > [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ] > > > > > -- > Daniel Wheeler > ___ > fipy mailing list > fipy@nist.gov > http://www.ctcms.nist.gov/fipy > [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ] ___ fipy mailing list fipy@nist.gov http://www.ctcms.nist.gov/fipy [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ]
Re: Memory Leakage & Object Build-up with FiPy Sweeps
Hi Ian, Could you possible post your code or a version of the code that demonstrates the problem? Also, do you have the same issue with different solver suites? Cheers, Daniel On Fri, Sep 30, 2016 at 12:41 PM, Campbell, Ian wrote: > Hi All, > > > > We are sweeping six PDEs in a time-stepping loop. We’ve noticed that as > CPU time progresses, the duration of each time-step increases, although the > sweep count remains constant. This is illustrated in the Excel file of data > logged from the simulation, which is available at the first hyperlink below. > > > > Hence, we suspected a memory leak may be occurring. After conducting > memory-focused line-profiling with the vprof tool, we observed a linear > increase in total memory consumption at a rate of approximately 3 MB per > timestep loop. This is evident in the graph at the second link below, which > illustrates the memory increase over three seconds of simulation. > > > > As a further step, we used Pympler to investigate the source of RAM > consumption increase for each timestep. The table below is an output from > Pympler’s SummaryTracker().print_diff(), which describe the additional > objects created within every time-step. Clearly, there are ~3.2 MB of > additional data being generated with every loop – this correlates perfectly > with the total rate of increase of memory consumption reported by vprof. > Although we are not yet sure, we suspect that the increasing time spent per > loop is the result of this apparent memory leak. > > > > We suspect this is the result of the calls to .sweep, since we are not > explicitly creating these objects. Can the origin of these objects be > traced, and furthermore, is there a way to avoid re-creating them and > consuming more memory with every loop? Without some method of unloading or > preventing this object build-up, it isn’t feasible to run our simulation > for long durations. > > dict > > 2684 > > 927.95 > > KB > > type > > 1716 > > 757.45 > > KB > > tuple > > 9504 > > 351.31 > > KB > > list > > 4781 > > 227.09 > > KB > > str > > 2582 > > 210.7 > > KB > > numpy.ndarray > > 396 > > 146.78 > > KB > > cell > > 3916 > > 107.08 > > KB > > property > > 2288 > > 98.31 > > KB > > weakref > > 2287 > > 98.27 > > KB > > function (getName) > > 1144 > > 67.03 > > KB > > function (getRank) > > 1144 > > 67.03 > > KB > > function (_calcValue_) > > 1144 > > 67.03 > > KB > > function (__init__) > > 1144 > > 67.03 > > KB > > function (_getRepresentation) > > 1012 > > 59.3 > > KB > > function (__setitem__) > > 572 > > 33.52 > > KB > > SUM > > 3285.88 > > KB > > > > > > https://imperialcollegelondon.box.com/s/zp9jj67du3mxdcfgbc4el8cqpxwnv0y4 > > > > https://imperialcollegelondon.box.com/s/ict9tnswqk9z57ovx8r3ll5po5ccrib9 > > > > With best regards, > > > > - Ian & Krishna > > > > P.S. Daniel, thank you very much for the excellent example solution you > provided in response to our question on obtaining the sharp discontinuity. > > > > Ian Campbell | PhD Candidate > > Electrochemical Science & Engineering Group > > Imperial College London, SW7 2AZ, United Kingdom > > > > ___ > fipy mailing list > fipy@nist.gov > http://www.ctcms.nist.gov/fipy > [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ] > > -- Daniel Wheeler ___ fipy mailing list fipy@nist.gov http://www.ctcms.nist.gov/fipy [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ]
Memory Leakage & Object Build-up with FiPy Sweeps
Hi All, We are sweeping six PDEs in a time-stepping loop. We've noticed that as CPU time progresses, the duration of each time-step increases, although the sweep count remains constant. This is illustrated in the Excel file of data logged from the simulation, which is available at the first hyperlink below. Hence, we suspected a memory leak may be occurring. After conducting memory-focused line-profiling with the vprof tool, we observed a linear increase in total memory consumption at a rate of approximately 3 MB per timestep loop. This is evident in the graph at the second link below, which illustrates the memory increase over three seconds of simulation. As a further step, we used Pympler to investigate the source of RAM consumption increase for each timestep. The table below is an output from Pympler's SummaryTracker().print_diff(), which describe the additional objects created within every time-step. Clearly, there are ~3.2 MB of additional data being generated with every loop - this correlates perfectly with the total rate of increase of memory consumption reported by vprof. Although we are not yet sure, we suspect that the increasing time spent per loop is the result of this apparent memory leak. We suspect this is the result of the calls to .sweep, since we are not explicitly creating these objects. Can the origin of these objects be traced, and furthermore, is there a way to avoid re-creating them and consuming more memory with every loop? Without some method of unloading or preventing this object build-up, it isn't feasible to run our simulation for long durations. dict 2684 927.95 KB type 1716 757.45 KB tuple 9504 351.31 KB list 4781 227.09 KB str 2582 210.7 KB numpy.ndarray 396 146.78 KB cell 3916 107.08 KB property 2288 98.31 KB weakref 2287 98.27 KB function (getName) 1144 67.03 KB function (getRank) 1144 67.03 KB function (_calcValue_) 1144 67.03 KB function (__init__) 1144 67.03 KB function (_getRepresentation) 1012 59.3 KB function (__setitem__) 572 33.52 KB SUM 3285.88 KB https://imperialcollegelondon.box.com/s/zp9jj67du3mxdcfgbc4el8cqpxwnv0y4 https://imperialcollegelondon.box.com/s/ict9tnswqk9z57ovx8r3ll5po5ccrib9 With best regards, - Ian & Krishna P.S. Daniel, thank you very much for the excellent example solution you provided in response to our question on obtaining the sharp discontinuity. Ian Campbell | PhD Candidate Electrochemical Science & Engineering Group Imperial College London, SW7 2AZ, United Kingdom ___ fipy mailing list fipy@nist.gov http://www.ctcms.nist.gov/fipy [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ]