On Wed, May 3, 2017 at 8:29 AM, Ned Batchelder <n...@nedbatchelder.com> wrote: > On Tuesday, May 2, 2017 at 11:49:37 PM UTC-4, larry....@gmail.com wrote: >> On Tue, May 2, 2017 at 7:01 PM, Erik <pyt...@lucidity.plus.com> wrote: >> > On 02/05/17 23:28, Larry Martell wrote: >> >>>> >> >>>> Anyone have any thoughts on how I can monitor the variables' memory >> >>>> usage as the script runs? >> >>> >> >>> >> >>> This is application-specific, but sometimes it helps to look at the >> >>> objects' types, or even their values. >> >> >> >> >> >> The types are dict and list, so they are not very useful, nor were the >> >> values. >> > >> > >> > Given your description, I would hazard a guess that you have a circular >> > reference somewhere (such that some objects may remain referenced after you >> > believe they should have been discarded). You could _probably_ break that >> > circle with some carefully placed 'del' statements somewhere - but you'd >> > need to analyze your data structures (and closures) first to see if that's >> > the case. >> >> This script is part of a django project, but it is not part of the >> core web django app. It's a stand alone script that is run from the >> command line. I added some profiling to the script and I found that >> the object count was increasing repidly with each loop (and of course >> with that the memory usage). I originally wrote it using the django >> ORM but now I have rewritten it using MySQLdb and embedded SQL. That >> stopped the increase in the object count and drastically reduced the >> memory leakage, but did not stop it. But at least now I can run it >> without it using all of the memory and getting killed. I'd still love >> to know where it's leaking memory, but I may not get the time for >> that. > > When you were using the Django ORM, did you have settings.DEBUG=True? > If so, Django stores the SQL queries for debugging.
Yes, I did have DEBUG=True, but I wasn't trying to see the SQL Django was generating. But looking at the objects I found that when the script entered its main loop there were around 28K objects, and each pass through the loop would add around 10K new objects. When I looked at their content most of them were lists of the column names of the table I was selecting from - many, many objects with the same values. When I switched to not using the ORM there were 28K objects throughout the entire run of the script (which was 100's of millions of iterations of its loop). -- https://mail.python.org/mailman/listinfo/python-list
