Steven D'Aprano wrote: > You've described an extremely artificial set of circumstances: you create > 40,000,000 distinct integers, then immediately destroy them. The obvious > solution to that "problem" of Python caching millions of integers you > don't need is not to create them in the first place.
I know it's a very artificial setup - I was trying to make the situation simple to demonstrate in a few lines. The point was that it's not caching the values of those integers, as they can never be read again through the Python interface. It's just holding onto the space they occupy in case it's needed again. > So what's your actual problem that you are trying to solve? I have a program which reads a few thousand text files, converts each to a list (with readlines()), creates a short summary of the contents of each (a few floating point numbers) and stores this summary in a master list. From the amount of memory it's using, I think that the lists containing the contents of each file are kept in memory, even after there are no references to them. Also, if I tell it to discard the master list and re-read all the files, the memory use nearly doubles so I presume it's keeping the lot in memory. The program may run through several collections of files, but it only keeps a reference to the master list of the most recent collection it's looked at. Obviously, it's not ideal if all the old collections hang around too, taking up space and causing the machine to swap. >> but is there anything I can do to get that memory back without closing >> Python? > > Why do you want to manage memory yourself anyway? It seems like a > horrible, horrible waste to use a language designed to manage memory for > you, then insist on over-riding it's memory management. I agree. I don't want to manage it myself. I just want it to re-use memory or hand it back to the OS if it's got an awful lot that it's not using. Wouldn't you say it was wasteful if (say) an image editor kept an uncompressed copy of an image around in memory after the image had been closed? -- I'm at CAMbridge, not SPAMbridge -- http://mail.python.org/mailman/listinfo/python-list