My apologies in advance if this has been addressed before. Google does not presently seem to return search results for this group from more than a couple of months ago.
I have some long-running Python processes that slowly increase in resident memory size, and whose resident size goes down only when they are restarted. I spent hours with gc and heapy but was unable to identify obvious culprits. I eventually tracked the problem down to buffering data to a queue for later processing. Putting to the queue increases resident size, but getting from it never decreases resident size. In fact I see the same behavior when I use plain lists instead of queue objects. I thought Evan Jones altered Python to deal with this very problem, and the change went into the release of 2.5. Here is Tim Peters announcing the change: http://mail.python.org/pipermail/python-dev/2006-March/061991.html He included this simple test program to show the improvement: """ x = [] for i in xrange(1000000): x.append([]) raw_input("full ") del x[:] raw_input("empty ") """ If you look at resident size in the "full" stage, the interpreter has grown to tens of megabytes. If you look at it in the "empty" stage, it goes back down to less than 10 megabytes. But if you run this trivial variation on the same program, memory use goes up and stays up: """ x = [] for i in xrange(1000000): x.append([]) raw_input("full ") del x[:] for i in xrange(1000000): x.append([]) del x[:] raw_input("empty ") """ At the "empty" prompt resident memory size has not decreased. I see this pattern of behavior in CPython 3.1.1, 2.6.3, 2.5.2, and Jython 2.5.1. I have tested under 32 and 64 bit Intel Linux. At this point I suspect that I am not going to be able to force my long-running processes to shrink their resident size, since I can't force it in much simpler tests. I am curious about why it happens though. That the second program should retain a larger resident memory footprint than the first is (to me) quite surprising. -- http://mail.python.org/mailman/listinfo/python-list