I had a massive memory leak in some of my code. It would basically end up using up all 1GB of my RAM or more if I don't kill the application. I managed to finally figure out which portion of the code was causing the leak (with great difficulty) and have a little example which exposes the leak. I am using numpy-0.9.8 and I'm wondering if perhaps this is already fixed in 1.0b2. Run this through valgrind with appropriate options (I used the recommended valgrind_py.sh that I found on scipy's site somewhere) and this will leak 100kB. Increase the xrange on the big loop and you can watch the memory increase over time in top.
The interesting thing is that the only difference between the leaky and non-leaky code is:
if not adjacencyMatrix[anInt2,anInt1] == 0: (leaky)
vs.
if not adjacencyMatrix[anInt2][anInt1] == 0: (non-leaky)
however another way to make the leaky code non-leaky is to change anArrayOfInts to just be [1]
Here's the code:
from numpy import array
def leakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix):
ys = set()
for aList in aListOfArrays:
for anInt1 in anArrayOfInts:
for anInt2 in aList:
if not adjacencyMatrix[anInt2,anInt1] == 0:
ys.add (anInt1)
return ys
def nonLeakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix):
ys = set()
for aList in aListOfArrays:
for anInt1 in anArrayOfInts:
for anInt2 in aList:
if not adjacencyMatrix[anInt2][anInt1] == 0:
ys.add(anInt1)
return ys
if __name__ == "__main__":
for i in xrange(10000):
aListOfArrays = [[0, 1]]
anArrayOfInts = array([1])
adjacencyMatrix = array([[0,1],[1,0]])
#COMMENT OUT ONE OF THE 2 LINES BELOW
#bar = nonLeakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix)
bar = leakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix)
--
David Grant
http://www.davidgrant.ca
------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
