Re: [Numpy-discussion] Interesting memory leak
> This leak is caused by add_docstring, but it's supposed to leak. I wonder if > there's a way to register some kind of on-exit handler in Python so that > this can also be cleaned up? import atexit atexit.register(your_cleanup_function) whose api is no args on input: def your_cleanup_function(): do_whatever... You could use here a little extension function which goes in and does the necessary free() calls on a pre-stored list of allocated pointers, if there's more than one (I don't really know what's going on here). Cheers, f - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Interesting memory leak
Hello all > > I decided to upgrade to 1.0b2 just to see what I get and now I get 7kB of > "possibly lost" memory, coming from PyObject_Malloc (in > /usr/lib/libpython2.4.so.1.0). This is a constant 7kB, however, and it > isn't getting any larger if I increase the loop iterations. Looks good > then. I don't really know the meaning of this "possibly lost" memory. http://projects.scipy.org/scipy/numpy/ticket/195 This leak is caused by add_docstring, but it's supposed to leak. I wonder if there's a way to register some kind of on-exit handler in Python so that this can also be cleaned up? Cheers, Albert - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Interesting memory leak
On 8/17/06, Robert Kern <[EMAIL PROTECTED]> wrote: David Grant wrote:> Hello all,>> I had a massive memory leak in some of my code. It would basically end> up using up all 1GB of my RAM or more if I don't kill the application. I> managed to finally figure out which portion of the code was causing the > leak (with great difficulty) and have a little example which exposes the> leak. I am using numpy-0.9.8 and I'm wondering if perhaps this is> already fixed in 1.0b2. Run this through valgrind with appropriate > options (I used the recommended valgrind_py.sh that I found on scipy's> site somewhere) and this will leak 100kB. Increase the xrange on the big> loop and you can watch the memory increase over time in top. I don't see a leak in 1.0b2.dev3002.Thanks Robert.I decided to upgrade to 1.0b2 just to see what I get and now I get 7kB of "possibly lost" memory, coming from PyObject_Malloc (in /usr/lib/libpython2.4.so.1.0). This is a constant 7kB, however, and it isn't getting any larger if I increase the loop iterations. Looks good then. I don't really know the meaning of this "possibly lost" memory. -- David Granthttp://www.davidgrant.ca - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Interesting memory leak
David Grant wrote: > Hello all, > > I had a massive memory leak in some of my code. It would basically end > up using up all 1GB of my RAM or more if I don't kill the application. I > managed to finally figure out which portion of the code was causing the > leak (with great difficulty) and have a little example which exposes the > leak. I am using numpy-0.9.8 and I'm wondering if perhaps this is > already fixed in 1.0b2. Run this through valgrind with appropriate > options (I used the recommended valgrind_py.sh that I found on scipy's > site somewhere) and this will leak 100kB. Increase the xrange on the big > loop and you can watch the memory increase over time in top. I don't see a leak in 1.0b2.dev3002. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] Interesting memory leak
Hello all,I had a massive memory leak in some of my code. It would basically end up using up all 1GB of my RAM or more if I don't kill the application. I managed to finally figure out which portion of the code was causing the leak (with great difficulty) and have a little example which exposes the leak. I am using numpy-0.9.8 and I'm wondering if perhaps this is already fixed in 1.0b2. Run this through valgrind with appropriate options (I used the recommended valgrind_py.sh that I found on scipy's site somewhere) and this will leak 100kB. Increase the xrange on the big loop and you can watch the memory increase over time in top. The interesting thing is that the only difference between the leaky and non-leaky code is:if not adjacencyMatrix[anInt2,anInt1] == 0: (leaky)vs.if not adjacencyMatrix[anInt2][anInt1] == 0: (non-leaky) however another way to make the leaky code non-leaky is to change anArrayOfInts to just be [1]Here's the code:from numpy import arraydef leakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix): ys = set() for aList in aListOfArrays: for anInt1 in anArrayOfInts: for anInt2 in aList: if not adjacencyMatrix[anInt2,anInt1] == 0: ys.add (anInt1) return ysdef nonLeakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix): ys = set() for aList in aListOfArrays: for anInt1 in anArrayOfInts: for anInt2 in aList: if not adjacencyMatrix[anInt2][anInt1] == 0: ys.add(anInt1) return ysif __name__ == "__main__": for i in xrange(1): aListOfArrays = [[0, 1]] anArrayOfInts = array([1]) adjacencyMatrix = array([[0,1],[1,0]])#COMMENT OUT ONE OF THE 2 LINES BELOW #bar = nonLeakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix) bar = leakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix) -- David Granthttp://www.davidgrant.ca - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
