Torgil Svensson wrote: > Hi > > ndarray.std(axis=1) seems to have memory issues on large 2D-arrays. I > first thought I had a performance issue but discovered that std() used > lots of memory and therefore caused lots of swapping. > There are certainly lots of intermediate arrays created as the calculation proceeds. The calculation is not particularly "smart." It just does the basic averaging and multiplication needed.
> I want to get an array where element i is the stadard deviation of row > i in the 2D array. Using valgrind on the std() function... > > $ valgrind --tool=massif python -c "from numpy import *; > a=reshape(arange(100000*100),(100000,100)).std(axis=1)" > > ... showed me a peak of 200Mb memory while iterating line by line... > > The C-code is basically a directy "translation" of the original Python code. There are lots of temporaries created (apparently 5 at one point :-). I did this before I had the _internal.py code in place where I place Python functions that need to be accessed from C. If I had to do it over again, I would place the std implementation there where it could be appropriately optimized. -Travis ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
