On 8/26/06, Torgil Svensson <[EMAIL PROTECTED]> wrote:
Why not use convolve for the std also? You can't subtract the average first, but you could convolve the square of the vector and then use some variant of std = sqrt((convsqrs - n*avg**2)/(n-1)). There are possible precision problems but they may not matter for your application, especially if the moving window isn't really big.
Chuck
Hi
ndarray.std(axis=1) seems to have memory issues on large 2D-arrays. I
first thought I had a performance issue but discovered that std() used
lots of memory and therefore caused lots of swapping.
I want to get an array where element i is the stadard deviation of row
i in the 2D array. Using valgrind on the std() function...
$ valgrind --tool=massif python -c "from numpy import *;
a=reshape(arange(100000*100),(100000,100)).std(axis=1)"
... showed me a peak of 200Mb memory while iterating line by line...
$ valgrind --tool=massif python -c "from numpy import *;
a=array([x.std() for x in reshape(arange(100000*100),(100000,100))])"
... got a peak of 40Mb memory.
This seems unnecessary since we know before calculations what the
output shape will be and should therefore be able to preallocate
memory.
My original problem was to get an moving average and a moving standard
deviation (120k rows and N=1000). For average I guess convolve should
perform good, but is there anything smart for std()? For now I use ...
Why not use convolve for the std also? You can't subtract the average first, but you could convolve the square of the vector and then use some variant of std = sqrt((convsqrs - n*avg**2)/(n-1)). There are possible precision problems but they may not matter for your application, especially if the moving window isn't really big.
Chuck
------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
