Re: segmentation fault in scipy?
[EMAIL PROTECTED] wrote: > I'm running operations large arrays of floats, approx 25,000 x 80. > Python (scipy) does not seem to come close to using 4GB of wired mem, > but segments at around a gig. Everything works fine on smaller batches > of data around 10,000 x 80 and uses a max of ~600mb of mem. Any Ideas? > Is this just too much data for scipy? > > Thanks Conor > > Traceback (most recent call last): > File "C:\Temp\CR_2\run.py", line 68, in ? >net.rProp(1.2, .5, .01, 50.0, input, output, 1) > File "/Users/conorrob/Desktop/CR_2/Network.py", line 230, in rProp >print scipy.trace(error*scipy.transpose(error)) > File "D:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line > 149, in > __mul__ >return N.dot(self, other) > MemoryError You should ask this question on the numpy-discussion list for better feedback. Does it actually segfault or give you this Memory Error? Temporary arrays that need to be created could be the source of the extra memory. Generally, you should be able to use all the memory on your system (unless you are on a 64-bit system and are not using Python 2.5). -Travis -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
If I run it from the shell (unix) I get: Segmentation fault and see a core dump in my processes. If I run it in the python shell I get as above: File "D:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line 149, in __mul__ return N.dot(self, other) MemoryError I your experience as one of the dev of scipy, is this too much data? thank you -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
[EMAIL PROTECTED] wrote: > I'm running operations large arrays of floats, approx 25,000 x 80. > Python (scipy) does not seem to come close to using 4GB of wired mem, > but segments at around a gig. Everything works fine on smaller batches > of data around 10,000 x 80 and uses a max of ~600mb of mem. Any Ideas? > Is this just too much data for scipy? > > Thanks Conor > > Traceback (most recent call last): > File "C:\Temp\CR_2\run.py", line 68, in ? >net.rProp(1.2, .5, .01, 50.0, input, output, 1) > File "/Users/conorrob/Desktop/CR_2/Network.py", line 230, in rProp >print scipy.trace(error*scipy.transpose(error)) > File "D:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line > 149, in > __mul__ >return N.dot(self, other) > MemoryError This is not a segfault. Is this the only error you see? Or are you actually seeing a segfault somewhere? If error.shape == (25000, 80), then dot(error, transpose(error)) will be returning an array of shape (25000, 25000). Assuming double precision floats, that array will take up about 4768 megabytes of memory, more than you have. The memory usage doesn't go up near 4 gigabytes because the allocation of the very large returned array fails, so the large chunk of memory never gets allocated. There are two possibilities: 1. As Travis mentioned, numpy won't create the array because it is still 32-bit-limited due to the Python 2.4 C API. This has been resolved with Python 2.5. 2. The default build of numpy uses plain-old malloc(3) to allocate memory, and it may be failing to create such large chunks of memory. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
Good point. Finding the SSE using an absolute error matrix of (25000 x 1) is insane. I pulled out the error function (for now) and I'm back in business. Thanks for all the great advise. -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
[EMAIL PROTECTED] wrote: > Good point. Finding the SSE using an absolute error matrix of (25000 x > 1) is insane. I pulled out the error function (for now) and I'm back > in business. Thanks for all the great advise. Could you go back for a second and describe your problem a little bit more. It sounds like you were doing the wrong operation. By SSE, you mean "Sum of Squared Errors" of the 25000 length-80 vectors, right? In that case, using matrix multiplication won't give you that. That will, in essence calculate the dot-product of each of the 25000 length-80 vectors with *each* of the other 25000 length-80 vectors in addition to themselves. It seems to me like you want something like this: SSE = sum(error * error, axis=-1) Then SSE.shape == (25000,). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
Im using rprop (not dependent on error function in this case ie. standard rprop vs. irprop or arprop) for an MLP tanh, sigmod nnet as part of a hybrid model. I guess I was using a little Matlab thought when I wrote the SSE funtion. My batches are about 25,000 x 80 so my absolute error (diff between net outputs and desired outputs) when using *one* output unit is shape(~25000,), am I wrong to assume trace(error*transpose(error)) is the sum of the squared errors which should be an shape(1,)? I'm just now starting to dig a little deeper into scipy, and I need to get the full doc. Thanks for all your input. -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
[EMAIL PROTECTED] wrote: > Im using rprop (not dependent on error function in this case ie. > standard rprop vs. irprop or arprop) for an MLP tanh, sigmod nnet as > part of a hybrid model. I guess I was using a little Matlab thought > when I wrote the SSE funtion. My batches are about 25,000 x 80 so my > absolute error (diff between net outputs and desired outputs) when > using *one* output unit is shape(~25000,), am I wrong to assume > trace(error*transpose(error)) is the sum of the squared errors which > should be an shape(1,)? I'm afraid you're using terminology (and abbreviations!) that I can't follow. Let me try to restate what's going on and you can correct me as I screw up. You have a neural net that has 80 output units. You have 25000 observations that you are using to train the neural net. Each observation vector (and consequently, each error vector) has 80 elements. Judging by your notation, you are using the matrix subclass of array to change * to matrix multiplication. In my message you are responding to (btw, please quote the emails you respond to so we can maintain some context), I gave an answer for you using regular arrays which have * as elementwise multiplication. The matrix object's behavior gets in the way of the most natural way to do these calculations, so I do recommend avoiding the matrix object and learning to use the dot() function to do matrix multiplication instead. But if you want to continue using matrix objects, then you can use the multiply() function to do element-wise multiplication. The answer I gave also used the wrong name for the result. It seems that you want the sum of the squared errors across all of the observations. In this case, you can use axis=None to specify that every element should be summed: SSE = sum(multiply(error, error), axis=None) trace(dot(error, transpose(error))) wastes a *huge* amount of time and memory since you are calculating (if your machine was capable of it) a gigantic matrix, then throwing away all of the off-diagonal elements. The method I give above wastes a little memory; there is one temporary matrix the size of error. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
> If I run it from the shell (unix) I get: Segmentation fault and see a > core dump in my processes. If I run it in the python shell I get as > above: > File "D:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line > 149, in That's a Window's path... Does Windows even make full use of 4GB? Im developing on a unix machine. However, you do have a sharp eye... So you win the blue ribbon and a free roundtable pizza! I'm actually running experiments on an HP workstation while I work (emailed myself the error), that's one of the beauties of python. Yes, Dennis there is a max of 4gb of addressable mem on that machine and 2 cpus. This is what a unix path looks like. Can you tell me what OS and shell Im running (both are obvious from the prompt) for the bonus prize? [gremlins:~/desktop/CR_2] conorrob% python run.py AL fold: 0 Done Loading Input Ready Network Training Initial SSE: Segmentation fault [gremlins:~/desktop/CR_2] conorrob% -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
>I'm afraid you're using terminology (and abbreviations!) that I can't follow. >Let me try to restate what's going on and you can correct me as I screw up. You >have a neural net that has 80 output units. You have 25000 observations that >you >are using to train the neural net. Each observation vector (and consequently, >each error vector) has 80 elements. First, sorry for not quoting and using abbreviations. Second, your obsevation is correct, except that I have *one* output unit, not 80. I have ~80 input units + bias (for each of the 25000 observations), x + bias number of hidden units and one output unit leaving me with an output array/matrix of shape =(25000,), as well as my desired output having the same shape. RPROP = Resilliant BackPropogation, uses chages in the error gradiant ignores the magnitude of the gradiant, which can be harmful. See "A Direct Adaptive Method for Faster Backpropogation Learning: The RPROP Algorithm" -Martin Riedmiller and Heinrich Braun, just google it and read Section D, it's well written. tanh = hyperbolic tangent function [-1, 1] y vals, often times better for its steeper derivative and wider range. sigmoid function = popular for its [0,1] range and can calculate posterior probabilities when using a the cross entropy error function which I have commented out since it takes more time to process and I'm not using my error function in this specific case at this point in time, thus SSE is not really needed, however I'd like use it as a check. Also, popular for its easily calculatable derivative. >The answer I gave also used the wrong name for the result. It seems that you >want the sum of the squared errors across all of the observations. In this >case, >you can use axis=None to specify that every element should be summed: > > SSE = sum(multiply(error, error), axis=None) > >trace(dot(error, transpose(error))) wastes a *huge* amount of time and memory >since you are calculating (if your machine was capable of it) a gigantic >matrix, >then throwing away all of the off-diagonal elements. The method I give above >wastes a little memory; there is one temporary matrix the size of error. This is great advise and much appreciated. It was the answer to my problem, thank you. However, isn't this faster... scipy.sum(scipy.array(scipy.matrix(error)*scipy.matrix(error)), axis = None) as we you explained in my other posting? Thanks again. -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
[EMAIL PROTECTED] wrote: > This is great advise and much appreciated. It was the answer to my > problem, thank you. However, isn't this faster... > scipy.sum(scipy.array(scipy.matrix(error)*scipy.matrix(error)), axis = > None) > as we you explained in my other posting? No! matrix objects use matrix multiplication for *. You seem to need elementwise multiplication. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
>No! matrix objects use matrix multiplication for *. You seem to need >elementwise >multiplication. No! when you mult a vector with itself transposed, the diagonal of the resulting matrix is the squares of each error (albeit you do a lot of extra calc), then sum the squares, ie trace(). Its a nifty trick, if you don't have too much data 25000x25000 matrix in mem and youre using matricies ie. batch learning. The actual equation includes multiply by 1/2*(sum of the squares), but mean squared error can be more telling about error and cross entropy is even better, becuase it tells you how well youre predicting the posterior probabilies... -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
py_genetic wrote: >>No! matrix objects use matrix multiplication for *. You seem to need >>elementwise >>multiplication. > > No! when you mult a vector with itself transposed, the diagonal of the > resulting matrix is the squares of each error (albeit you do a lot of > extra calc), then sum the squares, ie trace(). Its a nifty trick, if > you don't have too much data 25000x25000 matrix in mem and youre using > matricies ie. batch learning. The actual equation includes multiply by > 1/2*(sum of the squares), but mean squared error can be more telling > about error and cross entropy is even better, becuase it tells you how > well youre predicting the posterior probabilies... But, again, why bother calculating an enormous matrix just to take its trace when you can use elementwise multiplication and not waste any time or memory calculating values that you don't need? It's not a trick. It's just wrong. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
py_genetic wrote: >>No! matrix objects use matrix multiplication for *. You seem to need >>elementwise >>multiplication. > > No! when you mult a vector with itself transposed, the diagonal of the > resulting matrix is the squares of each error (albeit you do a lot of > extra calc), then sum the squares, ie trace(). Its a nifty trick, if > you don't have too much data 25000x25000 matrix in mem and youre using > matricies ie. batch learning. The actual equation includes multiply by > 1/2*(sum of the squares), but mean squared error can be more telling > about error and cross entropy is even better, becuase it tells you how > well youre predicting the posterior probabilies... Now I'm even more confused. What kind of array is "error" here? First you tell me it's a (25000, 80) array and now you are telling me it is a (25000,) array. Once you've defined what "error" is, then please tell me what the quantity is that you want to calculate. I think I told you several different wrong things, previously, based on wrong assumptions. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
True! it is rediculous/insane as I mentioned and noted and agreed with you (in all your responses) and was my problem, however, not wrong (same result), as I was just simply noting (not to be right), although, yes, insane. Thanks again. -- http://mail.python.org/mailman/listinfo/python-list
Re: segmentation fault in scipy?
>Now I'm even more confused. What kind of array is "error" here? First you tell >me it's a (25000, 80) array and now you are telling me it is a (25000,) array. >Once you've defined what "error" is, then please tell me what the quantity is >that you want to calculate. I think I told you several different wrong things, >previously, based on wrong assumptions. > It's just in my original post I was trying to get across maximum size of the arrays I'm using. sorry for the confusion, I didn't state actual size of my output vectors. I discovered the probelm when your first stated: >If error.shape == (25000, 80), then dot(error, transpose(error)) will be >returning an array of shape (25000, 25000) Which was exactly related to the excessive calculation I was running and set off the red flags and made it very clear. Later I was somewhat confused and believed that you we were talking about two different things regarding SSE when you said: SSE = sum(multiply(error, error), axis=None) And didn't realize that multipy() was an efficient element mult method for matricies, thinking it was like matrixmultiply() and gave you back the old trace sse method but with the casting (not meaning to contradict you). However all is now very clear, and I agree with element wise mult or even squaring the output error, and have no real reason why I was using trace, except I had a faint memory of using it in a class for small experiments in matlab (i guess the idea was to keep everything in linear algebra) and I spit it up for some reason when I was doing a quicky error function. -- http://mail.python.org/mailman/listinfo/python-list
