Re: Memory error with quadratic interpolation
On 23 January 2013 17:33, Isaac Won wrote: > On Wednesday, January 23, 2013 10:51:43 AM UTC-6, Oscar Benjamin wrote: >> On 23 January 2013 14:57, Isaac Won wrote: >> >> > On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote: >> >> Unless I've misunderstood how this function is supposed to be used, it >> just doesn't really seem to work for arrays of much more than a few >> hundred elements. >> The solution is to use UnivariateSpline. I don't know what the difference is but it works where the other fails: import numpy as np from scipy.interpolate import UnivariateSpline x = np.array(1 * [0.0], float) indices = np.arange(len(x)) interp = UnivariateSpline(indices, x, k=2) print(interp(1.5)) Oscar -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory error with quadratic interpolation
On Wednesday, January 23, 2013 10:51:43 AM UTC-6, Oscar Benjamin wrote:
> On 23 January 2013 14:57, Isaac Won wrote:
>
> > On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote:
>
> >> On 23 January 2013 14:28, Isaac Won wrote:
>
> >>
>
> [SNIP]
>
> >
>
> > Following is full error message after I adjusted following Ulich's advice:
>
> >
>
> > interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
>
> > File
> > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py",
> > line 311, in __init__
>
> > self._spline = splmake(x,oriented_y,order=order)
>
> > File
> > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py",
> > line 809, in splmake
>
> > coefs = func(xk, yk, order, conds, B)
>
> > File
> > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py",
> > line 530, in _find_smoothest
>
> > u,s,vh = np.dual.svd(B)
>
> > File
> > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py",
> > line 91, in svd
>
> > full_matrices=full_matrices, overwrite_a = overwrite_a)
>
> > MemoryError
>
>
>
> Where is the new code? You should show full working code (with the
>
> import statements) and the full error that is generated by exactly
>
> that code. If possible you should also write code that someone else
>
> could run even without having access to your data files. If you did
>
> that in your first post, you'd probably have an answer to your problem
>
> by now.
>
>
>
> Here is a version of your code that many people on this list can test
>
> straight away:
>
>
>
> import numpy as np
>
> from scipy.interpolate import interp1d
>
> x = np.array(31747 * [0.0], float)
>
> indices = np.arange(len(x))
>
> interp = interp1d(indices, x, kind='quadratic')
>
>
>
> Running this gives the following error:
>
>
>
> ~$ python tmp.py
>
> Traceback (most recent call last):
>
> File "tmp.py", line 5, in
>
> interp = interp1d(indices, x, kind='quadratic')
>
> File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
>
> line 308, in __init__
>
> self._spline = splmake(x,oriented_y,order=order)
>
> File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
>
> line 805, in splmake
>
> B = _fitpack._bsplmat(order, xk)
>
> MemoryError
>
>
>
> Unless I've misunderstood how this function is supposed to be used, it
>
> just doesn't really seem to work for arrays of much more than a few
>
> hundred elements.
>
>
>
>
>
> Oscar
Thank you Oscar for your help and advice.
I agree with you. So, I tried to find the way to solve this problem.
My full code adjusted is:
from scipy.interpolate import interp1d
import numpy as np
import matplotlib.pyplot as plt
with open(filin, "r") as f:
for line in f:
columns = line.strip().split()
a.append(columns[5])
x = np.array(a, float)
not_nan = np.logical_not(np.isnan(x))
indices = np.arange(len(x))
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
p = interp(indices)
k = np.arange(31747)
plt.subplot(211)
plt.plot(k, p)
plt.xlabel('Quadratic interpolation')
plt.subplot(212)
plt.plot(k, x)
plt.show()
-
Whole error message was:
Traceback (most recent call last):
File "QI1.py", line 22, in
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
File
"/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py",
line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File
"/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py",
line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File
"/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py",
line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File
"/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py",
line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError
--
Thank you again Oscar,
Isaac
--
http://mail.python.org/mailman/listinfo/python-list
Re: Memory error with quadratic interpolation
On 23 January 2013 14:57, Isaac Won wrote: > On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote: >> On 23 January 2013 14:28, Isaac Won wrote: >> [SNIP] > > Following is full error message after I adjusted following Ulich's advice: > > interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic') > File > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", > line 311, in __init__ > self._spline = splmake(x,oriented_y,order=order) > File > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", > line 809, in splmake > coefs = func(xk, yk, order, conds, B) > File > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", > line 530, in _find_smoothest > u,s,vh = np.dual.svd(B) > File > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", > line 91, in svd > full_matrices=full_matrices, overwrite_a = overwrite_a) > MemoryError Where is the new code? You should show full working code (with the import statements) and the full error that is generated by exactly that code. If possible you should also write code that someone else could run even without having access to your data files. If you did that in your first post, you'd probably have an answer to your problem by now. Here is a version of your code that many people on this list can test straight away: import numpy as np from scipy.interpolate import interp1d x = np.array(31747 * [0.0], float) indices = np.arange(len(x)) interp = interp1d(indices, x, kind='quadratic') Running this gives the following error: ~$ python tmp.py Traceback (most recent call last): File "tmp.py", line 5, in interp = interp1d(indices, x, kind='quadratic') File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py", line 308, in __init__ self._spline = splmake(x,oriented_y,order=order) File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py", line 805, in splmake B = _fitpack._bsplmat(order, xk) MemoryError Unless I've misunderstood how this function is supposed to be used, it just doesn't really seem to work for arrays of much more than a few hundred elements. Oscar -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory error with quadratic interpolation
On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote: > On 23 January 2013 14:28, Isaac Won wrote: > > > On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote: > > > > > > To Oscar > > > My actual error message is: > > > File > > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", > > line 311, in __init__ > > > self._spline = splmake(x,oriented_y,order=order) > > > File > > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", > > line 809, in splmake > > > coefs = func(xk, yk, order, conds, B) > > > File > > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", > > line 530, in _find_smoothest > > > u,s,vh = np.dual.svd(B) > > > File > > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", > > line 91, in svd > > > full_matrices=full_matrices, overwrite_a = overwrite_a) > > > MemoryError > > > > Are you sure that's the *whole* error message? The traceback only > > refers to the scipy modules. I can't see the line from your code that > > is generating the error. > > > > > > Oscar Dear Oscar, Following is full error message after I adjusted following Ulich's advice: interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic') File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__ self._spline = splmake(x,oriented_y,order=order) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake coefs = func(xk, yk, order, conds, B) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest u,s,vh = np.dual.svd(B) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd full_matrices=full_matrices, overwrite_a = overwrite_a) MemoryError -- Thank you, Hoonill -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory error with quadratic interpolation
On Wednesday, January 23, 2013 2:55:14 AM UTC-6, Ulrich Eckhardt wrote: > Am 23.01.2013 05:06, schrieb Isaac Won: > > > I have tried to use different interpolation methods with Scipy. My > > > code seems just fine with linear interpolation, but shows memory > > > error with quadratic. I am a novice for python. I will appreciate any > > > help. > > > > > > #code > > > f = open(filin, "r") > > > > Check out the "with open(...) as f" syntax. > > > > > > > for columns in ( raw.strip().split() for raw in f ): > > > > For the record, this first builds a sequence and then iterates over that > > sequence. This is not very memory-efficient, try this instead: > > > > for line in f: > > columns = line.strip().split() > > > > > > Concerning the rest of your problems, there is lots of code and the > > datafile missing. However, there is also too much of it, try replacing > > the file with generated data and remove everything from the code that is > > not absolutely necessary. > > > > Good luck! > > > > Uli Hi Ulich, I tried to change the code following your advice, but it doesn't seem to work still. My adjusted code is: a = [] with open(filin, "r") as f: for line in f: columns = line.strip().split() a.append(columns[5]) x = np.array(a, float) not_nan = np.logical_not(np.isnan(x)) indices = np.arange(len(x)) interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic') p = interp(indices) - And full error message is: interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic') File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__ self._spline = splmake(x,oriented_y,order=order) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake coefs = func(xk, yk, order, conds, B) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest u,s,vh = np.dual.svd(B) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd full_matrices=full_matrices, overwrite_a = overwrite_a) MemoryError --- Could you give me some advice for this situation? Thank you always, Isaac -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory error with quadratic interpolation
On 23 January 2013 14:28, Isaac Won wrote: > On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote: > > To Oscar > My actual error message is: > File > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", > line 311, in __init__ > self._spline = splmake(x,oriented_y,order=order) > File > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", > line 809, in splmake > coefs = func(xk, yk, order, conds, B) > File > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", > line 530, in _find_smoothest > u,s,vh = np.dual.svd(B) > File > "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", > line 91, in svd > full_matrices=full_matrices, overwrite_a = overwrite_a) > MemoryError Are you sure that's the *whole* error message? The traceback only refers to the scipy modules. I can't see the line from your code that is generating the error. Oscar -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory error with quadratic interpolation
On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote: > On 23 January 2013 08:55, Ulrich Eckhardt > > > > > Am 23.01.2013 05:06, schrieb Isaac Won: > > > > > >> I have tried to use different interpolation methods with Scipy. My > > >> code seems just fine with linear interpolation, but shows memory > > >> error with quadratic. I am a novice for python. I will appreciate any > > >> help. > > > > > [SNIP] > > > > > > > > > Concerning the rest of your problems, there is lots of code and the datafile > > > missing. However, there is also too much of it, try replacing the file with > > > generated data and remove everything from the code that is not absolutely > > > necessary. > > > > Also please copy paste the actual error message rather than paraphrasing it. > > > > > > Oscar I really appreciate to both Ulich and Oscar. To Oscar My actual error message is: File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__ self._spline = splmake(x,oriented_y,order=order) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake coefs = func(xk, yk, order, conds, B) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest u,s,vh = np.dual.svd(B) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd full_matrices=full_matrices, overwrite_a = overwrite_a) MemoryError -- Thank you, Hoonill -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory error with quadratic interpolation
On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote: > On 23 January 2013 08:55, Ulrich Eckhardt > > wrote: > > > Am 23.01.2013 05:06, schrieb Isaac Won: > > > > > >> I have tried to use different interpolation methods with Scipy. My > > >> code seems just fine with linear interpolation, but shows memory > > >> error with quadratic. I am a novice for python. I will appreciate any > > >> help. > > > > > [SNIP] > > > > > > > > > Concerning the rest of your problems, there is lots of code and the datafile > > > missing. However, there is also too much of it, try replacing the file with > > > generated data and remove everything from the code that is not absolutely > > > necessary. > > > > Also please copy paste the actual error message rather than paraphrasing it. > > > > > > Oscar I really appreciate to both Ulich and Oscar. To Oscar My actual error message is: File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__ self._spline = splmake(x,oriented_y,order=order) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake coefs = func(xk, yk, order, conds, B) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest u,s,vh = np.dual.svd(B) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd full_matrices=full_matrices, overwrite_a = overwrite_a) MemoryError -- Thank you, Isaac -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory error with quadratic interpolation
On Tuesday, January 22, 2013 10:06:41 PM UTC-6, Isaac Won wrote: > Hi all, > > > > I have tried to use different interpolation methods with Scipy. My code seems > just fine with linear interpolation, but shows memory error with quadratic. I > am a novice for python. I will appreciate any help. > > > > #code > > f = open(filin, "r") > > for columns in ( raw.strip().split() for raw in f ): > > a.append(columns[5]) > > x = np.array(a, float) > > > > > > not_nan = np.logical_not(np.isnan(x)) > > indices = np.arange(len(x)) > > interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic') > > > > p = interp(indices) > > > > > > The number of data is 31747. > > > > Thank you, > > > > Isaac I really appreciate to both Ulich and Oscar. To Oscar My actual error message is: File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__ self._spline = splmake(x,oriented_y,order=order) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake coefs = func(xk, yk, order, conds, B) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest u,s,vh = np.dual.svd(B) File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd full_matrices=full_matrices, overwrite_a = overwrite_a) MemoryError -- Thank you, Hoonill -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory error with quadratic interpolation
On 23 January 2013 08:55, Ulrich Eckhardt wrote: > Am 23.01.2013 05:06, schrieb Isaac Won: > >> I have tried to use different interpolation methods with Scipy. My >> code seems just fine with linear interpolation, but shows memory >> error with quadratic. I am a novice for python. I will appreciate any >> help. > [SNIP] > > > Concerning the rest of your problems, there is lots of code and the datafile > missing. However, there is also too much of it, try replacing the file with > generated data and remove everything from the code that is not absolutely > necessary. Also please copy paste the actual error message rather than paraphrasing it. Oscar -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory error with quadratic interpolation
Am 23.01.2013 05:06, schrieb Isaac Won: I have tried to use different interpolation methods with Scipy. My code seems just fine with linear interpolation, but shows memory error with quadratic. I am a novice for python. I will appreciate any help. > #code f = open(filin, "r") Check out the "with open(...) as f" syntax. for columns in ( raw.strip().split() for raw in f ): For the record, this first builds a sequence and then iterates over that sequence. This is not very memory-efficient, try this instead: for line in f: columns = line.strip().split() Concerning the rest of your problems, there is lots of code and the datafile missing. However, there is also too much of it, try replacing the file with generated data and remove everything from the code that is not absolutely necessary. Good luck! Uli -- http://mail.python.org/mailman/listinfo/python-list
Memory error with quadratic interpolation
Hi all, I have tried to use different interpolation methods with Scipy. My code seems just fine with linear interpolation, but shows memory error with quadratic. I am a novice for python. I will appreciate any help. #code f = open(filin, "r") for columns in ( raw.strip().split() for raw in f ): a.append(columns[5]) x = np.array(a, float) not_nan = np.logical_not(np.isnan(x)) indices = np.arange(len(x)) interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic') p = interp(indices) The number of data is 31747. Thank you, Isaac -- http://mail.python.org/mailman/listinfo/python-list
