Re: [Numpy-discussion] [SciPy-user] maskedarray
On Wed, 12 Dec 2007, "Travis E. Oliphant" apparently wrote:
> 2) The matrix object made a C-subclass (for speed).
This will probably be the last chance for such a change,
so I again hope that consideration will be given to *one*
change in the matrix object:
iteration over a matrix should return arrays
(instead of matrices).
So if A is a matrix,
A[1] should be an array,
but A[1,:] should be a matrix.
Obviously this is an argument from design rather
than from functionality. Current behavior is not "natural".
E.g., it makes it awkward to iterate over all elements in
the "natural" way, which I claim is::
for row in A:
for element in row:
print element
This example is just meant to illustrate in a simple way
what is "odd" about the current behavior. (It is not meant
to be an "argument" nor to suggest that the current absence
simple ways to do the same thing---e.g., using A.A.)
Whenever I am working with matrices, I notice some aspect of
this "oddity", and it is annoying when so much else is quite
aesthetic.
Cheers,
Alan Isaac
PS For those who do not know, here is an example of the
current behavior. (The following prints 2 matrices instead
of 4 elements.)
>>> A = N.mat('1 2;3 4')
>>> for row in A:
... for element in row:
... print element
...
[[1 2]]
[[3 4]]
>>>
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-user] maskedarray
Pierre GM wrote: > On Tuesday 11 December 2007 15:48:01 Tom Johnson wrote: > >> What is the status regarding 'maskedarray'? When will this become >> part (replace ma) of the standard distribution? Also, what is the >> recommended way to import sandbox modules so that code changes are >> minimized when (if) the package becomes part of the standard >> distribution. >> > > > Mmh, no answer from our leaders ? Well, for now maskedarray lives in the > scipy > sandbox, but it probably could be transferred to the numpy sandbox. There's > still no particular decision I'm aware of about putting it the official > release. More feedback is probably needed from users. > I apologize for not chiming in here. It has been difficult to keep up with all the lists on which I should ostensibly know what is being discussed as I have changed jobs and am settling in to new responsibilites. It seems to me that it is time to move the scipy.sandbox.maskedarray implementation over to numpy.ma. I'm a little concerned about doing this before 1.1, but if the API has not changed, then it should be fine. If there is no opposition, then it seems that this can move forward before 1.0.5 which I would like to see in January. For the future 1.1 release (late 2008), I would like to see (in no particular priority order): 1) The MA made a C-subclass 2) The matrix object made a C-subclass (for speed). 3) The sparse matrix structure (but not all the algorithms) brought over into NumPy with the indexing matching the matrix indexing. 4) Weave brought over into NumPy The additions should be created in such a way that they can be disabled should so that perhaps a numpy-lite could be produced as well. Or, perhaps better separate packages like numpy-devel, numpy-compat, numpy-sparse could be created. I'm looking forward to some good discussions over the weekend. -Travis O. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] linalg.lapack_lite has undefined symbols
Hello, I am converting my build scripts to use the Intel Compiler Suite, using today's svn code, on RHEL 4.0. Building numpy succeeds, but importing it fails: ImportError: /usr/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: dgesdd_ And sure enough: [EMAIL PROTECTED] Diamondback]$ ldd /usr/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so libpython2.5.so.1.0 => /usr/local/lib/libpython2.5.so.1.0 (0x002a9592a000) libimf.so => /usr/local/stow/intel_cce-10.1.008/lib/libimf.so (0x002a95c4e000) libsvml.so => /usr/local/stow/intel_cce-10.1.008/lib/libsvml.so (0x002a95fb) libm.so.6 => /lib64/tls/libm.so.6 (0x002a96138000) libintlc.so.5 => /usr/local/stow/intel_cce-10.1.008/lib/libintlc.so.5 (0x002a962be000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x002a963f7000) libc.so.6 => /lib64/tls/libc.so.6 (0x002a96505000) libdl.so.2 => /lib64/libdl.so.2 (0x002a9673a000) libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x002a9683d000) libutil.so.1 => /lib64/libutil.so.1 (0x002a96953000) /lib64/ld-linux-x86-64.so.2 (0x00552000) [EMAIL PROTECTED] Diamondback]$ nm /usr/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so | grep " U " | head U dgeev_ U dgelsd_ U dgeqrf_ U dgesdd_ U dorgqr_ U dsyevd_ U exit@@GLIBC_2.2.5 U floor U for_stop_core U for_write_seq_fmt I'm using ATLAS 3.8.0 also compiled with ICS, but the undefined symbol is in lapack_lite.so . Any suggestions on where to go from here? Thank you, Glen Mabey ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy large arrays?
On Dec 12, 2007 7:29 AM, Søren Dyrsting <[EMAIL PROTECTED]> wrote:
> Hi all
>
> I need to perform computations involving large arrays. A lot of rows and
> no more than e.g. 34 columns. My first choice is python/numpy because I'm
> already used to code in matlab.
>
> However I'm experiencing memory problems even though there is still 500 MB
> available (2 GB total). I have cooked down my code to following meaningless
> code snip. This code share some of the same structure and calls as my real
> program and shows the same behaviour.
>
>
> import numpy as N
> import scipy as S
>
> def stress():
> x = S.randn(20,80)
> for i in range(8):
> print "%(0)d" % {"0": i}
> s = N.dot(x.T, x)
> sd = N.array([s.diagonal()])
> r = N.dot(N.ones((N.size(x,0),1),'d'), sd)
> x = x + r
> x = x / 1.01
>
>
>
>
> To different symptoms depending how big x are:
> 1) the program becomes extremely slow after a few iterations.
This appears to be because you are overflowing your floating point
variables. Once your data has INFs in it, it will tend to run much slower.
>
> 2) if the size of x is increased a little the program fails with the
> message "MemoryError" for example at line 'x = x + r', but different places
> in the code depending on the matrice size and which computer I'm testing.
> This might also occur after several iterations, not just during the first
> pass.
Why it would occur after several iterations I'm not sure. It's possible that
there are some cycles that it takes a while for the garbage collector to get
to and in the meantime you are chewing through all of your memory. Their are
a couple different things you could try to address that, but before you do
that, you need to clean up your algorithm and right it in idiomatic numpy. I
realize that you said the above code is meaningless, but I'm going to assume
that it's indicative of how your numpy code is written. That can be
rewritten as:
def stress2(x):
for i in range(8):
print i
x += (x**2).sum(axis=0)
x /= 1.01
return x.sum()
Not only is the above about sixty times faster, it's considerably clearer as
well. FWIW, on my box, which has a very similar setup to yours, neither
version throws a memory error.
>
> I'm using Windows XP, ActivePython 2.5.1.1, NumPy 1.0.4, SciPy 0.6.0.
>
> - Is there an error under the hood in NumPy?
Probably not in this case.
>
> - Am I balancing on the edge of the performance of Python/NumPy and should
> consider other environments. Fortran, C, BLAS, LAPACK e.t.c.
Maybe, but try cleaning things up first.
>
> - Am I misusing NumPy? Changing coding style will be a good workaround and
> even perform on larger datasets without errors?
Your code is doing a lot of extra work and creating a lot of temporaries.
I'd clean it up before I did anything else.
>
>
> Thanks in advance
> /Søren
>
>
> ___
> Numpy-discussion mailing list
> Numpy-discussion@scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
--
. __
. |-\
.
. [EMAIL PROTECTED]
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] resize in old Numeric
Christian Meesters wrote: > Hi, > > For compatibility reasons (work with Biopython) I would like to use to > Numeric in some code of mine. (Of course, I could make a little detour > converting into numpy.array, but first I wonder whether somebody might > know a solution for the problem with Numeric.) > > I need to flatten a simple 3x3 array and try to do that with resize. > First time works, next time it doesn't. Here is an example: > > In [1]: from Numeric import * > > In [2]: a = array([[ 0.50622966, -0.54764389, -0.66619644], > [ 0.44338279, -0.4973092, 0.7457228 ],[-0.73969131, -0.67288704, > -0.00893311]]) > > In [3]: a.resize((9,)) > Out[3]: > array([ 0.50622966, -0.54764389, -0.66619644, 0.44338279, -0.4973092 , > 0.7457228 , > -0.73969131, -0.67288704, -0.00893311]) > > In [4]: a.resize((9,)) > --- > Traceback (most recent call > last) > > /home/cm/Documents/projects/SAXS/Eury/MC_rigid/ in > () > > : cannot resize an array that has been > referenced or is referencing > another array in this way. Use the resize function. > > Unfortunately, when applying resize on the actual array I have, I get a > similar, yet slightly different error: > The problem is that Numeric has no way of knowing whether a reference held is safe or not. In other-words, is the reference because of a view into a part of the array or a simple name binding (like having the '_' variable in the interpreter hold on to the old reference). A reallocation of the memory while other views are expecting to be able to walk over the old pointer is a recipe for problems and is the reason that Numeric has this restriction. NumPy has the same restriction but allows you to 'over-ride' it if you know what you are doing. To fix the problem, you need to get rid of all other references to the array that you want to resize. -Travis O. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] resize in old Numeric
Hi Christiaan On Wed, Dec 12, 2007 at 07:39:49PM +0100, Christian Meesters wrote: > I need to flatten a simple 3x3 array and try to do that with resize. > First time works, next time it doesn't. Here is an example: > > In [1]: from Numeric import * > > In [2]: a = array([[ 0.50622966, -0.54764389, -0.66619644], > [ 0.44338279, -0.4973092, 0.7457228 ],[-0.73969131, -0.67288704, > -0.00893311]]) > > In [3]: a.resize((9,)) > Out[3]: > array([ 0.50622966, -0.54764389, -0.66619644, 0.44338279, -0.4973092 , > 0.7457228 , > -0.73969131, -0.67288704, -0.00893311]) > > In [4]: a.resize((9,)) > --- > Traceback (most recent call > last) You'll notice that this doesn't happen when you run those commands as a script. IPython remembers the answer to the last command (as '_' and in the output history), hence the reference that Numeric complains about. > Unfortunately, when applying resize on the actual array I have, I get a > similar, yet slightly different error: > > Traceback (most recent call last): > File "sym_get.py", line 37, in > s.inquire_relation() > File "/usr/lib/python2.5/site-packages/mc_saxs/mc_rigid.py", line 915, > in inquire_relation > print rotation, rotation.resize((9,)) > ValueError: resize only works on contiguous arrays > > The arrays 'a' and 'rotation' are identical. They may have identical values without having identical layout in memory. Try a.ascontiguousarray(). Cheers Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] resize in old Numeric
On Dec 12, 2007 11:39 AM, Christian Meesters <[EMAIL PROTECTED]> wrote: > Hi, > > For compatibility reasons (work with Biopython) I would like to use to > Numeric in some code of mine. (Of course, I could make a little detour > converting into numpy.array, but first I wonder whether somebody might > know a solution for the problem with Numeric.) > > I need to flatten a simple 3x3 array and try to do that with resize. > First time works, next time it doesn't. Here is an example: The first thing that I would try is switching from resize to reshape. In general, you shouldn't be using resize except for certain odd applications as it can potentially result in reallocating the data. I'm pretty sure what you want is reshape. > > > In [1]: from Numeric import * > > In [2]: a = array([[ 0.50622966, -0.54764389, -0.66619644], > [ 0.44338279, -0.4973092, 0.7457228 ],[-0.73969131, -0.67288704, > -0.00893311]]) > > In [3]: a.resize((9,)) > Out[3]: > array([ 0.50622966, -0.54764389, -0.66619644, 0.44338279, -0.4973092 , > 0.7457228 , >-0.73969131, -0.67288704, -0.00893311]) > > In [4]: a.resize((9,)) > > --- > Traceback (most recent call > last) > > /home/cm/Documents/projects/SAXS/Eury/MC_rigid/ in > () > > : cannot resize an array that has been > referenced or is referencing > another array in this way. Use the resize function. > > Unfortunately, when applying resize on the actual array I have, I get a > similar, yet slightly different error: > > Traceback (most recent call last): > File "sym_get.py", line 37, in >s.inquire_relation() > File "/usr/lib/python2.5/site-packages/mc_saxs/mc_rigid.py", line 915, > in inquire_relation >print rotation, rotation.resize((9,)) > ValueError: resize only works on contiguous arrays > > The arrays 'a' and 'rotation' are identical. > > Any ideas, what's going on or how to solve this? > Numeric version is: 24.2 > > TIA > Christian > > ___ > Numpy-discussion mailing list > Numpy-discussion@scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > -- . __ . |-\ . . [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy large arrays?
On Wed, Dec 12, 2007 at 03:29:57PM +0100, Søren Dyrsting wrote: > I need to perform computations involving large arrays. A lot of rows and no > more than e.g. 34 columns. My first choice is python/numpy because I'm already > used to code in matlab. > > However I'm experiencing memory problems even though there is still 500 MB > available (2 GB total). I have cooked down my code to following meaningless > code snip. This code share some of the same structure and calls as my real > program and shows the same behaviour. I would guess that this is due to memory fragmentation. Have you tried the same experiment under Linux? This article details some of the problems you may encounter under Windows: http://www.ittvis.com/services/techtip.asp?ttid=3346 Regards Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] resize in old Numeric
Hi, For compatibility reasons (work with Biopython) I would like to use to Numeric in some code of mine. (Of course, I could make a little detour converting into numpy.array, but first I wonder whether somebody might know a solution for the problem with Numeric.) I need to flatten a simple 3x3 array and try to do that with resize. First time works, next time it doesn't. Here is an example: In [1]: from Numeric import * In [2]: a = array([[ 0.50622966, -0.54764389, -0.66619644], [ 0.44338279, -0.4973092, 0.7457228 ],[-0.73969131, -0.67288704, -0.00893311]]) In [3]: a.resize((9,)) Out[3]: array([ 0.50622966, -0.54764389, -0.66619644, 0.44338279, -0.4973092 , 0.7457228 , -0.73969131, -0.67288704, -0.00893311]) In [4]: a.resize((9,)) --- Traceback (most recent call last) /home/cm/Documents/projects/SAXS/Eury/MC_rigid/ in () : cannot resize an array that has been referenced or is referencing another array in this way. Use the resize function. Unfortunately, when applying resize on the actual array I have, I get a similar, yet slightly different error: Traceback (most recent call last): File "sym_get.py", line 37, in s.inquire_relation() File "/usr/lib/python2.5/site-packages/mc_saxs/mc_rigid.py", line 915, in inquire_relation print rotation, rotation.resize((9,)) ValueError: resize only works on contiguous arrays The arrays 'a' and 'rotation' are identical. Any ideas, what's going on or how to solve this? Numeric version is: 24.2 TIA Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
Fernando Perez wrote: >> I will put new binaries on the sourceforge site this weekend for both >> NumPy 1.0.4 and SciPy 0.6.0. I should be able to find an old PIII >> WinXP machine around somewhere, which I will devote to building the >> official non-SSE2 releases from here on out. >> > > Great, many thanks! > > Keep in mind that as Ryan's tests showed, Athlon XP chips also seem to > lack the SSE2 set, and those are a bit newer than the PIII, so they > might be a bit easier to find. The Athlon 64 and Athlon X2 do have > SSE2, but apparently the XP doesn't. I'm sure there's a wikipedia > page somewhere with a list of all the chips where SSE2 first > appeared... > I suspect the problem is that I was using an Athlon chip without SSE2 to build binaries in the past. That machine is now being used by my kids and my environment on it has been "messed-up" There may also be a problem with using the Microsoft compiler to build NumPy (and then calling out to ATLAS which was built with a different compiler). Issues like this have happened in the past. And, the most recent binaries of NumPy were built with the Microsoft compiler. -Travis O. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
> I will put new binaries on the sourceforge site this weekend for both > NumPy 1.0.4 and SciPy 0.6.0. I should be able to find an old PIII > WinXP machine around somewhere, which I will devote to building the > official non-SSE2 releases from here on out. Great, many thanks! Keep in mind that as Ryan's tests showed, Athlon XP chips also seem to lack the SSE2 set, and those are a bit newer than the PIII, so they might be a bit easier to find. The Athlon 64 and Athlon X2 do have SSE2, but apparently the XP doesn't. I'm sure there's a wikipedia page somewhere with a list of all the chips where SSE2 first appeared... Cheers, f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy large arrays?
Hi all
I need to perform computations involving large arrays. A lot of rows and no
more than e.g. 34 columns. My first choice is python/numpy because I'm
already used to code in matlab.
However I'm experiencing memory problems even though there is still 500 MB
available (2 GB total). I have cooked down my code to following meaningless
code snip. This code share some of the same structure and calls as my real
program and shows the same behaviour.
import numpy as N
import scipy as S
def stress():
x = S.randn(20,80)
for i in range(8):
print "%(0)d" % {"0": i}
s = N.dot(x.T, x)
sd = N.array([s.diagonal()])
r = N.dot(N.ones((N.size(x,0),1),'d'), sd)
x = x + r
x = x / 1.01
To different symptoms depending how big x are:
1) the program becomes extremely slow after a few iterations.
2) if the size of x is increased a little the program fails with the message
"MemoryError" for example at line 'x = x + r', but different places in the
code depending on the matrice size and which computer I'm testing. This
might also occur after several iterations, not just during the first pass.
I'm using Windows XP, ActivePython 2.5.1.1, NumPy 1.0.4, SciPy 0.6.0.
- Is there an error under the hood in NumPy?
- Am I balancing on the edge of the performance of Python/NumPy and should
consider other environments. Fortran, C, BLAS, LAPACK e.t.c.
- Am I misusing NumPy? Changing coding style will be a good workaround and
even perform on larger datasets without errors?
Thanks in advance
/Søren
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changing the distributed binary for numpy 1.0.4 for windows ?
On Dec 11, 2007 5:58 PM, Fernando Perez <[EMAIL PROTECTED]> wrote: > On Dec 11, 2007 6:45 PM, Ryan Krauss <[EMAIL PROTECTED]> wrote: > > Near as I can tell, this is still unresolved for people with non-sse2 > > machines. Is that right? > > Yup. Your more detailed testing seems to confirm the hunch I had at > the weekend workshop that SSE2 is the culprit. Thanks for the info. > > It would be really great if we could somehow resolve this quickly. I > have a WinXP install under Linux, but it seems to see my CPU as an > Athlon X2, so that won't work. But I also have an old laptop with a > dual-boot XP that's a PIII (no SSE2, and probably the oldest > reasonable hardware we can expect to support). I will put new binaries on the sourceforge site this weekend for both NumPy 1.0.4 and SciPy 0.6.0. I should be able to find an old PIII WinXP machine around somewhere, which I will devote to building the official non-SSE2 releases from here on out. Cheers, -- Jarrod Millman Computational Infrastructure for Research Labs 10 Giannini Hall, UC Berkeley phone: 510.643.4014 http://cirl.berkeley.edu/ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
