[Numpy-discussion] Behaviour of integer powers
Hi all, Ticket #955 (http://scipy.org/scipy/numpy/ticket/955) touches on the following issue: >>> 0.0 ** np.array([-1, 0, 1], dtype=np.int32) array([ Inf, 1., 0.]) >>> 0.0 ** np.array([-1, 0, 1], dtype=np.int32)[0] Traceback (most recent call last): File "", line 1, in ZeroDivisionError: 0.0 cannot be raised to a negative power This is on a 32-bit platform. As I understand this happens because, in the second case, Python sees that "-1" is an int and does the power operation. In other words, when we raise to the power of an array, the NumPy machinery is involved, whereas if we raise to np.int32(-1), it is not. This is due to the int32 type deriving from the Python int. I can't think of an easy way to address the problem, but I was hoping to get some advice from the list. Thanks Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Behaviour of integer powers
On Sun, Feb 8, 2009 at 03:54, Stéfan van der Walt wrote: > Hi all, > > Ticket #955 (http://scipy.org/scipy/numpy/ticket/955) touches on the > following issue: > 0.0 ** np.array([-1, 0, 1], dtype=np.int32) > array([ Inf, 1., 0.]) 0.0 ** np.array([-1, 0, 1], dtype=np.int32)[0] > > Traceback (most recent call last): > File "", line 1, in > ZeroDivisionError: 0.0 cannot be raised to a negative power > > This is on a 32-bit platform. > > As I understand this happens because, in the second case, Python sees > that "-1" is an int and does the power operation. In other words, > when we raise to the power of an array, the NumPy machinery is > involved, whereas if we raise to np.int32(-1), it is not. This is due > to the int32 type deriving from the Python int. > > I can't think of an easy way to address the problem, but I was hoping > to get some advice from the list. I don't think there is anything we can do to fix this except not to subclass from int. I think float.__pow__(self, other) checks that isinstance(other, int) and does its own thing. numpy.int_.__rpow__ will never get called, and that's the only place we can implement our logic. We can document the wart and recommend casting the base to a float64 scalar first. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Sun, Feb 8, 2009 at 3:21 AM, Ondrej Certik wrote: > Hi David, > >> Sorry for the confusion: numscons is NOT the preferred build system. >> The current numpy.distutils extensions, as shipped by numpy, is the >> preferred one. Numscons is more an experiment, if you want. > > Ah, I see, thanks for the clarification. > >>> So is it supposed to be in Debian? >> >> No, I don't think it should be. It is not yet stabilized code wise, so >> it does not make much sense to package it. > > Ok. > >> >>> Is numscons supposed to be >>> a build system for other projects as well? Why not to just send the >>> needed patches to scons and just use scons? >> >> Because you cannot just use scons. Numscons is a library build on top >> of scons, for the needs of numpy. There also needs to be some hook >> from numpy.distutils to use scons (numscons adds a new distutils >> command, which is used instead of build to build any compiled >> code-based extensions). Most of the changes needed for scons have been >> integrated upstream, though, except one or two things. > > I see. I think it's a bit confusing that one needs to build a new > build system just to build numpy, e.g. that both distutils and scons > are not good enough. I don't find it that surprising - numpy and scipy require some relatively advanced features (mixed language and cross-platform with support for many toolchains). Within the open source tools, I know only two which can handle those requirements: scons and cmake. For example, it would almost impossible to build numpy/scipy with autoconf. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Behaviour of integer powers
On Sun, Feb 8, 2009 at 5:09 AM, Robert Kern wrote: > On Sun, Feb 8, 2009 at 03:54, Stéfan van der Walt wrote: >> Hi all, >> >> Ticket #955 (http://scipy.org/scipy/numpy/ticket/955) touches on the >> following issue: >> > 0.0 ** np.array([-1, 0, 1], dtype=np.int32) >> array([ Inf, 1., 0.]) > 0.0 ** np.array([-1, 0, 1], dtype=np.int32)[0] >> >> Traceback (most recent call last): >> File "", line 1, in >> ZeroDivisionError: 0.0 cannot be raised to a negative power >> >> This is on a 32-bit platform. >> >> As I understand this happens because, in the second case, Python sees >> that "-1" is an int and does the power operation. In other words, >> when we raise to the power of an array, the NumPy machinery is >> involved, whereas if we raise to np.int32(-1), it is not. This is due >> to the int32 type deriving from the Python int. >> >> I can't think of an easy way to address the problem, but I was hoping >> to get some advice from the list. > > I don't think there is anything we can do to fix this except not to > subclass from int. I think float.__pow__(self, other) checks that > isinstance(other, int) and does its own thing. numpy.int_.__rpow__ > will never get called, and that's the only place we can implement our > logic. > > We can document the wart and recommend casting the base to a float64 > scalar first. > > -- > Robert Kern > I thought when in doubt about the domain, the useful way around this, is to call np.power directly. This is how it is used in stats.distributions. >>> 0.0 ** np.array([-1, 0, 1], dtype=np.int32)[0] Traceback (most recent call last): File "", line 1, in ZeroDivisionError: 0.0 cannot be raised to a negative power >>> np.power(0.0, np.array([-1, 0, 1], dtype=np.int32)[0]) 1.#INF >>> 0.01 ** np.array([-1, 0, 1], dtype=np.int32)[0] 100.0 >>> np.power(np.nan, np.array([-1, 0, 1], dtype=np.int32)[0]) -1.#IND >>> np.nan ** np.array([-1, 0, 1], dtype=np.int32)[0] Traceback (most recent call last): File "", line 1, in ValueError: (33, 'Domain error') But I just found that nan in the exponent in an array are not propagated: >>> 0.0 ** np.array([-np.nan, 0, 1], dtype=np.int32)[0] 1.0 >>> np.power(0.0, np.array([-np.nan, 0, 1], dtype=np.int32)[0]) 1.0 >>> np.power(0.0, -np.nan) 1.#QNAN >>> np.power(0.0, np.nan) -1.#IND >>> 0.0**np.nan 0.0 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Behaviour of integer powers
On Sun, Feb 8, 2009 at 8:02 AM, wrote: > But I just found that nan in the exponent in an array are not propagated: > 0.0 ** np.array([-np.nan, 0, 1], dtype=np.int32)[0] > 1.0 np.power(0.0, np.array([-np.nan, 0, 1], dtype=np.int32)[0]) > 1.0 correction: np.power propagates nans, it's again the casting of nan to zero int, that tripped me up. I shouln't cut and paste examples too fast. >>> np.power(0.0, np.array([-np.nan, 0, 1])) array([ NaN, 1., 0.]) >>> np.power(0.0, np.array([-np.nan, 0, 1]))[0] 1.#QNAN Josef ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Selection of only a certain number of fields
Francesc Alted pytables.org> writes: > > What are some common use cases for this feature? > > > > I use structured arrays quite a lot, but I haven't found myself > > wanting something like this. If I do need a subset of a structured > > array generally I use something like > > > > [rec[n] for n in 'name age gender'.split()] > > Good point. However, there are still some very valid reasons for having > an idiom like: > > newarr = arr[['name', 'age']] > > returning a record array. > > The first one (and most important IMO), is that newarr continues to be > an structured array (BTW, when changed this name from the original > record array?), and you can use all the features of these beasts with > it. Other reason (albeit a bit secondary) is that its data buffer can > be shared through the array interface with other applications, or plain > C code, in a relatively straightforward way. However, if newarr > becomes a list (or dictionary), this is simply not possible. > > Cheers, > That's not a sample use case ;) One of the things I love about Python is that it has a small core set of features and tries to avoid having many ways to do the same thing. This makes it extremely easy to learn. With every new feature, numpy gets a little bit harder to learn, there's more to document and the code base gets larger and so harder to maintain. In those senses, whenever you add a new function/feature to numpy, it gets a little bit worse. So I think it would be nice to have some concrete examples of what the new feature will be useful for, just to show how it outweighs those negatives. As a bonus, they'd provide nice examples to put in the documentation :). Neil PS. Thanks for your work on pytables! I've used it quite a bit, mostly for reading hdf5 files. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] segfaults when passing ndarray subclass to ufunc with out=None
I am seeing some really strange behavior when I try to pass an ndarray subclass and out=None to numpy's ufuncs. This example will reproduce the problem with svn numpy, the first print statement yields 1 as expected, the second yields "" and the third yields a segmentation fault: import numpy as np class MyArray(np.ndarray): __array_priority__ = 20 def __new__(cls): return np.asarray(1).view(cls).copy() def __repr__(self): return 'my_array' __str__ = __repr__ def __mul__(self, other): return super(MyArray, self).__mul__(other) def __rmul__(self, other): return super(MyArray, self).__rmul__(other) mine = MyArray() print np.multiply(1, 1, None) x = np.multiply(mine, mine, None) print type(x) print x Darren ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Sun, Feb 8, 2009 at 3:10 AM, David Cournapeau wrote: > On Sun, Feb 8, 2009 at 3:21 AM, Ondrej Certik wrote: >> Hi David, >> >>> Sorry for the confusion: numscons is NOT the preferred build system. >>> The current numpy.distutils extensions, as shipped by numpy, is the >>> preferred one. Numscons is more an experiment, if you want. >> >> Ah, I see, thanks for the clarification. >> So is it supposed to be in Debian? >>> >>> No, I don't think it should be. It is not yet stabilized code wise, so >>> it does not make much sense to package it. >> >> Ok. >> >>> Is numscons supposed to be a build system for other projects as well? Why not to just send the needed patches to scons and just use scons? >>> >>> Because you cannot just use scons. Numscons is a library build on top >>> of scons, for the needs of numpy. There also needs to be some hook >>> from numpy.distutils to use scons (numscons adds a new distutils >>> command, which is used instead of build to build any compiled >>> code-based extensions). Most of the changes needed for scons have been >>> integrated upstream, though, except one or two things. >> >> I see. I think it's a bit confusing that one needs to build a new >> build system just to build numpy, e.g. that both distutils and scons >> are not good enough. > > I don't find it that surprising - numpy and scipy require some > relatively advanced features (mixed language and cross-platform with > support for many toolchains). Within the open source tools, I know > only two which can handle those requirements: scons and cmake. For > example, it would almost impossible to build numpy/scipy with > autoconf. Yes, I am investigating cmake, it's pretty cool. I wrote some macros for cython etc. What I like about cmake is that it is cross platform and it just produces makefiles on linux, or visual studio files (or whatever) on windows. When I get more experience with it, I'll post here. What I don't like on cmake is that it uses it's own language, instead of python, on the other hand, so far everything seems to just work. Contrary to numscons, where it looks almost like a new python program just to build numpy. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
> I don't find it that surprising - numpy and scipy require some > relatively advanced features (mixed language and cross-platform with > support for many toolchains). Within the open source tools, I know > only two which can handle those requirements: scons and cmake. For > example, it would almost impossible to build numpy/scipy with > autoconf. I know some autotools/autoconf hackers that could probably get it to build numpy/scipy, ..., oh what, you want Windows support? ;-) Cheers, Brian > David > ___ > Numpy-discussion mailing list > Numpy-discussion@scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
> Yes, I am investigating cmake, it's pretty cool. I wrote some macros > for cython etc. What I like about cmake is that it is cross platform > and it just produces makefiles on linux, or visual studio files (or > whatever) on windows. When I get more experience with it, I'll post > here. Yes, while I haven't played with cmake much, it does look very nice. > What I don't like on cmake is that it uses it's own language, instead > of python, on the other hand, so far everything seems to just work. I too don't like the idea of cmake having its own language. While I love to learn new languages, I like to have good reasons and I'm not sure that building software is a good reason. I have been playing with Scons and I really do like the fact that it is Python. While I haven't tried it yet, I am pretty sure that we could use IPython to interactively debug a Scons build. That would be really nice! > Contrary to numscons, where it looks almost like a new python program > just to build numpy. But hold on, it is not fair to compare cmake with numscons (this is an apples and oranges thing). You should compare cmake with *Scons* itself. The problem with comparing cmake with numscons is that cmake can't do what numscons does - i.e., plug into distutils. There is a lot of extra things that distutils does other than build extensions and cmake won't do these things out of the box. Obviously, someone could get cmake to do these things, but then you are developing a complete distutils replacement. And I think that any distutils replacement should done in Python. Cheers, Brian > Ondrej > ___ > Numpy-discussion mailing list > Numpy-discussion@scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Sun, Feb 8, 2009 at 12:26 PM, Brian Granger wrote: >> Yes, I am investigating cmake, it's pretty cool. I wrote some macros >> for cython etc. What I like about cmake is that it is cross platform >> and it just produces makefiles on linux, or visual studio files (or >> whatever) on windows. When I get more experience with it, I'll post >> here. > > Yes, while I haven't played with cmake much, it does look very nice. > >> What I don't like on cmake is that it uses it's own language, instead >> of python, on the other hand, so far everything seems to just work. > > I too don't like the idea of cmake having its own language. While I > love to learn new languages, I like to have good reasons and I'm not > sure that building software is a good reason. > > I have been playing with Scons and I really do like the fact that it > is Python. While I haven't tried it yet, I am pretty sure that we > could use IPython to interactively debug a Scons build. That would be > really nice! > >> Contrary to numscons, where it looks almost like a new python program >> just to build numpy. > > But hold on, it is not fair to compare cmake with numscons (this is an > apples and oranges thing). You should compare cmake with *Scons* > itself. The problem with comparing cmake with numscons is that cmake > can't do what numscons does - i.e., plug into distutils. There is a > lot of extra things that distutils does other than build extensions > and cmake won't do these things out of the box. Obviously, someone > could get cmake to do these things, but then you are developing a > complete distutils replacement. And I think that any distutils > replacement should done in Python. Yes, I agree with what you said. I still want to try cmake though, if nothing, at least to know where we want to go. Yes, I should compare cmake with scons. We can either develop python front end to cmake, or just help improve scons/numscons etc. I really like that cmake generates native makefiles and windows visual studio files etc. In any case, whatever we choose to use, I want to have experience with both scons and cmake. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Mon, Feb 9, 2009 at 4:39 AM, Ondrej Certik wrote: > > Yes, I am investigating cmake, it's pretty cool. I wrote some macros > for cython etc. What I like about cmake is that it is cross platform > and it just produces makefiles on linux, or visual studio files (or > whatever) on windows. When I get more experience with it, I'll post > here. That's exactly what I don't like about cmake - it means you can't produce accurate builds (you need to rerun cmake everytime you change the configuration or dependencies, whereas this is automatic with scons/waf). It also have (used to have) very poor documentation unless you buy the book - but it looks like this is changing. > What I don't like on cmake is that it uses it's own language, instead > of python, on the other hand, so far everything seems to just work. > Contrary to numscons, where it looks almost like a new python program > just to build numpy. Again, numscons is just a library on top of scons to support things we need in numpy, it is not really a new program - it is a separate package to avoid adding experimental code to numpy itself. Numscons is ~ 3000 LOC, of which 1000 is for the core, 1000 for blas/lapack/fortran support, and 1000 for tools which are not properly supported in scons (recent MSVC, python extensions). I think you would have almost as much work with cmake if not more - when I started numscons, cmake did not have fortran support (it now has, although I don't know how far - it does not seem to handle mixed fortran/C, for example). If you don't mind fast changing software, you should look at waf: it is extremely fast, based on python. It also handles a lot of distribution issues already (tarball generation, compiler selection, etc...) which scons does not. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
On Mon, Feb 9, 2009 at 5:17 AM, Brian Granger wrote: >> I don't find it that surprising - numpy and scipy require some >> relatively advanced features (mixed language and cross-platform with >> support for many toolchains). Within the open source tools, I know >> only two which can handle those requirements: scons and cmake. For >> example, it would almost impossible to build numpy/scipy with >> autoconf. > > I know some autotools/autoconf hackers that could probably get it to > build numpy/scipy, ..., oh what, you want Windows support? ;-) Yes, autotools support for MS compilers is very poor - I think it is fair to say it is unusable as of today. And one of the goal of numscons is to make the process easier to use/hack (it is a mixed success on this front - the build knowledge is much easier to read in numscons than in distutils IMHO, but it is still too arcane on some fronts, partly because distutils is way too flexible in areas it should not). David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
> That's exactly what I don't like about cmake - it means you can't > produce accurate builds (you need to rerun cmake everytime you change > the configuration or dependencies, whereas this is automatic with > scons/waf). It also have (used to have) very poor documentation unless > you buy the book - but it looks like this is changing. You can always rerun cmake if you want (and make it automatic). Imho that's not a problem. But maybe it is done in a better way with scons. > >> What I don't like on cmake is that it uses it's own language, instead >> of python, on the other hand, so far everything seems to just work. >> Contrary to numscons, where it looks almost like a new python program >> just to build numpy. > > Again, numscons is just a library on top of scons to support things we > need in numpy, it is not really a new program - it is a separate > package to avoid adding experimental code to numpy itself. Numscons is > ~ 3000 LOC, of which 1000 is for the core, 1000 for > blas/lapack/fortran support, and 1000 for tools which are not properly > supported in scons (recent MSVC, python extensions). > > I think you would have almost as much work with cmake if not more - > when I started numscons, cmake did not have fortran support (it now > has, although I don't know how far - it does not seem to handle mixed > fortran/C, for example). > > If you don't mind fast changing software, you should look at waf: it > is extremely fast, based on python. It also handles a lot of > distribution issues already (tarball generation, compiler selection, > etc...) which scons does not. Yes, waf is pretty cool, even though the last time I looked at it, it wasn't able to compile my project, which was larger than just couple files. But it seems it's development is progressing fast. As to the other things, one nice thing about cmake is that it is production ready right now, it is well tested (kde4) and it is in distributions (e.g. Debian). Everything else needs nontrivial adjustments, or is not as well tested (yet). Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] preferred numpy build system
Ondrej Certik wrote: >> That's exactly what I don't like about cmake - it means you can't >> produce accurate builds (you need to rerun cmake everytime you change >> the configuration or dependencies, whereas this is automatic with >> scons/waf). It also have (used to have) very poor documentation unless >> you buy the book - but it looks like this is changing. >> > > You can always rerun cmake if you want (and make it automatic). Imho > that's not a problem. But maybe it is done in a better way with scons. > I think it is a problem - it means you have to update it explicitly when the configuration changes. In scons, the signature concept is quite powerful: not only are file dependencies handled, but also command lines, etc... For example, in numscons, if you change the blas/lapack from atlas to say MKL, only linking and eventually configuration changes are rebuilt. If you change the fortran compiler but not the C compiler, only fortran code is rebuilt. All of this is 100 % automatic. When testing software on many toolchains/platforms, this is very valuable - it is one of the main reason why Intel, Vmware use it, at least from the information I understood from the Mailing list. In terms of accuracy, scons is way beyond anything cmake has to offer. In the case of numpy, I think cmake would be difficult to integrate into distutils > Yes, waf is pretty cool, even though the last time I looked at it, it > wasn't able to compile my project, which was larger than just couple > files. > I doubt this has anything to do with the project size. Waf seems more scalable than scons, and scons can handle big projects (almost all vmware projects are built using scons, for example), albeit slowly. > As to the other things, one nice thing about cmake is that it is > production ready right now, it is well tested (kde4) and it is in > distributions (e.g. Debian). For portability reasons, I think the build system should always distributed with the project you are building (like autoconf does, actually); this is another drawback of cmake. Waf's main author is against any waf packaging - waf can be distributed as one < 100 kb python file, which is very nice. I don't want to sound overly critical against cmake: obviously, since it is used by KDE4, it is definitely a good software. It supports some features which I wish scons had (rpath and install-relinking, distribution related features). David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] how to get the corresponding eigenvector for a specific eigen value?
Hey all, I am currently working on a large matrix, and I already have a specific eigen value that I want to use in order to find out its corresponding eigen vector. Is there an easy way to do so? I have tried with linalg.solve(a, b), where I put a as the Matrix A - eigen value* unit matrix, and b as the zero matrix. But the solution returned is a zero matrix, which I really find disappointing. I have also tried with eig(A), which finds out all the eigen vectors of matrix A, but it takes too long to run especially the order of my matrix is like 10,000. So right now, I really find myself stuck. Is there anyone who can help me? My great gratitude. Best, Beryl ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to get the corresponding eigenvector for a specific eigen value?
> I have tried with linalg.solve(a, b), where I put a as the > Matrix A - eigen value* unit matrix, and b as the zero matrix. But the > solution returned is a zero matrix, which I really find disappointing. So if you're trying to solve (A - \lambda I) x = b, try appending an extra row to your (A - \lambda I) matrix with all ones; the output of this will be the some of the elements of your eigenvector. Append a 1 to the end of your b. This will exclude the case where x = 0 as valid solution, as you'll then require that \sum_i x_i = 1. You'll probably want to renormalize properly afterwards. Quick and dirty, but should work. --Hoyt -- + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoy...@gmail.com ++ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] NumPy buildbot
Hi all, Due to changes in our firewall configuration, the buildbot was offline since 14 January. Everything is sorted out now, so you can access the buildbot again at http://buildbot.scipy.org Some of the build slaves are still offline, and I have contacted their maintainers. Regards Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
