[Numpy-discussion] Build failure at rev8246
Hi, I tried upgrading today and had trouble building numpy (after rm -rf build). My full build log is here: http://www.filedump.net/dumped/build1274454454.txt If someone can point me in the right direction, I'd appreciate it very much. To excerpts from the log file: Running from numpy source directory.numpy/core/setup_common.py:86: MismatchCAPIWarning: API mismatch detected, the C API version numbers have to be updated. Current C api version is 4, with checksum 59750b518272c8987f02d66445afd3f1, but recorded checksum for C API version 4 in codegen_dir/cversions.txt is 3d8940bf7b0d2a4e25be4338c14c3c85. If functions were added in the C API, you have to update C_API_VERSION in numpy/core/setup_common.pyc. MismatchCAPIWarning) In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:36: numpy/core/src/multiarray/buffer.c: At top level: numpy/core/src/multiarray/buffer.c:715: error: conflicting types for ‘_descriptor_from_pep3118_format’ numpy/core/src/multiarray/common.c:221: note: previous implicit declaration of ‘_descriptor_from_pep3118_format’ was here numpy/core/src/multiarray/buffer.c: In function ‘_descriptor_from_pep3118_format’: numpy/core/src/multiarray/buffer.c:751: warning: assignment from incompatible pointer type In file included from numpy/core/src/multiarray/multiarraymodule_onefile.c:45: numpy/core/src/multiarray/multiarraymodule.c: In function ‘initmultiarray’: numpy/core/src/multiarray/multiarraymodule.c:3062: warning: implicit declaration of function ‘_numpymemoryview_init’ error: Command gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -Inumpy/core/include -Ibuild/src.linux-i686-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.linux-i686-2.6/numpy/core/src/multiarray -Ibuild/src.linux-i686-2.6/numpy/core/src/umath -c numpy/core/src/multiarray/multiarraymodule_onefile.c -o build/temp.linux-i686-2.6/numpy/core/src/multiarray/multiarraymodule_onefile.o failed with exit status 1 Hope this helps. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Build failure at rev8246
Fri, 21 May 2010 08:09:55 -0700, T J wrote: I tried upgrading today and had trouble building numpy (after rm -rf build). My full build log is here: http://www.filedump.net/dumped/build1274454454.txt Your SVN checkout might be corrupted, containing a mix of old and new files. Try building from a clean checkout. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Build failure at rev8246
On Fri, May 21, 2010 at 8:51 AM, Pauli Virtanen p...@iki.fi wrote: Fri, 21 May 2010 08:09:55 -0700, T J wrote: I tried upgrading today and had trouble building numpy (after rm -rf build). My full build log is here: http://www.filedump.net/dumped/build1274454454.txt Your SVN checkout might be corrupted, containing a mix of old and new files. Try building from a clean checkout. That was it! When I did an svn update, I noticed conflicts all over the place and accepted with tf but I guess that was not enough. Given that I haven't really touched this directory, what was the likely cause of this corruption? The last time I svn update'd was a few months ago. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Summation of large float32/float64 arrays
On Fri, May 21, 2010 at 15:13, Matthew Turk matthewt...@gmail.com wrote: Hi all, I have a possibly naive question. I don't really understand this particular set of output: In [1]: import numpy In [2]: a1 = numpy.random.random((512,512,512)).astype(float32) In [3]: a1.sum(axis=0).sum(axis=0).sum(axis=0) Out[3]: 67110312.0 In [4]: a1.sum() Out[4]: 16777216.0 I recognize that the intermediate sums may accumulate error differently than a single call to .sum(), but I guess my concern is that it's accumulating a lot faster than I anticipated. (Interesting to note that a1.sum() returns 0.5*512^3, down to the decimal; is it summing up the mean, which should be ~0.5?) However, with a 256^3 array: In [1]: import numpy In [2]: a1 = numpy.random.random((256,256,256)).astype(float32) In [3]: a1.sum(axis=0).sum(axis=0).sum(axis=0) Out[3]: 8389703.0 In [4]: a1.sum() Out[4]: 8389245.0 The errors are much more reasonable. Is there an overflow or something that occurs with the 512^3? It's not quite an overflow. In [1]: from numpy import * In [2]: x = float32(16777216.0) In [3]: x + float32(0.9) Out[3]: 16777216.0 You are accumulating your result in a float32. With the a.sum() approach, you eventually hit a level where the next number to add is always less than the relative epsilon of float32 precision. So the result doesn't change. And will never change again as long as you only add one number at a time. Summing along the other axes creates smaller intermediate sums such that you are usually adding together numbers roughly in the same regime as each other, so you don't lose as much precision. Use a.sum(dtype=np.float64) to use a float64 accumulator. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Summation of large float32/float64 arrays
Hi Robert, It's not quite an overflow. In [1]: from numpy import * In [2]: x = float32(16777216.0) In [3]: x + float32(0.9) Out[3]: 16777216.0 You are accumulating your result in a float32. With the a.sum() approach, you eventually hit a level where the next number to add is always less than the relative epsilon of float32 precision. So the result doesn't change. And will never change again as long as you only add one number at a time. Summing along the other axes creates smaller intermediate sums such that you are usually adding together numbers roughly in the same regime as each other, so you don't lose as much precision. Thank you very much for that explanation; that completely makes sense. Use a.sum(dtype=np.float64) to use a float64 accumulator. I didn't know about the dtype for accumulators/operators -- that did just the trick. Much obliged, Matt ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Summation of large float32/float64 arrays
On 5/21/2010 4:13 PM, Matthew Turk wrote: a1 = numpy.random.random((512,512,512)).astype(float32) This consistently gives me a MemoryError. I believe I have plenty of physical memory. (That should require about 1.3G during creation, right? I have 8G.) It seems I'm hitting some kind of 1G memory use limit.How to think about this? I can create the initial 64 bit array no problem. However I cannot create the second 32 bit array, despite having plenty of physical memory. The copy method also fails; or even creating a second array the same size fails, unless I first delete `a1`. I realize this is probably a naive and operating system dependent question. Apologies if it is off topic. Thanks, Alan Isaac (running 32bit Python 2.6.5 on 64bit Vista; NumPy version import 1.4.1rc2) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Summation of large float32/float64 arrays
On 5/21/2010 2:30 PM, Alan G Isaac wrote: On 5/21/2010 4:13 PM, Matthew Turk wrote: a1 = numpy.random.random((512,512,512)).astype(float32) This consistently gives me a MemoryError. I believe I have plenty of physical memory. (That should require about 1.3G during creation, right? I have 8G.) It seems I'm hitting some kind of 1G memory use limit.How to think about this? I can create the initial 64 bit array no problem. However I cannot create the second 32 bit array, despite having plenty of physical memory. The copy method also fails; or even creating a second array the same size fails, unless I first delete `a1`. I realize this is probably a naive and operating system dependent question. Apologies if it is off topic. Thanks, Alan Isaac (running 32bit Python 2.6.5 on 64bit Vista; NumPy version import 1.4.1rc2) The MemoryError is not not unexpected. First, the 32-bit Python interpreter can only use 2 GB of your memory. Second, numpy arrays need a contiguous, un-fragmented, range of memory to be available within that 2 GB address space. In this example there is no contiguous 512 MB space left after creating the 1 GB array. Practically it does not seem possible to allocate a single array larger than about 1.3 GB on win32. The solution is to use a 64-bit version of Python and numpy. Christoph ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
