Re: [Numpy-discussion] numpy.mean problems
On Fri, Dec 9, 2011 at 11:47 AM, ferreirafm wrote: > > Hi everyone, > I'm quite new to numpy and python either. Could someone, please, tell me > what I'm doing wrong? > Here goes my peace of code: > > def stats(filename): > """Utilility to perform some basic statistics on columns.""" > tab = get_textab(filename) > stat_list = [ ] > for row in sort_tab(tab): > if row['length'] >= 15: > stat_list.append(row) > stat_array = np.array(stat_list) > print type(sort_tab(tab)) > print type(stat_array) > #print stat_array.mean(axis=0) > print np.mean(stat_array, axis=0) > > Which results in: > > When posting to the mailing list, it's a good idea to have a small, self contained example (otherwise we can't reproduce your problem). In this specific case, I'd like to be able to see what the outputs of "print tab" and "print stat_array" are. Regards Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Warning related to __multiarray_api.h
Hi, I don't know if this is of importance, but when I compile code using the numpy C API, I get the warning: site-packages/numpy/core/include/numpy/__multiarray_api.h:1532: warning: 'int _import_array()' defined but not used Might be worth cleaning it up. Best regards, Mads ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] EXTERNAL: Re: Problem with using PyArray_AsCArray
I actually figured it out. I went one level down in the array and it took it. -Original Message- From: numpy-discussion-boun...@scipy.org [mailto:numpy-discussion-boun...@scipy.org] On Behalf Of Chris.Barker Sent: Friday, December 09, 2011 3:33 PM To: Discussion of Numerical Python Subject: EXTERNAL: Re: [Numpy-discussion] Problem with using PyArray_AsCArray On 12/9/11 11:25 AM, Ng, Enrico wrote: > I am trying to pass a multi-dimensional ndarray to C as a multi-dimensional C > array for the purposes of passing it to mathematica. I am using > PyArray_AsCArray but getting an error. I understand that SWIG, Boost, et. al are perhaps too heavyweight for this one use, but you might want to give Cython a try. It makes it really easy to grab a numpy array (and tst it for compliance), and then you can do whatever you want with the data pointer: http://wiki.cython.org/tutorials/numpy http://wiki.cython.org/WrappingNumpy (this one is marked as depricated, but may be what you want in your case, as you do want a raw C array) Some wore googling and browsing of the Cython list will likely yield examples similar to yours. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problem with using PyArray_AsCArray
On 12/9/11 11:25 AM, Ng, Enrico wrote: > I am trying to pass a multi-dimensional ndarray to C as a multi-dimensional C > array for the purposes of passing it to mathematica. I am using > PyArray_AsCArray but getting an error. I understand that SWIG, Boost, et. al are perhaps too heavyweight for this one use, but you might want to give Cython a try. It makes it really easy to grab a numpy array (and tst it for compliance), and then you can do whatever you want with the data pointer: http://wiki.cython.org/tutorials/numpy http://wiki.cython.org/WrappingNumpy (this one is marked as depricated, but may be what you want in your case, as you do want a raw C array) Some wore googling and browsing of the Cython list will likely yield examples similar to yours. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy.mean problems
Hi everyone, I'm quite new to numpy and python either. Could someone, please, tell me what I'm doing wrong? Here goes my peace of code: def stats(filename): """Utilility to perform some basic statistics on columns.""" tab = get_textab(filename) stat_list = [ ] for row in sort_tab(tab): if row['length'] >= 15: stat_list.append(row) stat_array = np.array(stat_list) print type(sort_tab(tab)) print type(stat_array) #print stat_array.mean(axis=0) print np.mean(stat_array, axis=0) Which results in: Traceback (most recent call last): File "/home/ferreirafm/bin/cross.py", line 213, in main() File "/home/ferreirafm/bin/cross.py", line 204, in main stats(filename) File "/home/ferreirafm/bin/cross.py", line 146, in stats print np.mean(stat_array, axis=0) File "/usr/lib64/python2.7/site-packages/numpy/core/fromnumeric.py", line 2374, in mean return mean(axis, dtype, out) TypeError: unsupported operand type(s) for +: 'numpy.void' and 'numpy.void' -- View this message in context: http://old.nabble.com/numpy.mean-problems-tp32945124p32945124.html Sent from the Numpy-discussion mailing list archive at Nabble.com. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Problem with using PyArray_AsCArray
I am trying to pass a multi-dimensional ndarray to C as a multi-dimensional C
array for the purposes of passing it to mathematica. I am using
PyArray_AsCArray but getting an error.
##
Python Code:
import Image
from scipy.misc import fromimage
img = Image.open("../APLS/image709_enhanced_2.tif")
nimg = fromimage(img)
...
mathlink_PutIntegerArray(nimg)
##
C Wrapper Code:
static PyObject * mathlink_PutIntegerArray(mathlink_Link *self, PyObject *args)
{
npy_intp dims[3]; /* PyArray_AsCArray is for ndim <= 3 */
PyObject *o1;
double *d1;
longresult;
i=PyArray_AsCArray(&o1, (void *)&d1, dims, PyArray_NDIM(o1),
PyArray_DescrFromType(PyArray_TYPE(o1)));
...
}
Program received signal SIGSEGV, Segmentation fault.
0x2efa3d2e in PyArray_AsCArray (op=0x7fffdf68, ptr=0x7fffdf60,
dims=0x7fffdf40, nd=3, typedescr=)
at numpy/core/src/multiarray/multiarraymodule.c:218
218 ptr3[i][j] = ap->data + i*ap->strides[0] +
j*ap->strides[1]
#
I am able to read i,j,ap->data, ap->strides[0], ap->strides[1] so the error
seems to be in the assignment to ptr3[i][j]
This happens on the first instance i=0 j=0
#
PyArray_AsCArray code is below.
NPY_NO_EXPORT int
PyArray_AsCArray(PyObject **op, void *ptr, npy_intp *dims, int nd,
PyArray_Descr* typedescr)
{
PyArrayObject *ap;
npy_intp n, m, i, j;
char **ptr2;
char ***ptr3;
if ((nd < 1) || (nd > 3)) {
PyErr_SetString(PyExc_ValueError,
"C arrays of only 1-3 dimensions available");
Py_XDECREF(typedescr);
return -1;
}
if ((ap = (PyArrayObject*)PyArray_FromAny(*op, typedescr, nd, nd,
CARRAY, NULL)) == NULL) {
return -1;
}
switch(nd) {
case 1:
*((char **)ptr) = ap->data;
break;
case 2:
n = ap->dimensions[0];
ptr2 = (char **)_pya_malloc(n * sizeof(char *));
if (!ptr2) {
goto fail;
}
for (i = 0; i < n; i++) {
ptr2[i] = ap->data + i*ap->strides[0];
}
*((char ***)ptr) = ptr2;
break;
case 3:
n = ap->dimensions[0];
m = ap->dimensions[1];
ptr3 = (char ***)_pya_malloc(n*(m+1) * sizeof(char *));
if (!ptr3) {
goto fail;
}
for (i = 0; i < n; i++) {
ptr3[i] = ptr3[n + (m-1)*i];
for (j = 0; j < m; j++) {
ptr3[i][j] = ap->data + i*ap->strides[0] + j*ap->strides[1];
}
}
*((char )ptr) = ptr3;
}
memcpy(dims, ap->dimensions, nd*sizeof(npy_intp));
*op = (PyObject *)ap;
return 0;
fail:
PyErr_SetString(PyExc_MemoryError, "no memory");
return -1;
}
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] trouble building numpy 1.6.1 on Scientific Linux 5
I'm trying to build numpy 1.6.1 on Scientific Linux 5 but the unit tests
claim the wrong version of fortran was used. I thought I knew how to
avoid that, but it's not working.
I don't have atlas (this needs to run on a lot of
similar-but-not-identical machines). I believe blas and lapack were
built against gfortran:
-bash-3.2$ ldd /usr/lib64/libblas.so
linux-vdso.so.1 => (0x7fff4bffd000)
libm.so.6 => /lib64/libm.so.6 (0x2ab26a0c8000)
libgfortran.so.1 => /usr/lib64/libgfortran.so.1 (0x2ab26a34c000)
libc.so.6 => /lib64/libc.so.6 (0x2ab26a5e3000)
/lib64/ld-linux-x86-64.so.2 (0x003b2ba0)
-bash-3.2$ ldd /usr/lib64/liblapack.so
linux-vdso.so.1 => (0x7fffe97fd000)
libblas.so.3 => /usr/lib64/libblas.so.3 (0x2b6438d75000)
libm.so.6 => /lib64/libm.so.6 (0x2b6438fca000)
libgfortran.so.1 => /usr/lib64/libgfortran.so.1 (0x2b643924d000)
libc.so.6 => /lib64/libc.so.6 (0x2b64394e4000)
/lib64/ld-linux-x86-64.so.2 (0x003b2ba0)
The sysadmins have provided a gcc 4.4.0 compiler that I access using
symlinks on my $PATH:
-bash-3.2$ which gcc g++ gfortran
~/local/bin/gcc
~/local/bin/g++
~/local/bin/gfortran
-bash-3.2$ ls -l ~/local/bin
lrwxrwxrwx 1 rowen astro 14 Oct 28 2010 g++ -> /usr/bin/g++44
lrwxrwxrwx 1 rowen astro 14 Oct 28 2010 gcc -> /usr/bin/gcc44
lrwxrwxrwx 1 rowen astro 19 Dec 5 16:40 gfortran -> /usr/bin/gfortran44
-bash-3.2$ gfortran --version
GNU Fortran (GCC) 4.4.0 20090514 (Red Hat 4.4.0-6)
Copyright (C) 2009 Free Software Foundation, Inc.
For this log I used a home-bulit python 2.6.5 that is widely used.
However, I've tried it with other builds of python that are on our
system, as well, with no better success (including a Python 2.7.2).
-bash-3.2$ which python
/astro/apps/pkg/python64/bin/python
-bash-3.2$ python
Python 2.6.5 (r265:79063, Aug 4 2010, 11:27:53)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
numpy seems to see gfortran when it builds:
-bash-3.2$ python setup.py build --fcompiler=gnu95
Running from numpy source directory.non-existing path in
'numpy/distutils': 'site.cfg'
F2PY Version 2
blas_opt_info:
blas_mkl_info:
... NOT AVAILABLE
atlas_blas_threads_info:
... NOT AVAILABLE
atlas_blas_info:
... NOT AVAILABLE
blas_info:
... FOUND:
libraries = ['blas']
library_dirs = ['/usr/lib64']
language = f77
FOUND:
libraries = ['blas']
library_dirs = ['/usr/lib64']
define_macros = [('NO_ATLAS_INFO', 1)]
language = f77
lapack_opt_info:
lapack_mkl_info:
mkl_info:
... NOT AVAILABLE
NOT AVAILABLE
atlas_threads_info:
... NOT AVAILABLE
atlas_info:
... NOT AVAILABLE
/astro/users/rowen/build/numpy-1.6.1/numpy/distutils/system_info.py:1330:
UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
warnings.warn(AtlasNotFoundError.__doc__)
lapack_info:
libraries lapack not found in
/astro/apps/lsst_w12_sl5/Linux64/external/python/2.7.2+2/lib
... FOUND:
libraries = ['lapack']
library_dirs = ['/usr/lib64']
language = f77
FOUND:
libraries = ['lapack', 'blas']
library_dirs = ['/usr/lib64']
define_macros = [('NO_ATLAS_INFO', 1)]
language = f77
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands
--compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands
--fcompiler options
running build_src
build_src
building py_modules sources
creating build
creating build/src.linux-x86_64-2.7
creating build/src.linux-x86_64-2.7/numpy
creating build/src.linux-x86_64-2.7/numpy/distutils
building library "npymath" sources
customize Gnu95FCompiler
Found executable /astro/users/rowen/local/bin/gfortran
# I install it in an out-of-the-way location just so I can test it
-bash-3.2$ python setup.py install --home=~/local
...
-bash-3.2$ cd
-bash-3.2$ python
>>> import numpy
>>> numpy.__path__
['/astro/users/rowen/local/lib/python/numpy']
>>> numpy.test()
Running unit tests for numpy
NumPy version 1.6.1
NumPy is installed in /astro/users/rowen/local/lib/python/numpy
Python version 2.6.5 (r265:79063, Aug 4 2010, 11:27:53) [GCC 4.1.2
20080704 (Red Hat 4.1.2-46)]
nose version 0.11.4
==
FAIL: test_lapack (test_build.TestF77Mismatch)
--
Traceback (most recent call last):
File
"/astro/users/rowen/local/lib/python/numpy/testing/decorators.py", line
146, in skipper_func
return f(*args, **kwargs)
File
"/astro/users/rowen/local/lib/python/numpy/linalg/tests/test_build.py",
line 50, in test_lapack
information.""")
AssertionError: Both g77 and gfortran runtimes linked in lapack_lite !
This is
Re: [Numpy-discussion] Long-standing issue with using numpy in embedded CPython
Hi Armando, No comment on the Java thing ;-) However, http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.18/contents.html is more up-to-date and we are on github too: https://github.com/DiamondLightSource Peter On 9 December 2011 13:05, Vicente Sole wrote: > Quoting Robert Kern : > >> On Fri, Dec 9, 2011 at 11:00, Yang Zhang wrote: >> >>> Thanks for the clarification. Alas. So is there no simple workaround >>> to making numpy work in environments such as Jepp? >> >> I don't think so, no. >> > > It is far from being an optimal solution (in fact I dislike it) but > there is a couple of research facilities that like the python > interpreter, they like numpy, but prefer to use java for all their > graphical interfaces. They have rewritten part of numpy in java in > order to use it from Jython. > > http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.16/scisoftpy.html > > > Armando > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Long-standing issue with using numpy in embedded CPython
Le 09/12/2011 15:00, Robert Kern a écrit : Using multiprocessing is fine. That starts up multiple interpreters in *different* processes. Yang is using a non-Python program that embeds the CPython interpreter and starts up multiple copies of it in the same process. Ok, now I think I understand. I was not aware it was possible to embed multiple CPython instances into one process. So I guess IPython's multiprocessing infrastructure I once briefly considered is also safe since it runs multiple kernels. I'm relieved to hear that ! Thank you very much for the explanation. Pierre ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Long-standing issue with using numpy in embedded CPython
On Fri, Dec 9, 2011 at 13:18, Pierre Haessig wrote: > Le 09/12/2011 09:31, Robert Kern a écrit : >> We have some global state >> that we need to keep, and this gets interfered with in a multiple >> interpreter environment. > I recently got interested in multiprocessing computation with numpy and > now I get scare by your statement ! > Please don't tell me it is unsafe to launch multiple jobs (for instance > with multiprocressing's Pool.map) just doing some ndarray arithmetic ! > > That's why I'd like to understand better the issue raised by Yang. For > instance, what does exactly "multiple CPython interpreters" stands for ? Using multiprocessing is fine. That starts up multiple interpreters in *different* processes. Yang is using a non-Python program that embeds the CPython interpreter and starts up multiple copies of it in the same process. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Long-standing issue with using numpy in embedded CPython
Le 09/12/2011 09:31, Robert Kern a écrit : > We have some global state > that we need to keep, and this gets interfered with in a multiple > interpreter environment. I recently got interested in multiprocessing computation with numpy and now I get scare by your statement ! Please don't tell me it is unsafe to launch multiple jobs (for instance with multiprocressing's Pool.map) just doing some ndarray arithmetic ! That's why I'd like to understand better the issue raised by Yang. For instance, what does exactly "multiple CPython interpreters" stands for ? Best, Pierre ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Long-standing issue with using numpy in embedded CPython
Quoting Robert Kern : > On Fri, Dec 9, 2011 at 11:00, Yang Zhang wrote: > >> Thanks for the clarification. Alas. So is there no simple workaround >> to making numpy work in environments such as Jepp? > > I don't think so, no. > It is far from being an optimal solution (in fact I dislike it) but there is a couple of research facilities that like the python interpreter, they like numpy, but prefer to use java for all their graphical interfaces. They have rewritten part of numpy in java in order to use it from Jython. http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.16/scisoftpy.html Armando ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Long-standing issue with using numpy in embedded CPython
On Fri, Dec 9, 2011 at 11:00, Yang Zhang wrote: > Thanks for the clarification. Alas. So is there no simple workaround > to making numpy work in environments such as Jepp? I don't think so, no. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Long-standing issue with using numpy in embedded CPython
On Fri, Dec 9, 2011 at 12:31 AM, Robert Kern wrote: > On Thu, Dec 8, 2011 at 23:15, Yang Zhang wrote: >> On Tue, Oct 4, 2011 at 12:05 PM, Yang Zhang wrote: >>> On Tue, Oct 4, 2011 at 1:28 AM, Robin wrote: On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang wrote: > It turns out that there's a long-standing problem in numpy that > prevents it from being used in embedded CPython environments: Just wanted to make the point for reference that in general Numpy does work fine in (non-threaded) embedded CPython situations, see for example pymex [1] which embeds Python + Numpy in a Matlab mex file and works really well. This seems to a be a problem specific to Jepp. Just wanted to mention it in case it puts someone off trying something unnecessarily in the future. >>> >>> My (second-hand) understanding is that this is a problem with having >>> multiple CPython interpreters, which both Jepp and numpy utilize, >>> incompatibly - is that right? I.e., if either one were restricted to >>> using a single CPython interpreter, we wouldn't see this problem? >>> >>> I'm curious how to disable threads in numpy (not an ideal solution). >>> Googling seems to point me to setting NPY_ALLOW_THREADS to >>> 0somewhere. >> >> Anyone? > > numpy does not use multiple interpreters. The threading options have > nothing to do with multiple interpreters, and will not let you use > multiple CPython interpreters in your application. The problem is that > Python does not have good isolation between multiple interpreters for > extension modules. Many extension modules happen to work in this > environment, but numpy is not one of them. We have some global state > that we need to keep, and this gets interfered with in a multiple > interpreter environment. Thanks for the clarification. Alas. So is there no simple workaround to making numpy work in environments such as Jepp? > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Yang Zhang http://yz.mit.edu/ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Long-standing issue with using numpy in embedded CPython
On Thu, Dec 8, 2011 at 23:15, Yang Zhang wrote: > On Tue, Oct 4, 2011 at 12:05 PM, Yang Zhang wrote: >> On Tue, Oct 4, 2011 at 1:28 AM, Robin wrote: >>> On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang wrote: It turns out that there's a long-standing problem in numpy that prevents it from being used in embedded CPython environments: >>> >>> Just wanted to make the point for reference that in general Numpy does >>> work fine in (non-threaded) embedded CPython situations, see for >>> example pymex [1] which embeds Python + Numpy in a Matlab mex file and >>> works really well. >>> >>> This seems to a be a problem specific to Jepp. >>> >>> Just wanted to mention it in case it puts someone off trying something >>> unnecessarily in the future. >> >> My (second-hand) understanding is that this is a problem with having >> multiple CPython interpreters, which both Jepp and numpy utilize, >> incompatibly - is that right? I.e., if either one were restricted to >> using a single CPython interpreter, we wouldn't see this problem? >> >> I'm curious how to disable threads in numpy (not an ideal solution). >> Googling seems to point me to setting NPY_ALLOW_THREADS to >> 0somewhere. > > Anyone? numpy does not use multiple interpreters. The threading options have nothing to do with multiple interpreters, and will not let you use multiple CPython interpreters in your application. The problem is that Python does not have good isolation between multiple interpreters for extension modules. Many extension modules happen to work in this environment, but numpy is not one of them. We have some global state that we need to keep, and this gets interfered with in a multiple interpreter environment. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
