[Numpy-discussion] checking for array type in C extension
Hello,
I do have problems for checking for integer array types in a C extension
under Windows. I've put together a small example to illustrate the
problem:
h...@pc090498 ~/pytest $ cat tst.c
#include stdio.h
#include Python.h
#include numpy/arrayobject.h
#define TRY(E) (E) ; if(PyErr_Occurred()) {fprintf(stderr, %s:%d\n, __FILE__,
__LINE__); return NULL;}
static PyObject*
inttest_cfunc (PyObject *dummy, PyObject *args) {
PyArrayObject *array;
TRY(PyArg_ParseTuple(args, O!:inttest, PyArray_Type, array));
fprintf(stderr, PyArray_TYPE(array): %ld; NPY_INT: %ld\n,
PyArray_TYPE(array), NPY_INT);
if (PyArray_TYPE(array) == NPY_INT) { fprintf(stderr, found NPY_INT\n);
} else { fprintf(stderr, NPY_INT not found\n);
}
Py_RETURN_NONE;
}
static PyMethodDef mymethods[] = {
{ inttestfunc, inttest_cfunc, METH_VARARGS, Doc string},
{NULL, NULL, 0, NULL} /* Sentinel */
};
PyMODINIT_FUNC
inittst(void) {
(void)Py_InitModule(tst, mymethods);
import_array();
}
h...@pc090498 ~/pytest $ python setup.py build
running build
...
h...@pc090498 ~/pytest $ cat xx.py
import tst, sys
import numpy as np
print sys.stderr, np.__version__, np.__path__
tst.inttestfunc(np.array((1,2),dtype=np.int))
tst.inttestfunc(np.array((1,2),dtype=np.int8))
tst.inttestfunc(np.array((1,2),dtype=np.int16))
tst.inttestfunc(np.array((1,2),dtype=np.int32))
tst.inttestfunc(np.array((1,2),dtype=np.int64))
h...@pc090498 ~/pytest $ PYTHONPATH=build/lib.win32-2.5/ python xx.py
1.4.1 ['C:\\Python25\\lib\\site-packages\\numpy']
PyArray_TYPE(array): 7; NPY_INT: 5
NPY_INT not found
PyArray_TYPE(array): 1; NPY_INT: 5
NPY_INT not found
PyArray_TYPE(array): 3; NPY_INT: 5
NPY_INT not found
PyArray_TYPE(array): 7; NPY_INT: 5
NPY_INT not found
PyArray_TYPE(array): 9; NPY_INT: 5
NPY_INT not found
NPY_INT32 is 7, but shouldn't NPY_INT correspond to numpy.int. And what
kind of int is NPY_INT in this case?
Kind regards
Berthold Höllmann
--
Germanischer Lloyd AG
Berthold Höllmann
Project Engineer, CAE Development
Brooktorkai 18
20457 Hamburg
Germany
Phone: +49(0)40 36149-7374
Fax: +49(0)40 36149-7320
e-mail: berthold.hoellm...@gl-group.com
Internet: http://www.gl-group.com
This e-mail and any attachment thereto may contain confidential information
and/or information protected by intellectual property rights for the exclusive
attention of the intended addressees named above. Any access of third parties
to this e-mail is unauthorised. Any use of this e-mail by unintended recipients
such as total or partial copying, distribution, disclosure etc. is prohibited
and may be unlawful. When addressed to our clients the content of this e-mail
is subject to the General Terms and Conditions of GL's Group of Companies
applicable at the date of this e-mail.
If you have received this e-mail in error, please notify the sender either by
telephone or by e-mail and delete the material from any computer.
GL's Group of Companies does not warrant and/or guarantee that this message at
the moment of receipt is authentic, correct and its communication free of
errors, interruption etc.
Germanischer Lloyd AG, 31393 AG HH, Hamburg, Vorstand: Dr. Hermann J. Klein,
Dr. Joachim Segatz, Pekka Paasivaara, Vorsitzender des Aufsichtsrats: Dr.
Wolfgang Peiner
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] reading big-endian uint16 into array on little-endian machine
Den 17.06.2010 16:29, skrev greg whittier:
I have files (from an external source) that contain ~10 GB of
big-endian uint16's that I need to read into a series of arrays. What
I'm doing now is
import numpy as np
import struct
fd = open('file.raw', 'rb')
for n in range(1)
count = 1024*1024
a = np.array([struct.unpack('H', fd.read(2)) for i in range(count)])
# do something with a
It doesn't seem very efficient to call struct.unpack one element at a
time, but struct doesn't have an unpack_farray version like xdrlib
does. I also thought of using the array module and .byteswap() but
the help says it only work on 4 and 8 byte arrays.
Any ideas?
t's just a matter of swapping the bytes:
arr = 1D array of uint16
bytes = arr.view(dtype=np.uint8)
tmp = bytes[::2].copy()
bytes[::2] = bytes[1::2]
bytes[1::2] = tmp
Or like this:
arr = 1D array of uint16
arr = (arr 8) | (arr 8)
The latter generates three temporary arrays, the first generates one.
You can avoid this with C:
__declspec(dllexport)
void byteswap(unsigned short *arr, int n)
{
for (int i=0; in; i++) {
arr[i] = (arr[i] 8) | (arr[i] 8);
}
}
Sturla
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] checking for array type in C extension
pe, 2010-06-18 kello 12:49 +0200, Berthold Hoellmann kirjoitti: [clip] tst.inttestfunc(np.array((1,2),dtype=np.int)) tst.inttestfunc(np.array((1,2),dtype=np.int8)) tst.inttestfunc(np.array((1,2),dtype=np.int16)) tst.inttestfunc(np.array((1,2),dtype=np.int32)) tst.inttestfunc(np.array((1,2),dtype=np.int64)) h...@pc090498 ~/pytest $ PYTHONPATH=build/lib.win32-2.5/ python xx.py 1.4.1 ['C:\\Python25\\lib\\site-packages\\numpy'] PyArray_TYPE(array): 7; NPY_INT: 5 NPY_INT not found PyArray_TYPE(array): 1; NPY_INT: 5 NPY_INT not found PyArray_TYPE(array): 3; NPY_INT: 5 NPY_INT not found PyArray_TYPE(array): 7; NPY_INT: 5 NPY_INT not found PyArray_TYPE(array): 9; NPY_INT: 5 NPY_INT not found NPY_INT32 is 7, but shouldn't NPY_INT correspond to numpy.int. And what kind of int is NPY_INT in this case? I think the explanation is the following: - NPY_INT is a virtual type that is either int32 or int64, depending on the native platform size. - It has its own type code, distinct from NPY_SHORT (= 3 = int32) and NPY_LONG (= 7 = int64). - But the type specifier is replaced either by NPY_SHORT or NPY_LONG on array creation, so no array is of this dtype. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] reading big-endian uint16 into array on little-endian machine
On Fri, Jun 18, 2010 at 6:15 AM, Sturla Molden stu...@molden.no wrote:
Den 17.06.2010 16:29, skrev greg whittier:
I have files (from an external source) that contain ~10 GB of
big-endian uint16's that I need to read into a series of arrays. What
I'm doing now is
import numpy as np
import struct
fd = open('file.raw', 'rb')
for n in range(1)
count = 1024*1024
a = np.array([struct.unpack('H', fd.read(2)) for i in
range(count)])
# do something with a
It doesn't seem very efficient to call struct.unpack one element at a
time, but struct doesn't have an unpack_farray version like xdrlib
does. I also thought of using the array module and .byteswap() but
the help says it only work on 4 and 8 byte arrays.
Any ideas?
t's just a matter of swapping the bytes:
arr = 1D array of uint16
bytes = arr.view(dtype=np.uint8)
tmp = bytes[::2].copy()
bytes[::2] = bytes[1::2]
bytes[1::2] = tmp
Or like this:
arr = 1D array of uint16
arr = (arr 8) | (arr 8)
The latter generates three temporary arrays, the first generates one.
You can avoid this with C:
__declspec(dllexport)
void byteswap(unsigned short *arr, int n)
{
for (int i=0; in; i++) {
arr[i] = (arr[i] 8) | (arr[i] 8);
}
}
Sturla
Just for my own knowledge, would Robert's suggestion of using 'i2' as the
dtype be considered the best solution, mostly because of its simplicity,
but also because it does not assume the endian-ness of the host computer?
Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] checking for array type in C extension
Pauli Virtanen p...@iki.fi writes: pe, 2010-06-18 kello 12:49 +0200, Berthold Hoellmann kirjoitti: [clip] tst.inttestfunc(np.array((1,2),dtype=np.int)) tst.inttestfunc(np.array((1,2),dtype=np.int8)) tst.inttestfunc(np.array((1,2),dtype=np.int16)) tst.inttestfunc(np.array((1,2),dtype=np.int32)) tst.inttestfunc(np.array((1,2),dtype=np.int64)) h...@pc090498 ~/pytest $ PYTHONPATH=build/lib.win32-2.5/ python xx.py 1.4.1 ['C:\\Python25\\lib\\site-packages\\numpy'] PyArray_TYPE(array): 7; NPY_INT: 5 NPY_INT not found PyArray_TYPE(array): 1; NPY_INT: 5 NPY_INT not found PyArray_TYPE(array): 3; NPY_INT: 5 NPY_INT not found PyArray_TYPE(array): 7; NPY_INT: 5 NPY_INT not found PyArray_TYPE(array): 9; NPY_INT: 5 NPY_INT not found NPY_INT32 is 7, but shouldn't NPY_INT correspond to numpy.int. And what kind of int is NPY_INT in this case? I think the explanation is the following: - NPY_INT is a virtual type that is either int32 or int64, depending on the native platform size. - It has its own type code, distinct from NPY_SHORT (= 3 = int32) and NPY_LONG (= 7 = int64). - But the type specifier is replaced either by NPY_SHORT or NPY_LONG on array creation, so no array is of this dtype. Pauli The documentation (i refere to NumPy User Guide, Release 1.5.0.dev8106) claims that numpy.int is platform int and that NPY_INT is a C based name, thus referring to int also. So if I want to use int platform independent in my extension module, what do I do to check whether the correct data is proivided. I want to avoid one of the casting routines, such as PyArray_FROM_OTF or PyArray_FromAny? Do I have to use PyArray_ITEMSIZE? But PyArray_ITEMSIZE indicates that numpy.int are not platform integers. It returns 8 on Linux64, where sizeof(int)==4. So is this a bug in 1.4.1, in the 1.5 beta documentation or a misunderstanding on my side? Kind regards Berthold Höllmann -- Germanischer Lloyd AG Berthold Höllmann Project Engineer, CAE Development Brooktorkai 18 20457 Hamburg Germany Phone: +49(0)40 36149-7374 Fax: +49(0)40 36149-7320 e-mail: berthold.hoellm...@gl-group.com Internet: http://www.gl-group.com ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] reading big-endian uint16 into array on little-endian machine
On Fri, Jun 18, 2010 at 09:39, Benjamin Root ben.v.r...@gmail.com wrote: Just for my own knowledge, would Robert's suggestion of using 'i2' as the dtype be considered the best solution, mostly because of its simplicity, but also because it does not assume the endian-ness of the host computer? It does come with the tradeoff that math will be a little bit slower on it. A quick .astype(np.uint16) will fix that. That said, the cost of creating the memory for the new native array will probably wipe away those gains unless if the data is reused a number of times for calculations. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] checking for array type in C extension
On Fri, Jun 18, 2010 at 09:43, Berthold Hoellmann berthold.hoellm...@gl-group.com wrote: Pauli Virtanen p...@iki.fi writes: pe, 2010-06-18 kello 12:49 +0200, Berthold Hoellmann kirjoitti: [clip] tst.inttestfunc(np.array((1,2),dtype=np.int)) tst.inttestfunc(np.array((1,2),dtype=np.int8)) tst.inttestfunc(np.array((1,2),dtype=np.int16)) tst.inttestfunc(np.array((1,2),dtype=np.int32)) tst.inttestfunc(np.array((1,2),dtype=np.int64)) h...@pc090498 ~/pytest $ PYTHONPATH=build/lib.win32-2.5/ python xx.py 1.4.1 ['C:\\Python25\\lib\\site-packages\\numpy'] PyArray_TYPE(array): 7; NPY_INT: 5 NPY_INT not found PyArray_TYPE(array): 1; NPY_INT: 5 NPY_INT not found PyArray_TYPE(array): 3; NPY_INT: 5 NPY_INT not found PyArray_TYPE(array): 7; NPY_INT: 5 NPY_INT not found PyArray_TYPE(array): 9; NPY_INT: 5 NPY_INT not found NPY_INT32 is 7, but shouldn't NPY_INT correspond to numpy.int. And what kind of int is NPY_INT in this case? I think the explanation is the following: - NPY_INT is a virtual type that is either int32 or int64, depending on the native platform size. - It has its own type code, distinct from NPY_SHORT (= 3 = int32) and NPY_LONG (= 7 = int64). - But the type specifier is replaced either by NPY_SHORT or NPY_LONG on array creation, so no array is of this dtype. Pauli The documentation (i refere to NumPy User Guide, Release 1.5.0.dev8106) claims that numpy.int is platform int and that NPY_INT is a C based name, thus referring to int also. It is referring to the Python int type, not the C int type. Python ints are actually C longs underneath. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] checking for array type in C extension
pe, 2010-06-18 kello 16:43 +0200, Berthold Hoellmann kirjoitti: [clip] The documentation (i refere to NumPy User Guide, Release 1.5.0.dev8106) claims that numpy.int is platform int and that NPY_INT is a C based name, thus referring to int also. So if I want to use int platform independent in my extension module, what do I do to check whether the correct data is provided. I want to avoid one of the casting routines, such as PyArray_FROM_OTF or PyArray_FromAny? Do I have to use PyArray_ITEMSIZE? But PyArray_ITEMSIZE indicates that numpy.int are not platform integers. It returns 8 on Linux64, where sizeof(int)==4. So is this a bug in 1.4.1, in the 1.5 beta documentation or a misunderstanding on my side? I guess the documentation needs clarification, as numpy.int is not actually the platform-size integer, but the Python-size integer. The platform-size integer corresponding to C int is IIRC numpy.intc, which should result to the same sizes as NPY_INT. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] checking for array type in C extension
Robert Kern robert.k...@gmail.com writes:
On Fri, Jun 18, 2010 at 09:43, Berthold Hoellmann
berthold.hoellm...@gl-group.com wrote:
Pauli Virtanen p...@iki.fi writes:
pe, 2010-06-18 kello 12:49 +0200, Berthold Hoellmann kirjoitti:
[clip]
tst.inttestfunc(np.array((1,2),dtype=np.int))
tst.inttestfunc(np.array((1,2),dtype=np.int8))
tst.inttestfunc(np.array((1,2),dtype=np.int16))
tst.inttestfunc(np.array((1,2),dtype=np.int32))
tst.inttestfunc(np.array((1,2),dtype=np.int64))
h...@pc090498 ~/pytest $ PYTHONPATH=build/lib.win32-2.5/ python xx.py
1.4.1 ['C:\\Python25\\lib\\site-packages\\numpy']
PyArray_TYPE(array): 7; NPY_INT: 5
NPY_INT not found
PyArray_TYPE(array): 1; NPY_INT: 5
NPY_INT not found
PyArray_TYPE(array): 3; NPY_INT: 5
NPY_INT not found
PyArray_TYPE(array): 7; NPY_INT: 5
NPY_INT not found
PyArray_TYPE(array): 9; NPY_INT: 5
NPY_INT not found
NPY_INT32 is 7, but shouldn't NPY_INT correspond to numpy.int. And what
kind of int is NPY_INT in this case?
I think the explanation is the following:
- NPY_INT is a virtual type that is either int32 or int64, depending on
the native platform size.
- It has its own type code, distinct from NPY_SHORT (= 3 = int32) and
NPY_LONG (= 7 = int64).
- But the type specifier is replaced either by NPY_SHORT or NPY_LONG
on array creation, so no array is of this dtype.
Pauli
The documentation (i refere to NumPy User Guide, Release 1.5.0.dev8106)
claims that numpy.int is platform int and that NPY_INT is a C based
name, thus referring to int also.
It is referring to the Python int type, not the C int type. Python
ints are actually C longs underneath.
So it should be called Platform python int instead of Platform int
in the documetation.
I now use 'sizeof' and 'c_int' from ctypes tp construct the dtype
description:
dtype('i%d' % sizeof(c_int))
Is there a way to avoid the usage of ctypes by using information
provided by numpy?
Kind regards
Berthold Höllmann
--
Germanischer Lloyd AG
Berthold Höllmann
Project Engineer, CAE Development
Brooktorkai 18
20457 Hamburg
Germany
Phone: +49(0)40 36149-7374
Fax: +49(0)40 36149-7320
e-mail: berthold.hoellm...@gl-group.com
Internet: http://www.gl-group.com
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] checking for array type in C extension
On Fri, Jun 18, 2010 at 10:22, Berthold Hoellmann
berthold.hoellm...@gl-group.com wrote:
I now use 'sizeof' and 'c_int' from ctypes tp construct the dtype
description:
dtype('i%d' % sizeof(c_int))
Is there a way to avoid the usage of ctypes by using information
provided by numpy?
np.intc
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth.
-- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] SciPy docs marathon: a little more info
On Mon, Jun 14, 2010 at 2:05 AM, David Goldsmith d.l.goldsm...@gmail.comwrote: Hi, all! The scipy doc marathon has gotten off to a very slow start this summer. We are producing less than 1000 words a week, perhaps because many universities are still finishing up spring classes. So, this is a second appeal to everyone to pitch in and help get scipy documented so that it's easy to learn how to use it. Because some of the packages are quite specialized, we need both regular contributors to write lots of pages, and some people experienced in using each module (and the mathematics behind the software) to make sure we don't water it down or make it wrong in the process. If you can help, please, now is the time to step forward. Thanks! On behalf of Joe and myself, David Goldsmith Olympia, WA (Apparently this didn't go through the first time.) OK, a few people have come forward - thanks! Let me enumerate the categories that still have no declared volunteer writer-editors (all categories are in need of leaders): Max. Entropy, Misc., Image Manip. (Milestone 6) Signal processing (Milestone 8) Sparse Matrices (Milestone 9) Spatial Algorithms., Special funcs. (Milestone 10) C/C++ Integration (Milestone 13) As for the rest, only Interpolation (Milestone 3) has more than one person (but I'm one of the two), and I'm the only person on four others. So, hopefully, knowing specifically which areas are in dire need will inspire people skilled in those areas to sign up. Thanks for your time and help, DG PS: For your convenience, here's the link to the scipy Milestoneshttp://docs.scipy.org/scipy/Milestones/page. (Note that the Milestones link at the top of each Wiki page links, incorrectly in the case of the SciPy pages, to the NumPy Milestones page, which we are not actively working on in this Marathon; this is a known, reported bug in the Wiki program.) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Beginer question about Boost Python Numeric
Hi,
I am new to boost python, thanks for the list, and hopefully, I will
find some one to help me.
I am adding the boost python example:
#include boost/python/numeric.hpp
#include boost/python/tuple.hpp
// sets the first element in a 2d numeric array
void set_first_element(::boost::python::numeric::array y, float value)
{
y[::boost::python::make_tuple(0,0)] = value;
}
into quickstart project, and I belive it compiles and run without this function.
When I add this function, in python, I did:
a=array([1.])
set_first_element(a,3.)
and come up with the following error message:
Boost.Python.ArgumentError: Python argument types in
extending.set_first_element(numpy.ndarray, float)
did not match C++ signature:
set_first_element(boost::python::numeric::array {lvalue}, float)
Could any one please help me with what's wrong?
Thanks in advance!
Yong
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
