Re: [Numpy-discussion] help using np.correlate to produce correlograms.
As a side note, I've still in mind the proposal I made back in 2013 to make np.correlate faster http://numpy-discussion.10968.n7.nabble.com/another-discussion-on-numpy-correlate-and-convolution-td32925.html The basic idea is to enable the user to select the exact range of lags he wants. Unfortunately I didn't take the time to go further than the specification above... best, Pierre Le 10/12/2014 21:40, Jose Guzman a écrit : > Dear Pierre, > > thank you very much for your time to correct my notebook and to point me > in the direction of my wrong lag estimation. It has been very useful! > > Best > > Jose > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] help using np.correlate to produce correlograms.
I think it is a good time to discuss/implement further correlate improvements. I kind of favor the mode=(tuple of integers) api for your proposed change. Concerning the C-API we probably need to add a new wrapper function but thats ok, the C-API does not need to be as nice as the python API as it has far less and typically more experienced users. I also think its time we can remove the old_behavior flag which has been there since 1.4. Are there objections to that? Also on a side note, in 1.10 np.convolve/correlate has been significantly speed up if one of the sequences is less than 12 elements long. On 12/11/2014 09:54 AM, Pierre Haessig wrote: > As a side note, I've still in mind the proposal I made back in 2013 to > make np.correlate faster > > http://numpy-discussion.10968.n7.nabble.com/another-discussion-on-numpy-correlate-and-convolution-td32925.html > > The basic idea is to enable the user to select the exact range of lags > he wants. Unfortunately I didn't take the time to go further than the > specification above... > > best, > Pierre > > > Le 10/12/2014 21:40, Jose Guzman a écrit : >> Dear Pierre, >> >> thank you very much for your time to correct my notebook and to point me >> in the direction of my wrong lag estimation. It has been very useful! >> >> Best >> >> Jose >> > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] help using np.correlate to produce correlograms.
Le 11/12/2014 11:19, Julian Taylor a écrit : > Also on a side note, in 1.10 np.convolve/correlate has been > significantly speed up if one of the sequences is less than 12 elements Interesting! What is the origin of this speed up, and why a magic number 12? -- Pierre ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Add a function to broadcast arrays to a given shape to numpy's stride_tricks?
Le 11/12/2014 01:00, Nathaniel Smith a écrit : > Seems like a useful addition to me -- I've definitely wanted this in > the past. I agree with Stephan that reshape() might not be the best > place, though; I wouldn't think to look for it there. > > Two API ideas, which are not mutually exclusive: > > [...] > > 2) Add a broadcast_to(arr, shape) function, which broadcasts the array > to exactly the shape given, or else errors out if this is not > possible. That's also possible. Then there could be a point in `reshape` docstring. Could this function be named `broadcast` instead of `broadcast_to` ? (in coherence with `reshape`) best, Pierre ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] help using np.correlate to produce correlograms.
On 12/11/2014 03:24 PM, Pierre Haessig wrote: > Le 11/12/2014 11:19, Julian Taylor a écrit : >> Also on a side note, in 1.10 np.convolve/correlate has been >> significantly speed up if one of the sequences is less than 12 elements > Interesting! What is the origin of this speed up, and why a magic number 12? > previously numpy called dot for the convolution part, this is fine for large convolutions as dot goes out to BLAS which is superfast. For small convolutions unfortunately it is terrible as generic dot in BLAS libraries have enormous overheads they only amortize on large data. So one part was computing the dot in a simple numpy internal loop if the data is small. The second part is the number of registers typical machines have, e.g. amd64 has 16 floating point registers. If you can put all elements of a convolution kernel into these registers you save reloading them from stack on each iteration. 11 is the largest number I could reliably use without the compiler spilling them to the stack. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Add a function to broadcast arrays to a given shape to numpy's stride_tricks?
On 11 Dec 2014 14:31, "Pierre Haessig" wrote: > > > Le 11/12/2014 01:00, Nathaniel Smith a écrit : > > Seems like a useful addition to me -- I've definitely wanted this in > > the past. I agree with Stephan that reshape() might not be the best > > place, though; I wouldn't think to look for it there. > > > > Two API ideas, which are not mutually exclusive: > > > > [...] > > > > 2) Add a broadcast_to(arr, shape) function, which broadcasts the array > > to exactly the shape given, or else errors out if this is not > > possible. > That's also possible. Then there could be a point in `reshape` docstring. > > Could this function be named `broadcast` instead of `broadcast_to` ? > (in coherence with `reshape`) It could, but then there wouldn't be much to distinguish it from broadcast_arrays. Broadcasting is generally a symmetric operation - see broadcast_arrays or arr1 + arr2. So the 'to' is there to give a clue that this function is not symmetric, and rather has a specific goal in mind. -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] help using np.correlate to produce correlograms.
Le 11/12/2014 15:39, Julian Taylor a écrit : > previously numpy called dot for the convolution part, this is fine for > large convolutions as dot goes out to BLAS which is superfast. > For small convolutions unfortunately it is terrible as generic dot in > BLAS libraries have enormous overheads they only amortize on large data. > So one part was computing the dot in a simple numpy internal loop if the > data is small. > > The second part is the number of registers typical machines have, e.g. > amd64 has 16 floating point registers. If you can put all elements of a > convolution kernel into these registers you save reloading them from > stack on each iteration. > 11 is the largest number I could reliably use without the compiler > spilling them to the stack. Thanks Julian! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FFTS for numpy's FFTs (was: Re: Choosing between NumPy and SciPy functions)
On Tue, Oct 28, 2014 at 5:28 AM, Nathaniel Smith wrote: > On 28 Oct 2014 04:07, "Matthew Brett" wrote: > > > > Hi, > > > > On Mon, Oct 27, 2014 at 8:07 PM, Sturla Molden > wrote: > > > Sturla Molden wrote: > > > > > >> If we really need a > > >> kick-ass fast FFT we need to go to libraries like FFTW, Intel MKL or > > >> Apple's Accelerate Framework, > > > > > > I should perhaps also mention FFTS here, which claim to be faster than > FFTW > > > and has a BSD licence: > > > > > > http://anthonix.com/ffts/index.html > > > > Nice. And a funny New Zealand name too. > > > > Is this an option for us? Aren't we a little behind the performance > > curve on FFT after we lost FFTW? > > It's definitely attractive. Some potential issues that might need dealing > with, based on a quick skim: > > - seems to have a hard requirement for a processor supporting SSE, AVX, or > NEON. No fallback for old CPUs or other architectures. (I'm not even sure > whether it has x86-32 support.) > > - no runtime CPU detection, e.g. SSE vs AVX appears to be a compile time > decision > > - not sure if it can handle non-power-of-two problems at all, or at all > efficiently. (FFTPACK isn't great here either but major regressions would > be bad.) > > - not sure if it supports all the modes we care about (e.g. rfft) > > This stuff is all probably solveable though, so if someone has a hankering > to make numpy (or scipy) fft dramatically faster then you should get in > touch with the author and see what they think. > > -n > I recently became aware of another C-library for doing FFTs (and other things): https://github.com/arrayfire/arrayfire They claim to have comparable FFT performance to MKL when run on a CPU (they also support running on the GPU but that is probably outside the scope of numpy or scipy). It used to be proprietary but now it is under a BSD-3-Clause license. It seems it supports non-power-of-2 FFT operations as well (although those are slower). I don't know much beyond that, but it is probably worth looking in to. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Add a function to broadcast arrays to a given shape to numpy's stride_tricks?
On Thu, Dec 11, 2014 at 2:47 PM, Nathaniel Smith wrote: > > On 11 Dec 2014 14:31, "Pierre Haessig" wrote: > > > > > > Le 11/12/2014 01:00, Nathaniel Smith a écrit : > > > Seems like a useful addition to me -- I've definitely wanted this in > > > the past. I agree with Stephan that reshape() might not be the best > > > place, though; I wouldn't think to look for it there. > > > > > > Two API ideas, which are not mutually exclusive: > > > > > > [...] > > > > > > 2) Add a broadcast_to(arr, shape) function, which broadcasts the array > > > to exactly the shape given, or else errors out if this is not > > > possible. > > That's also possible. Then there could be a point in `reshape` docstring. > > > > Could this function be named `broadcast` instead of `broadcast_to` ? > > (in coherence with `reshape`) > > It could, but then there wouldn't be much to distinguish it from broadcast_arrays. Broadcasting is generally a symmetric operation - see broadcast_arrays or arr1 + arr2. So the 'to' is there to give a clue that this function is not symmetric, and rather has a specific goal in mind. And we already have a numpy.broadcast() function. http://docs.scipy.org/doc/numpy/reference/generated/numpy.broadcast.html -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FFTS for numpy's FFTs (was: Re: Choosing between NumPy and SciPy functions)
On Thu, Dec 11, 2014 at 10:41 AM, Todd wrote: > On Tue, Oct 28, 2014 at 5:28 AM, Nathaniel Smith wrote: > >> On 28 Oct 2014 04:07, "Matthew Brett" wrote: >> > >> > Hi, >> > >> > On Mon, Oct 27, 2014 at 8:07 PM, Sturla Molden >> wrote: >> > > Sturla Molden wrote: >> > > >> > >> If we really need a >> > >> kick-ass fast FFT we need to go to libraries like FFTW, Intel MKL or >> > >> Apple's Accelerate Framework, >> > > >> > > I should perhaps also mention FFTS here, which claim to be faster >> than FFTW >> > > and has a BSD licence: >> > > >> > > http://anthonix.com/ffts/index.html >> > >> > Nice. And a funny New Zealand name too. >> > >> > Is this an option for us? Aren't we a little behind the performance >> > curve on FFT after we lost FFTW? >> >> It's definitely attractive. Some potential issues that might need dealing >> with, based on a quick skim: >> >> - seems to have a hard requirement for a processor supporting SSE, AVX, >> or NEON. No fallback for old CPUs or other architectures. (I'm not even >> sure whether it has x86-32 support.) >> >> - no runtime CPU detection, e.g. SSE vs AVX appears to be a compile time >> decision >> >> - not sure if it can handle non-power-of-two problems at all, or at all >> efficiently. (FFTPACK isn't great here either but major regressions would >> be bad.) >> >> - not sure if it supports all the modes we care about (e.g. rfft) >> >> This stuff is all probably solveable though, so if someone has a >> hankering to make numpy (or scipy) fft dramatically faster then you should >> get in touch with the author and see what they think. >> >> -n >> > > I recently became aware of another C-library for doing FFTs (and other > things): > > https://github.com/arrayfire/arrayfire > > They claim to have comparable FFT performance to MKL when run on a CPU > (they also support running on the GPU but that is probably outside the > scope of numpy or scipy). It used to be proprietary but now it is under a > BSD-3-Clause license. It seems it supports non-power-of-2 FFT operations > as well (although those are slower). I don't know much beyond that, but it > is probably worth looking in > AFAICT the cpu backend is a FFTW wrapper. Eric ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FFTS for numpy's FFTs (was: Re: Choosing between NumPy and SciPy functions)
On Thu, Dec 11, 2014 at 3:53 PM, Eric Moore wrote: > > On Thu, Dec 11, 2014 at 10:41 AM, Todd wrote: >> I recently became aware of another C-library for doing FFTs (and other things): >> >> https://github.com/arrayfire/arrayfire >> >> They claim to have comparable FFT performance to MKL when run on a CPU (they also support running on the GPU but that is probably outside the scope of numpy or scipy). It used to be proprietary but now it is under a BSD-3-Clause license. It seems it supports non-power-of-2 FFT operations as well (although those are slower). I don't know much beyond that, but it is probably worth looking in > > AFAICT the cpu backend is a FFTW wrapper. Indeed. https://github.com/arrayfire/arrayfire/blob/devel/src/backend/cpu/fft.cpp#L16 -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Add a function to broadcast arrays to a given shape to numpy's stride_tricks?
Le 11/12/2014 16:52, Robert Kern a écrit : > > And we already have a numpy.broadcast() function. > > http://docs.scipy.org/doc/numpy/reference/generated/numpy.broadcast.html True, I once read the docstring of this function. but never used it though. Pierre ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FFTS for numpy's FFTs (was: Re: Choosing between NumPy and SciPy functions)
On Thu, Dec 11, 2014 at 4:55 PM, Robert Kern wrote: > On Thu, Dec 11, 2014 at 3:53 PM, Eric Moore > wrote: > > > > On Thu, Dec 11, 2014 at 10:41 AM, Todd wrote: > > >> I recently became aware of another C-library for doing FFTs (and other > things): > >> > >> https://github.com/arrayfire/arrayfire > >> > >> They claim to have comparable FFT performance to MKL when run on a CPU > (they also support running on the GPU but that is probably outside the > scope of numpy or scipy). It used to be proprietary but now it is under a > BSD-3-Clause license. It seems it supports non-power-of-2 FFT operations > as well (although those are slower). I don't know much beyond that, but it > is probably worth looking in > > > > AFAICT the cpu backend is a FFTW wrapper. > > Indeed. > https://github.com/arrayfire/arrayfire/blob/devel/src/backend/cpu/fft.cpp#L16 > Oh, nevermind then. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Add a function to broadcast arrays to a given shape to numpy's stride_tricks?
On Do, 2014-12-11 at 16:56 +0100, Pierre Haessig wrote: > Le 11/12/2014 16:52, Robert Kern a écrit : > > > > And we already have a numpy.broadcast() function. > > > > http://docs.scipy.org/doc/numpy/reference/generated/numpy.broadcast.html > True, I once read the docstring of this function. but never used it though. > I am not sure it is really the right thing for most things since it returns an old style iterator. On the other hand arrays with 0-strides need a bit more care (if we add this top level, one might have copy=True as a default or so?). Also because of that it is currently limited to NPY_MAXARGS (32). Personally, I would like to see this type of functionality implemented in C, and may be willing to help with it. This kind of code exists in enough places in numpy so it can be stolen pretty readily. One option would also be to have something like: np.common_shape(*arrays) np.broadcast_to(array, shape) # (though I would like many arrays too) and then broadcast_ar rays could be implemented in terms of these two. Just some thoughts, Sebastian > Pierre > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Add a function to broadcast arrays to a given shape to numpy's stride_tricks?
On Thu, Dec 11, 2014 at 4:17 PM, Sebastian Berg wrote: > > On Do, 2014-12-11 at 16:56 +0100, Pierre Haessig wrote: > > Le 11/12/2014 16:52, Robert Kern a écrit : > > > > > > And we already have a numpy.broadcast() function. > > > > > > http://docs.scipy.org/doc/numpy/reference/generated/numpy.broadcast.html > > True, I once read the docstring of this function. but never used it though. > > I am not sure it is really the right thing for most things since it > returns an old style iterator. Indeed. That's why I wrote broadcast_arrays(). > On the other hand arrays with 0-strides > need a bit more care (if we add this top level, one might have copy=True > as a default or so?). > Also because of that it is currently limited to NPY_MAXARGS (32). > Personally, I would like to see this type of functionality implemented > in C, and may be willing to help with it. This kind of code exists in > enough places in numpy so it can be stolen pretty readily. One option > would also be to have something like: > > np.common_shape(*arrays) > np.broadcast_to(array, shape) > # (though I would like many arrays too) > > and then broadcast_ar rays could be implemented in terms of these two. Why? What benefit does broadcast_arrays() get from being reimplemented in C? -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] error: no matching function for call to 'PyArray_DATA'
I am trying to patch pymol to avoid the warnings... /sw/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings] #warning "Using deprecated NumPy API, disable it by " \ ^ If I pass the compiler flag -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION, the warning becomes an error... # gcc -DNDEBUG -g -fwrapv -fwrapv -O3 -Wall -Wstrict-prototypes -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -D_PYMOL_LIBPNG -D_PYMOL_INLINE -D_PYMOL_OPENGL_SHADERS -D_PYMOL_CGO_DRAWARRAYS -D_PYMOL_CGO_DRAWBUFFERS -D_PYMOL_GL_CALLLISTS -D_PYMOL_VMD_PLUGINS -D_HAVE_LIBXML -D_PYMOL_FREETYPE -DNO_MMLIBS -D_CGO_DRAWARRAYS -DOPENGL_ES_2 -D_PYMOL_NUMPY -Iov/src -Ilayer0 -Ilayer1 -Ilayer2 -Ilayer3 -Ilayer4 -Ilayer5 -Imodules/cealign/src -Ibuild/generated -Icontrib/uiuc/plugins/include -Icontrib/uiuc/plugins/molfile_plugin/src -I/sw/lib/python2.7/site-packages/numpy/core/include -I/sw/include -I/sw/include/freetype2 -I/sw/include/libxml2 -I/usr/include -I/usr/include/libxml2 -I/usr/X11/include -I/usr/X11/include/freetype2 -I/sw/include/python2.7 -c layer0/Field.cpp -o build/temp.macosx-10.10-x86_64-2.7/layer0/Field.o -Werror=implicit-function-declaration -Werror=declaration-after-statement -Wno-write-strings -Wno-unused-function -Wno-empty-body -Wno-char-subscripts -ffast-math -funroll-loops -O3 -fcommon layer0/Field.cpp:76:14: error: no matching function for call to 'PyArray_DATA' memcpy(PyArray_DATA(result), field->data, field->size); ^~~~ /sw/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1460:1: note: candidate function not viable: no known conversion from 'PyObject *' (aka '_object *') to 'PyArrayObject *' (aka 'tagPyArrayObject *') for 1st argument PyArray_DATA(PyArrayObject *arr) ^ 1 error generated. What is the correct coding to eliminate this error? I have found some threads which seems to suggest that PyArray_DATA is still available in numpy 1.9 as an inline but I haven't found any examples of projects patching their code to convert to that usage. Jack ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] error: no matching function for call to 'PyArray_DATA'
Using
http://nbviewer.ipython.org/url/refreweb.phys.ethz.ch/hope/notebooks/native_cpp_gen.ipynb
as an example, I have found that the error using
-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION is suppressed with the
change...
--- pymol-1.7.4.0/layer0/Field.cpp.orig 2014-12-11 10:46:41.0 -0500
+++ pymol-1.7.4.0/layer0/Field.cpp 2014-12-11 11:52:23.0 -0500
@@ -73,7 +73,7 @@
if(copy) {
if((result = PyArray_SimpleNew(field->n_dim, dims, typenum)))
- memcpy(PyArray_DATA(result), field->data, field->size);
+ memcpy(PyArray_DATA((PyArrayObject *)result), field->data, field->size);
} else {
result = PyArray_SimpleNewFromData(field->n_dim, dims, typenum,
field->data);
}
On Thu, Dec 11, 2014 at 11:21 AM, Jack Howarth
wrote:
>I am trying to patch pymol to avoid the warnings...
>
> /sw/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2:
> warning: "Using deprecated NumPy API, disable it by " "#defining
> NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
> #warning "Using deprecated NumPy API, disable it by " \
> ^
>
> If I pass the compiler flag
> -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION, the warning becomes an
> error...
>
> # gcc -DNDEBUG -g -fwrapv -fwrapv -O3 -Wall -Wstrict-prototypes
> -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -D_PYMOL_LIBPNG
> -D_PYMOL_INLINE -D_PYMOL_OPENGL_SHADERS -D_PYMOL_CGO_DRAWARRAYS
> -D_PYMOL_CGO_DRAWBUFFERS -D_PYMOL_GL_CALLLISTS -D_PYMOL_VMD_PLUGINS
> -D_HAVE_LIBXML -D_PYMOL_FREETYPE -DNO_MMLIBS -D_CGO_DRAWARRAYS
> -DOPENGL_ES_2 -D_PYMOL_NUMPY -Iov/src -Ilayer0 -Ilayer1 -Ilayer2
> -Ilayer3 -Ilayer4 -Ilayer5 -Imodules/cealign/src -Ibuild/generated
> -Icontrib/uiuc/plugins/include
> -Icontrib/uiuc/plugins/molfile_plugin/src
> -I/sw/lib/python2.7/site-packages/numpy/core/include -I/sw/include
> -I/sw/include/freetype2 -I/sw/include/libxml2 -I/usr/include
> -I/usr/include/libxml2 -I/usr/X11/include -I/usr/X11/include/freetype2
> -I/sw/include/python2.7 -c layer0/Field.cpp -o
> build/temp.macosx-10.10-x86_64-2.7/layer0/Field.o
> -Werror=implicit-function-declaration
> -Werror=declaration-after-statement -Wno-write-strings
> -Wno-unused-function -Wno-empty-body -Wno-char-subscripts -ffast-math
> -funroll-loops -O3 -fcommon
>
> layer0/Field.cpp:76:14: error: no matching function for call to 'PyArray_DATA'
> memcpy(PyArray_DATA(result), field->data, field->size);
> ^~~~
>
> /sw/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1460:1:
> note: candidate function not viable: no known conversion from
> 'PyObject *' (aka '_object *') to
> 'PyArrayObject *' (aka 'tagPyArrayObject *') for 1st argument
> PyArray_DATA(PyArrayObject *arr)
> ^
> 1 error generated.
>
> What is the correct coding to eliminate this error? I have found some
> threads which seems to suggest that PyArray_DATA is still available in
> numpy 1.9 as an inline but I haven't found any examples of projects
> patching their code to convert to that usage.
> Jack
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Add a function to broadcast arrays to a given shape to numpy's stride_tricks?
On Sunday, December 7, 2014, Stephan Hoyer wrote: > I recently wrote function to manually broadcast an ndarray to a given > shape according to numpy's broadcasting rules (using strides): > > https://github.com/xray/xray/commit/7aee4a3ed2dfd3b9aff7f3c5c6c68d51df2e3ff3 > > The same functionality can be done pretty straightforwardly with > np.broadcast_arrays, but that function does both too much (I don't actually > have a second array that needs to be broadcast) and not enough (I need to > create a dummy array to broadcast against it). > > This approach is simpler, and also, according to my benchmarks, about 3x > faster than np.broadcast_arrays: > > In [1]: import xray > In [2]: import numpy as np > In [3]: x = np.random.randn(4) > In [4]: y = np.empty((2, 3, 4)) > In [5]: %timeit xray.core.utils.as_shape(x, y.shape) > 10 loops, best of 3: 17 µs per loop > In [6]: %timeit np.broadcast_arrays(x, y)[0] > 1 loops, best of 3: 47.4 µs per loop > > Would this be a welcome addition to numpy's lib.stride_tricks? If so, I > will put together a PR. > > In my search, I turned up a Stack Overflow post looking for similar > functionality: > > https://stackoverflow.com/questions/11622692/is-there-a-better-way-to-broadcast-arrays > That would be excellent - I ran into exactly the same problem, with the same conclusions, but I was lazier than you were and I did write a routine for making a dummy array in order to use broadcast_arrays: https://github.com/nipy/nibabel/blob/master/nibabel/fileslice.py#L722 https://github.com/nipy/nibabel/blob/master/nibabel/parrec.py#L577 Having a function to do this would be much clearer, thanks for doing that. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Add a function to broadcast arrays to a given shape to numpy's stride_tricks?
Hi, On Monday, December 8, 2014, Pierre Haessig wrote: > Hi, > > Le 07/12/2014 08:10, Stephan Hoyer a écrit : > > In [5]: %timeit xray.core.utils.as_shape(x, y.shape) > > 10 loops, best of 3: 17 µs per loop > > > > Would this be a welcome addition to numpy's lib.stride_tricks? If so, > > I will put together a PR. > > > > > > Instead of putting this function in stride_tricks (which is quite > hidden), could it be added instead as a boolean flag to the existing > `reshape` method ? Something like: > > x.reshape(y.shape, broadcast=True) > That might be a bit odd, because the non-broadcast version would allow entirely different parameters for shape than the broadcast version. For example, what would these do? a = np.zeros((2, 3, 4)) a.reshape((6, 4), broadcast=True) a.reshape((2, -1), broadcast=True) So I think 'reshape' is doing something different enough that this should be a separate function. Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] help using np.correlate to produce correlograms.
On 11/12/14 09:54, Pierre Haessig wrote: > The basic idea is to enable the user to select the exact range of lags > he wants. Unfortunately I didn't take the time to go further than the > specification above... I would be particularly interested in computing cross-correlations in a range of +-4000 sampling points lags. Unfortunately, my cross-correlations require vectors of ~8e6 of points, and np.correlate performs very slowly if I compute the whole range. I also heard that a faster alternative to compute the cross-correlation is to perform the product of the Fourier transform of the 2 vectors and then performing the inverse Fourier of the result. Best Jose -- Jose Guzman http://www.ist.ac.at/~jguzman/ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] help using np.correlate to produce correlograms.
On 11.12.2014 19:01, Jose Guzman wrote: > On 11/12/14 09:54, Pierre Haessig wrote: >> The basic idea is to enable the user to select the exact range of lags >> he wants. Unfortunately I didn't take the time to go further than the >> specification above... > > I would be particularly interested in computing cross-correlations in a > range of +-4000 sampling points lags. Unfortunately, my > cross-correlations require vectors of ~8e6 of points, and np.correlate > performs very slowly if I compute the whole range. > > I also heard that a faster alternative to compute the cross-correlation > is to perform the product of the Fourier transform of the 2 vectors and > then performing the inverse Fourier of the result. > Large convolutions/correlations are generally faster in fourier space as they have O(NlogN) instead of O(N^2) complexity, for 1e6 points this should be very significant. You can use scipy.signal.fftconvolve to do that conveniently (with performance optimal zero padding). Convolution of a flipped input (and conjugated?) is the same as a correlation. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Add a function to broadcast arrays to a given shape to numpy's stride_tricks?
On Thu, Dec 11, 2014 at 8:17 AM, Sebastian Berg wrote: > One option > would also be to have something like: > > np.common_shape(*arrays) > np.broadcast_to(array, shape) > # (though I would like many arrays too) > > and then broadcast_ar rays could be implemented in terms of these two. > It looks like np.broadcast let's us write the common_shape function very easily; def common_shape(*args): return np.broadcast(*args).shape And it's also very fast: 100 loops, best of 3: 1.04 µs per loop So that does seem like a feasible refactor/simplification for np.broadcast_arrays. Sebastian -- if you're up for writing np.broadcast_to in C, that's great! If you're not sure if you'll be able to get around to that in the near future, I'll submit my PR with a Python implementation (which will have tests that will be useful in any case). ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Should ndarray be a context manager?
On Wed, Dec 10, 2014 at 8:40 PM, Sturla Molden wrote: > > I haven't managed to trigger a segfault yet but it sure looks like I > > could... > > You can also trigger random errors. If the array is small, Python's memory > mamager might keep the memory in the heap for reuse by PyMem_Malloc. And > then you can actually modify some random Python object with bogus bits. It > can screw things up even if you do not see a segfault. > I should have said that -- I did see random garbage -- just not an actual segfault -- probably didn't allocate a large enough array for the system to reclaim that memory. Anyway, the point is that if we wanted this to be a used-more-than-very-rarely in only very special cases feature de-allocating an array's data buffer), then ndarray would need to grow a check for an invalid buffer on access. I have no idea if that would make for a noticeable performance hit. -Chris > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] error: no matching function for call to 'PyArray_DATA'
Jack Howarth wrote: > What is the correct coding to eliminate this error? I have found some > threads which seems to suggest that PyArray_DATA is still available in > numpy 1.9 as an inline but I haven't found any examples of projects > patching their code to convert to that usage. In the deprecated API, PyArray_DATA is a macro. In the new API, PyArray_DATA is an inline function. While almost idential from the perspective of user code, the inline function has an argument with a type. Judging from the error message, the problem is that your NumPy array is represented by PyObject* instead of PyArrayObject*, and your C compiler says it cannot do the conversion automatically. The old macro version does the typecast, so it does not matter if you give it PyObject*. Solution? Use PyArrayObject* consistently in your code or typecast the PyObject* pointer on each function call to PyArray_DATA. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
