Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 8:33 PM, wrote: -np.inf>-np.inf > False > > If the only value is -np.inf, you will return nan, I guess. > np.nanmax([-np.inf, np.nan]) > -inf That's a great corner case. Thanks, Josef. This looks like it would fix it: change if ai > amax: amax = ai to if ai >= amax: amax = ai ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 10:59 PM, Keith Goodman wrote: > On Fri, Nov 19, 2010 at 7:51 PM, wrote: >> >> does this give you the correct answer? >> > 1>np.nan >> False >> >> What's the starting value for amax? -inf? > > Because "1 > np.nan" is False, the current running max does not get > updated, which is what we want. > >>> import nanny as ny >>> np.nanmax([1, np.nan]) > 1.0 >>> np.nanmax([np.nan, 1]) > 1.0 >>> np.nanmax([np.nan, 1, np.nan]) > 1.0 > > Starting value is -np.inf for floats and stuff like this for ints: > > cdef np.int32_t MININT32 = np.iinfo(np.int32).min > cdef np.int64_t MININT64 = np.iinfo(np.int64).min That's what I thought halfway through typing the question. >>> -np.inf>-np.inf False If the only value is -np.inf, you will return nan, I guess. >>> np.nanmax([-np.inf, np.nan]) -inf Josef (being picky) > > Numpy does this: > >>> np.nanmax([]) > > ValueError: zero-size array to ufunc.reduce without identity > > Nanny does this: > >>> ny.nanmax([]) > nan > > So I haven't taken care of that corner case yet. I'll commit nanmax to > github in case anyone wants to give it a try. > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 8:05 PM, Charles R Harris wrote: > This doesn't look right: > > @cython.boundscheck(False) > @cython.wraparound(False) > def nanmax_2d_float64_axisNone(np.ndarray[np.float64_t, ndim=2] a): > "nanmax of 2d numpy array with dtype=np.float64 along axis=None." > cdef Py_ssize_t i, j > cdef int arow = a.shape[0], acol = a.shape[1], allnan = 1 > cdef np.float64_t amax = 0, aij > for i in range(arow): > for j in range(acol): > aij = a[i,j] > if aij == aij: > amax += aij > allnan = 0 > if allnan == 0: > return np.float64(amax) > else: > return NAN > > It's doing a sum, not a comparison. That was a placeholder. Looks at the latest commit. Sorry for the confusion. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 8:42 PM, Keith Goodman wrote: > On Fri, Nov 19, 2010 at 7:19 PM, Charles R Harris > wrote: > > > > > > On Fri, Nov 19, 2010 at 1:50 PM, Keith Goodman > wrote: > >> > >> On Fri, Nov 19, 2010 at 12:29 PM, Keith Goodman > >> wrote: > >> > On Fri, Nov 19, 2010 at 12:19 PM, Pauli Virtanen wrote: > >> >> Fri, 19 Nov 2010 11:19:57 -0800, Keith Goodman wrote: > >> >> [clip] > >> >>> My guess is that having separate underlying functions for each > dtype, > >> >>> ndim, and axis would be a nightmare for a large project like Numpy. > >> >>> But > >> >>> manageable for a focused project like nanny. > >> >> > >> >> Might be easier to migrate the nan* functions to using Ufuncs. > >> >> > >> >> Unless I'm missing something, > >> >> > >> >>np.nanmax -> np.fmax.reduce > >> >>np.nanmin -> np.fmin.reduce > >> >> > >> >> For `nansum`, we'd need to add an ufunc `nanadd`, and for > >> >> `nanargmax/min`, we'd need `argfmin/fmax'. > >> > > >> > How about that! I wasn't aware of fmax/fmin. Yes, I'd like a nanadd, > >> > please. > >> > > >> >>> arr = np.random.rand(1000, 1000) > >> >>> arr[arr > 0.5] = np.nan > >> >>> np.nanmax(arr) > >> > 0.4625409581072 > >> >>> np.fmax.reduce(arr, axis=None) > >> > > >> > TypeError: an integer is required > >> >>> np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0) > >> > 0.4625409581072 > >> > > >> >>> timeit np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0) > >> > 100 loops, best of 3: 12.7 ms per loop > >> >>> timeit np.nanmax(arr) > >> > 10 loops, best of 3: 39.6 ms per loop > >> > > >> >>> timeit np.nanmax(arr, axis=0) > >> > 10 loops, best of 3: 46.5 ms per loop > >> >>> timeit np.fmax.reduce(arr, axis=0) > >> > 100 loops, best of 3: 12.7 ms per loop > >> > >> Cython is faster than np.fmax.reduce. > >> > >> I wrote a cython version of np.nanmax, called nanmax below. (It only > >> handles the 2d, float64, axis=None case, but since the array is large > >> I don't think that explains the time difference). > >> > >> Note that fmax.reduce is slower than np.nanmax when there are no NaNs: > >> > >> >> arr = np.random.rand(1000, 1000) > >> >> timeit np.nanmax(arr) > >> 100 loops, best of 3: 5.82 ms per loop > >> >> timeit np.fmax.reduce(np.fmax.reduce(arr)) > >> 100 loops, best of 3: 9.14 ms per loop > >> >> timeit nanmax(arr) > >> 1000 loops, best of 3: 1.17 ms per loop > >> > >> >> arr[arr > 0.5] = np.nan > >> > >> >> timeit np.nanmax(arr) > >> 10 loops, best of 3: 45.5 ms per loop > >> >> timeit np.fmax.reduce(np.fmax.reduce(arr)) > >> 100 loops, best of 3: 12.7 ms per loop > >> >> timeit nanmax(arr) > >> 1000 loops, best of 3: 1.17 ms per loop > > > > There seem to be some odd hardware/compiler dependencies. I get quite a > > different pattern of times: > > > > In [1]: arr = np.random.rand(1000, 1000) > > > > In [2]: timeit np.nanmax(arr) > > 100 loops, best of 3: 10.4 ms per loop > > > > In [3]: timeit np.fmax.reduce(arr.flat) > > 100 loops, best of 3: 2.09 ms per loop > > > > In [4]: arr[arr > 0.5] = np.nan > > > > In [5]: timeit np.nanmax(arr) > > 100 loops, best of 3: 12.9 ms per loop > > > > In [6]: timeit np.fmax.reduce(arr.flat) > > 100 loops, best of 3: 7.09 ms per loop > > > > > > I've tweaked fmax with the reduce loop option but the nanmax times don't > > look like yours at all. I'm also a bit surprised that > > you don't see any difference in times when the array contains a lot of > nans. > > I'm running on AMD Phenom, gcc 4.4.5. > > Ubuntu 10.04 64 bit, numpy 1.4.1. > > Difference in which times? nanny.nanmax with and wintout NaNs? The > code doesn't explictily check for NaNs (it does check for all NaNs). > It basically loops through the data and does: > > allnan = 1 > ai = ai[i,k] > if ai > amax: >amax = ai >allnan = 0 > > I should make a benchmark suite. > _ > This doesn't look right: @cython.boundscheck(False) @cython.wraparound(False) def nanmax_2d_float64_axisNone(np.ndarray[np.float64_t, ndim=2] a): "nanmax of 2d numpy array with dtype=np.float64 along axis=None." cdef Py_ssize_t i, j cdef int arow = a.shape[0], acol = a.shape[1], allnan = 1 cdef np.float64_t amax = 0, aij for i in range(arow): for j in range(acol): aij = a[i,j] if aij == aij: amax += aij allnan = 0 if allnan == 0: return np.float64(amax) else: return NAN It's doing a sum, not a comparison. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 7:51 PM, wrote: > > does this give you the correct answer? > 1>np.nan > False > > What's the starting value for amax? -inf? Because "1 > np.nan" is False, the current running max does not get updated, which is what we want. >> import nanny as ny >> np.nanmax([1, np.nan]) 1.0 >> np.nanmax([np.nan, 1]) 1.0 >> np.nanmax([np.nan, 1, np.nan]) 1.0 Starting value is -np.inf for floats and stuff like this for ints: cdef np.int32_t MININT32 = np.iinfo(np.int32).min cdef np.int64_t MININT64 = np.iinfo(np.int64).min Numpy does this: >> np.nanmax([]) ValueError: zero-size array to ufunc.reduce without identity Nanny does this: >> ny.nanmax([]) nan So I haven't taken care of that corner case yet. I'll commit nanmax to github in case anyone wants to give it a try. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 7:51 PM, wrote: > On Fri, Nov 19, 2010 at 10:42 PM, Keith Goodman wrote: >> It basically loops through the data and does: >> >> allnan = 1 >> ai = ai[i,k] >> if ai > amax: >> amax = ai >> allnan = 0 > > does this give you the correct answer? > 1>np.nan > False Yes -- notice he does the comparison the other way, and >>> 1 < np.nan False (All comparisons involving NaN return false, including, famously, NaN == NaN, which is why we need np.isnan.) -- Nathaniel ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 10:42 PM, Keith Goodman wrote: > On Fri, Nov 19, 2010 at 7:19 PM, Charles R Harris > wrote: >> >> >> On Fri, Nov 19, 2010 at 1:50 PM, Keith Goodman wrote: >>> >>> On Fri, Nov 19, 2010 at 12:29 PM, Keith Goodman >>> wrote: >>> > On Fri, Nov 19, 2010 at 12:19 PM, Pauli Virtanen wrote: >>> >> Fri, 19 Nov 2010 11:19:57 -0800, Keith Goodman wrote: >>> >> [clip] >>> >>> My guess is that having separate underlying functions for each dtype, >>> >>> ndim, and axis would be a nightmare for a large project like Numpy. >>> >>> But >>> >>> manageable for a focused project like nanny. >>> >> >>> >> Might be easier to migrate the nan* functions to using Ufuncs. >>> >> >>> >> Unless I'm missing something, >>> >> >>> >> np.nanmax -> np.fmax.reduce >>> >> np.nanmin -> np.fmin.reduce >>> >> >>> >> For `nansum`, we'd need to add an ufunc `nanadd`, and for >>> >> `nanargmax/min`, we'd need `argfmin/fmax'. >>> > >>> > How about that! I wasn't aware of fmax/fmin. Yes, I'd like a nanadd, >>> > please. >>> > >>> >>> arr = np.random.rand(1000, 1000) >>> >>> arr[arr > 0.5] = np.nan >>> >>> np.nanmax(arr) >>> > 0.4625409581072 >>> >>> np.fmax.reduce(arr, axis=None) >>> > >>> > TypeError: an integer is required >>> >>> np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0) >>> > 0.4625409581072 >>> > >>> >>> timeit np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0) >>> > 100 loops, best of 3: 12.7 ms per loop >>> >>> timeit np.nanmax(arr) >>> > 10 loops, best of 3: 39.6 ms per loop >>> > >>> >>> timeit np.nanmax(arr, axis=0) >>> > 10 loops, best of 3: 46.5 ms per loop >>> >>> timeit np.fmax.reduce(arr, axis=0) >>> > 100 loops, best of 3: 12.7 ms per loop >>> >>> Cython is faster than np.fmax.reduce. >>> >>> I wrote a cython version of np.nanmax, called nanmax below. (It only >>> handles the 2d, float64, axis=None case, but since the array is large >>> I don't think that explains the time difference). >>> >>> Note that fmax.reduce is slower than np.nanmax when there are no NaNs: >>> >>> >> arr = np.random.rand(1000, 1000) >>> >> timeit np.nanmax(arr) >>> 100 loops, best of 3: 5.82 ms per loop >>> >> timeit np.fmax.reduce(np.fmax.reduce(arr)) >>> 100 loops, best of 3: 9.14 ms per loop >>> >> timeit nanmax(arr) >>> 1000 loops, best of 3: 1.17 ms per loop >>> >>> >> arr[arr > 0.5] = np.nan >>> >>> >> timeit np.nanmax(arr) >>> 10 loops, best of 3: 45.5 ms per loop >>> >> timeit np.fmax.reduce(np.fmax.reduce(arr)) >>> 100 loops, best of 3: 12.7 ms per loop >>> >> timeit nanmax(arr) >>> 1000 loops, best of 3: 1.17 ms per loop >> >> There seem to be some odd hardware/compiler dependencies. I get quite a >> different pattern of times: >> >> In [1]: arr = np.random.rand(1000, 1000) >> >> In [2]: timeit np.nanmax(arr) >> 100 loops, best of 3: 10.4 ms per loop >> >> In [3]: timeit np.fmax.reduce(arr.flat) >> 100 loops, best of 3: 2.09 ms per loop >> >> In [4]: arr[arr > 0.5] = np.nan >> >> In [5]: timeit np.nanmax(arr) >> 100 loops, best of 3: 12.9 ms per loop >> >> In [6]: timeit np.fmax.reduce(arr.flat) >> 100 loops, best of 3: 7.09 ms per loop >> >> >> I've tweaked fmax with the reduce loop option but the nanmax times don't >> look like yours at all. I'm also a bit surprised that >> you don't see any difference in times when the array contains a lot of nans. >> I'm running on AMD Phenom, gcc 4.4.5. > > Ubuntu 10.04 64 bit, numpy 1.4.1. > > Difference in which times? nanny.nanmax with and wintout NaNs? The > code doesn't explictily check for NaNs (it does check for all NaNs). > It basically loops through the data and does: > > allnan = 1 > ai = ai[i,k] > if ai > amax: > amax = ai > allnan = 0 does this give you the correct answer? >>> 1>np.nan False What's the starting value for amax? -inf? Josef > > I should make a benchmark suite. > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 7:19 PM, Charles R Harris wrote: > > > On Fri, Nov 19, 2010 at 1:50 PM, Keith Goodman wrote: >> >> On Fri, Nov 19, 2010 at 12:29 PM, Keith Goodman >> wrote: >> > On Fri, Nov 19, 2010 at 12:19 PM, Pauli Virtanen wrote: >> >> Fri, 19 Nov 2010 11:19:57 -0800, Keith Goodman wrote: >> >> [clip] >> >>> My guess is that having separate underlying functions for each dtype, >> >>> ndim, and axis would be a nightmare for a large project like Numpy. >> >>> But >> >>> manageable for a focused project like nanny. >> >> >> >> Might be easier to migrate the nan* functions to using Ufuncs. >> >> >> >> Unless I'm missing something, >> >> >> >> np.nanmax -> np.fmax.reduce >> >> np.nanmin -> np.fmin.reduce >> >> >> >> For `nansum`, we'd need to add an ufunc `nanadd`, and for >> >> `nanargmax/min`, we'd need `argfmin/fmax'. >> > >> > How about that! I wasn't aware of fmax/fmin. Yes, I'd like a nanadd, >> > please. >> > >> >>> arr = np.random.rand(1000, 1000) >> >>> arr[arr > 0.5] = np.nan >> >>> np.nanmax(arr) >> > 0.4625409581072 >> >>> np.fmax.reduce(arr, axis=None) >> > >> > TypeError: an integer is required >> >>> np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0) >> > 0.4625409581072 >> > >> >>> timeit np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0) >> > 100 loops, best of 3: 12.7 ms per loop >> >>> timeit np.nanmax(arr) >> > 10 loops, best of 3: 39.6 ms per loop >> > >> >>> timeit np.nanmax(arr, axis=0) >> > 10 loops, best of 3: 46.5 ms per loop >> >>> timeit np.fmax.reduce(arr, axis=0) >> > 100 loops, best of 3: 12.7 ms per loop >> >> Cython is faster than np.fmax.reduce. >> >> I wrote a cython version of np.nanmax, called nanmax below. (It only >> handles the 2d, float64, axis=None case, but since the array is large >> I don't think that explains the time difference). >> >> Note that fmax.reduce is slower than np.nanmax when there are no NaNs: >> >> >> arr = np.random.rand(1000, 1000) >> >> timeit np.nanmax(arr) >> 100 loops, best of 3: 5.82 ms per loop >> >> timeit np.fmax.reduce(np.fmax.reduce(arr)) >> 100 loops, best of 3: 9.14 ms per loop >> >> timeit nanmax(arr) >> 1000 loops, best of 3: 1.17 ms per loop >> >> >> arr[arr > 0.5] = np.nan >> >> >> timeit np.nanmax(arr) >> 10 loops, best of 3: 45.5 ms per loop >> >> timeit np.fmax.reduce(np.fmax.reduce(arr)) >> 100 loops, best of 3: 12.7 ms per loop >> >> timeit nanmax(arr) >> 1000 loops, best of 3: 1.17 ms per loop > > There seem to be some odd hardware/compiler dependencies. I get quite a > different pattern of times: > > In [1]: arr = np.random.rand(1000, 1000) > > In [2]: timeit np.nanmax(arr) > 100 loops, best of 3: 10.4 ms per loop > > In [3]: timeit np.fmax.reduce(arr.flat) > 100 loops, best of 3: 2.09 ms per loop > > In [4]: arr[arr > 0.5] = np.nan > > In [5]: timeit np.nanmax(arr) > 100 loops, best of 3: 12.9 ms per loop > > In [6]: timeit np.fmax.reduce(arr.flat) > 100 loops, best of 3: 7.09 ms per loop > > > I've tweaked fmax with the reduce loop option but the nanmax times don't > look like yours at all. I'm also a bit surprised that > you don't see any difference in times when the array contains a lot of nans. > I'm running on AMD Phenom, gcc 4.4.5. Ubuntu 10.04 64 bit, numpy 1.4.1. Difference in which times? nanny.nanmax with and wintout NaNs? The code doesn't explictily check for NaNs (it does check for all NaNs). It basically loops through the data and does: allnan = 1 ai = ai[i,k] if ai > amax: amax = ai allnan = 0 I should make a benchmark suite. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 8:19 PM, Charles R Harris wrote: > > > On Fri, Nov 19, 2010 at 1:50 PM, Keith Goodman wrote: > >> On Fri, Nov 19, 2010 at 12:29 PM, Keith Goodman >> wrote: >> > On Fri, Nov 19, 2010 at 12:19 PM, Pauli Virtanen wrote: >> >> Fri, 19 Nov 2010 11:19:57 -0800, Keith Goodman wrote: >> >> [clip] >> >>> My guess is that having separate underlying functions for each dtype, >> >>> ndim, and axis would be a nightmare for a large project like Numpy. >> But >> >>> manageable for a focused project like nanny. >> >> >> >> Might be easier to migrate the nan* functions to using Ufuncs. >> >> >> >> Unless I'm missing something, >> >> >> >>np.nanmax -> np.fmax.reduce >> >>np.nanmin -> np.fmin.reduce >> >> >> >> For `nansum`, we'd need to add an ufunc `nanadd`, and for >> >> `nanargmax/min`, we'd need `argfmin/fmax'. >> > >> > How about that! I wasn't aware of fmax/fmin. Yes, I'd like a nanadd, >> please. >> > >> >>> arr = np.random.rand(1000, 1000) >> >>> arr[arr > 0.5] = np.nan >> >>> np.nanmax(arr) >> > 0.4625409581072 >> >>> np.fmax.reduce(arr, axis=None) >> > >> > TypeError: an integer is required >> >>> np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0) >> > 0.4625409581072 >> > >> >>> timeit np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0) >> > 100 loops, best of 3: 12.7 ms per loop >> >>> timeit np.nanmax(arr) >> > 10 loops, best of 3: 39.6 ms per loop >> > >> >>> timeit np.nanmax(arr, axis=0) >> > 10 loops, best of 3: 46.5 ms per loop >> >>> timeit np.fmax.reduce(arr, axis=0) >> > 100 loops, best of 3: 12.7 ms per loop >> >> Cython is faster than np.fmax.reduce. >> >> I wrote a cython version of np.nanmax, called nanmax below. (It only >> handles the 2d, float64, axis=None case, but since the array is large >> I don't think that explains the time difference). >> >> Note that fmax.reduce is slower than np.nanmax when there are no NaNs: >> >> >> arr = np.random.rand(1000, 1000) >> >> timeit np.nanmax(arr) >> 100 loops, best of 3: 5.82 ms per loop >> >> timeit np.fmax.reduce(np.fmax.reduce(arr)) >> 100 loops, best of 3: 9.14 ms per loop >> >> timeit nanmax(arr) >> 1000 loops, best of 3: 1.17 ms per loop >> >> >> arr[arr > 0.5] = np.nan >> >> >> timeit np.nanmax(arr) >> 10 loops, best of 3: 45.5 ms per loop >> >> timeit np.fmax.reduce(np.fmax.reduce(arr)) >> 100 loops, best of 3: 12.7 ms per loop >> >> timeit nanmax(arr) >> 1000 loops, best of 3: 1.17 ms per loop >> > > There seem to be some odd hardware/compiler dependencies. I get quite a > different pattern of times: > > In [1]: arr = np.random.rand(1000, 1000) > > In [2]: timeit np.nanmax(arr) > 100 loops, best of 3: 10.4 ms per loop > > In [3]: timeit np.fmax.reduce(arr.flat) > 100 loops, best of 3: 2.09 ms per loop > > In [4]: arr[arr > 0.5] = np.nan > > In [5]: timeit np.nanmax(arr) > 100 loops, best of 3: 12.9 ms per loop > > In [6]: timeit np.fmax.reduce(arr.flat) > 100 loops, best of 3: 7.09 ms per loop > > > I've tweaked fmax with the reduce loop option but the nanmax times don't > look like yours at all. I'm also a bit surprised that > you don't see any difference in times when the array contains a lot of > nans. I'm running on AMD Phenom, gcc 4.4.5. > > However, I noticed that the build wants to be -O1 by default. I have my own CFLAGS that make it -O2, but It looks like ubuntu's python might be built with -O1. Hmm. That could certainly cause some odd timings. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 1:50 PM, Keith Goodman wrote: > On Fri, Nov 19, 2010 at 12:29 PM, Keith Goodman > wrote: > > On Fri, Nov 19, 2010 at 12:19 PM, Pauli Virtanen wrote: > >> Fri, 19 Nov 2010 11:19:57 -0800, Keith Goodman wrote: > >> [clip] > >>> My guess is that having separate underlying functions for each dtype, > >>> ndim, and axis would be a nightmare for a large project like Numpy. But > >>> manageable for a focused project like nanny. > >> > >> Might be easier to migrate the nan* functions to using Ufuncs. > >> > >> Unless I'm missing something, > >> > >>np.nanmax -> np.fmax.reduce > >>np.nanmin -> np.fmin.reduce > >> > >> For `nansum`, we'd need to add an ufunc `nanadd`, and for > >> `nanargmax/min`, we'd need `argfmin/fmax'. > > > > How about that! I wasn't aware of fmax/fmin. Yes, I'd like a nanadd, > please. > > > >>> arr = np.random.rand(1000, 1000) > >>> arr[arr > 0.5] = np.nan > >>> np.nanmax(arr) > > 0.4625409581072 > >>> np.fmax.reduce(arr, axis=None) > > > > TypeError: an integer is required > >>> np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0) > > 0.4625409581072 > > > >>> timeit np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0) > > 100 loops, best of 3: 12.7 ms per loop > >>> timeit np.nanmax(arr) > > 10 loops, best of 3: 39.6 ms per loop > > > >>> timeit np.nanmax(arr, axis=0) > > 10 loops, best of 3: 46.5 ms per loop > >>> timeit np.fmax.reduce(arr, axis=0) > > 100 loops, best of 3: 12.7 ms per loop > > Cython is faster than np.fmax.reduce. > > I wrote a cython version of np.nanmax, called nanmax below. (It only > handles the 2d, float64, axis=None case, but since the array is large > I don't think that explains the time difference). > > Note that fmax.reduce is slower than np.nanmax when there are no NaNs: > > >> arr = np.random.rand(1000, 1000) > >> timeit np.nanmax(arr) > 100 loops, best of 3: 5.82 ms per loop > >> timeit np.fmax.reduce(np.fmax.reduce(arr)) > 100 loops, best of 3: 9.14 ms per loop > >> timeit nanmax(arr) > 1000 loops, best of 3: 1.17 ms per loop > > >> arr[arr > 0.5] = np.nan > > >> timeit np.nanmax(arr) > 10 loops, best of 3: 45.5 ms per loop > >> timeit np.fmax.reduce(np.fmax.reduce(arr)) > 100 loops, best of 3: 12.7 ms per loop > >> timeit nanmax(arr) > 1000 loops, best of 3: 1.17 ms per loop > There seem to be some odd hardware/compiler dependencies. I get quite a different pattern of times: In [1]: arr = np.random.rand(1000, 1000) In [2]: timeit np.nanmax(arr) 100 loops, best of 3: 10.4 ms per loop In [3]: timeit np.fmax.reduce(arr.flat) 100 loops, best of 3: 2.09 ms per loop In [4]: arr[arr > 0.5] = np.nan In [5]: timeit np.nanmax(arr) 100 loops, best of 3: 12.9 ms per loop In [6]: timeit np.fmax.reduce(arr.flat) 100 loops, best of 3: 7.09 ms per loop I've tweaked fmax with the reduce loop option but the nanmax times don't look like yours at all. I'm also a bit surprised that you don't see any difference in times when the array contains a lot of nans. I'm running on AMD Phenom, gcc 4.4.5. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] seeking advice on a fast string->array conversion
I am wrapping up a small package to parse a particular ascii-encoded file format generated by a program we use heavily here at the lab. (In the unlikely event that you work at a synchrotron, and use Certified Scientific's "spec" program, and are actually interested, the code is currently available at https://github.com/darrendale/praxes/tree/specformat/praxes/io/spec/ .) I have been benchmarking the project against another python package developed by a colleague, which is an extension module written in pure C. My python/cython project takes about twice as long to parse and index a file (~0.8 seconds for 100MB), which is acceptable. However, actually converting ascii strings to numpy arrays, which is done using numpy.fromstring, takes a factor of 10 longer than the extension module. So I am wondering about the performance of np.fromstring: import time import numpy as np s = b'1 ' * 2048 *1200 d = time.time() x = np.fromstring(s) print time.time() - d ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] changing 'nan' string for np.nan in np.savetxt
Hello everybody,
I was wondering if there is an elegant way of overwriting 'nan' string
representation with 'NaN' when saving numpy array containing np.nan values with
numpy.savetxt() function. numpy.set_printoptions(nanstr='NaN') setting (which
has default value of 'NaN' already) does not seem to have any effect when
writing array to the file, only when printing to the screen:
$ python
Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03)
[GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> print np.__version__
1.3.0
>>> np.get_printoptions()
{'infstr': 'Inf', 'threshold': 1000, 'suppress': False, 'linewidth': 75,
'edgeitems': 3, 'precision': 8, 'nanstr': 'NaN'}
>>>
>>> a = np.array([1, 2, 3, np.nan])
>>> a
array([ 1., 2., 3., NaN])
>>> np.savetxt('testa.txt', a)
>>>
$ cat testa.txt
1.00e+00
2.00e+00
3.00e+00
nan
$
Many thanks,
Masha
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] seeking advice on a fast string->array conversion
Apologies, I accidentally hit send... On Tue, Nov 16, 2010 at 9:20 AM, Darren Dale wrote: > I am wrapping up a small package to parse a particular ascii-encoded > file format generated by a program we use heavily here at the lab. (In > the unlikely event that you work at a synchrotron, and use Certified > Scientific's "spec" program, and are actually interested, the code is > currently available at > https://github.com/darrendale/praxes/tree/specformat/praxes/io/spec/ > .) > > I have been benchmarking the project against another python package > developed by a colleague, which is an extension module written in pure > C. My python/cython project takes about twice as long to parse and > index a file (~0.8 seconds for 100MB), which is acceptable. However, > actually converting ascii strings to numpy arrays, which is done using > numpy.fromstring, takes a factor of 10 longer than the extension > module. So I am wondering about the performance of np.fromstring: import time import numpy as np s = b'1 ' * 2048 *1200 d = time.time() x = np.fromstring(s, dtype='d', sep=b' ') print time.time() - d That takes about 1.3 seconds on my machine. A similar metric for the extension module is to load 1200 of these 2048-element arrays from the file: d=time.time() x=[s.mca(i+1) for i in xrange(1200)] print time.time()-d That takes about 0.127 seconds on my machine. This discrepancy is unacceptable for my usecase, so I need to develop an alternative to fromstring. Here is bit of testing with cython: import time cdef extern from 'stdlib.h': double atof(char*) py_string = '100' cdef char* c_string = py_string cdef int i, j j=2048*1200 d = time.time() while ihttp://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 3:18 PM, Christopher Barker wrote: > On 11/19/10 11:19 AM, Keith Goodman wrote: > > On Fri, Nov 19, 2010 at 10:55 AM, Nathaniel Smith wrote: > >> Why not make this a patch to numpy/scipy instead? > > > > My guess is that having separate underlying functions for each dtype, > > ndim, and axis would be a nightmare for a large project like Numpy. > > True, but: > > 1) Having special-cases for the most common cases is not such a bad idea. > > 2) could one use some sort of templating approach to get all the dtypes > and such that you want? > > 3) as for number of dimensions, I don't think to would be to hard to > generalize that -- at least for contiguous arrays. > > Note that the fmax/fmin versions can be sped up in the same way as sum.reduce was. Also, you should pass the flattened array to the routine for the axis=None case. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] seeking advice on a fast string->array conversion
Tue, 16 Nov 2010 09:20:29 -0500, Darren Dale wrote: [clip] > module. So I am wondering about the performance of np.fromstring: Fromstring is slow, probably because it must work around locale- dependence of the underlying C parsing functions. Moreover, the Numpy parsing mechanism generates many indirect calls. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On 11/19/10 11:19 AM, Keith Goodman wrote: > On Fri, Nov 19, 2010 at 10:55 AM, Nathaniel Smith wrote: >> Why not make this a patch to numpy/scipy instead? > > My guess is that having separate underlying functions for each dtype, > ndim, and axis would be a nightmare for a large project like Numpy. True, but: 1) Having special-cases for the most common cases is not such a bad idea. 2) could one use some sort of templating approach to get all the dtypes and such that you want? 3) as for number of dimensions, I don't think to would be to hard to generalize that -- at least for contiguous arrays. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] seeking advice on a fast string->array conversion
Tue, 16 Nov 2010 09:41:04 -0500, Darren Dale wrote: [clip] > That loop takes 0.33 seconds to execute, which is a good start. I need > some help converting this example to return an actual numpy array. Could > anyone please offer a suggestion? Easiest way is probably to use ndarray buffers and resize them when needed. For example: https://github.com/pv/scipy-work/blob/enh/interpnd-smooth/scipy/spatial/qhull.pyx#L980 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 12:29 PM, Keith Goodman wrote: > On Fri, Nov 19, 2010 at 12:19 PM, Pauli Virtanen wrote: >> Fri, 19 Nov 2010 11:19:57 -0800, Keith Goodman wrote: >> [clip] >>> My guess is that having separate underlying functions for each dtype, >>> ndim, and axis would be a nightmare for a large project like Numpy. But >>> manageable for a focused project like nanny. >> >> Might be easier to migrate the nan* functions to using Ufuncs. >> >> Unless I'm missing something, >> >> np.nanmax -> np.fmax.reduce >> np.nanmin -> np.fmin.reduce >> >> For `nansum`, we'd need to add an ufunc `nanadd`, and for >> `nanargmax/min`, we'd need `argfmin/fmax'. > > How about that! I wasn't aware of fmax/fmin. Yes, I'd like a nanadd, please. > >>> arr = np.random.rand(1000, 1000) >>> arr[arr > 0.5] = np.nan >>> np.nanmax(arr) > 0.4625409581072 >>> np.fmax.reduce(arr, axis=None) > > TypeError: an integer is required >>> np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0) > 0.4625409581072 > >>> timeit np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0) > 100 loops, best of 3: 12.7 ms per loop >>> timeit np.nanmax(arr) > 10 loops, best of 3: 39.6 ms per loop > >>> timeit np.nanmax(arr, axis=0) > 10 loops, best of 3: 46.5 ms per loop >>> timeit np.fmax.reduce(arr, axis=0) > 100 loops, best of 3: 12.7 ms per loop Cython is faster than np.fmax.reduce. I wrote a cython version of np.nanmax, called nanmax below. (It only handles the 2d, float64, axis=None case, but since the array is large I don't think that explains the time difference). Note that fmax.reduce is slower than np.nanmax when there are no NaNs: >> arr = np.random.rand(1000, 1000) >> timeit np.nanmax(arr) 100 loops, best of 3: 5.82 ms per loop >> timeit np.fmax.reduce(np.fmax.reduce(arr)) 100 loops, best of 3: 9.14 ms per loop >> timeit nanmax(arr) 1000 loops, best of 3: 1.17 ms per loop >> arr[arr > 0.5] = np.nan >> timeit np.nanmax(arr) 10 loops, best of 3: 45.5 ms per loop >> timeit np.fmax.reduce(np.fmax.reduce(arr)) 100 loops, best of 3: 12.7 ms per loop >> timeit nanmax(arr) 1000 loops, best of 3: 1.17 ms per loop ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 12:19 PM, Pauli Virtanen wrote: > Fri, 19 Nov 2010 11:19:57 -0800, Keith Goodman wrote: > [clip] >> My guess is that having separate underlying functions for each dtype, >> ndim, and axis would be a nightmare for a large project like Numpy. But >> manageable for a focused project like nanny. > > Might be easier to migrate the nan* functions to using Ufuncs. > > Unless I'm missing something, > > np.nanmax -> np.fmax.reduce > np.nanmin -> np.fmin.reduce > > For `nansum`, we'd need to add an ufunc `nanadd`, and for > `nanargmax/min`, we'd need `argfmin/fmax'. How about that! I wasn't aware of fmax/fmin. Yes, I'd like a nanadd, please. >> arr = np.random.rand(1000, 1000) >> arr[arr > 0.5] = np.nan >> np.nanmax(arr) 0.4625409581072 >> np.fmax.reduce(arr, axis=None) TypeError: an integer is required >> np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0) 0.4625409581072 >> timeit np.fmax.reduce(np.fmax.reduce(arr, axis=0), axis=0) 100 loops, best of 3: 12.7 ms per loop >> timeit np.nanmax(arr) 10 loops, best of 3: 39.6 ms per loop >> timeit np.nanmax(arr, axis=0) 10 loops, best of 3: 46.5 ms per loop >> timeit np.fmax.reduce(arr, axis=0) 100 loops, best of 3: 12.7 ms per loop ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 12:10 PM, wrote: > What's the speed advantage of nanny compared to np.nansum that you > have if the arrays are larger, say (1000,10) or (1,100) axis=0 ? Good point. In the small examples I showed so far maybe the speed up was all in overhead. Fortunately, that's not the case: >> arr = np.random.rand(1000, 1000) >> timeit np.nansum(arr) 100 loops, best of 3: 4.79 ms per loop >> timeit ny.nansum(arr) 1000 loops, best of 3: 1.53 ms per loop >> arr[arr > 0.5] = np.nan >> timeit np.nansum(arr) 10 loops, best of 3: 44.5 ms per loop >> timeit ny.nansum(arr) 100 loops, best of 3: 6.18 ms per loop >> timeit np.nansum(arr, axis=0) 10 loops, best of 3: 52.3 ms per loop >> timeit ny.nansum(arr, axis=0) 100 loops, best of 3: 12.2 ms per loop np.nansum makes a copy of the input array and makes a mask (another copy) and then uses the mask to set the NaNs to zero in the copy. So not only is nanny faster, but it uses less memory. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
Fri, 19 Nov 2010 11:19:57 -0800, Keith Goodman wrote: [clip] > My guess is that having separate underlying functions for each dtype, > ndim, and axis would be a nightmare for a large project like Numpy. But > manageable for a focused project like nanny. Might be easier to migrate the nan* functions to using Ufuncs. Unless I'm missing something, np.nanmax -> np.fmax.reduce np.nanmin -> np.fmin.reduce For `nansum`, we'd need to add an ufunc `nanadd`, and for `nanargmax/min`, we'd need `argfmin/fmax'. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 2:35 PM, Keith Goodman wrote: > On Fri, Nov 19, 2010 at 11:12 AM, Benjamin Root wrote: > >> That's why I use masked arrays. It is dtype agnostic. >> >> I am curious if there are any lessons that were learned in making Nanny that >> could be applied to the masked array functions? > > I suppose you could write a cython function that operates on masked > arrays. But other than that, I can't think of any lessons. All I can > think about is speed: > >>> x = np.ma.array([[1, 2], [3, 4]], mask=[[0, 1], [1, 0]]) >>> timeit np.sum(x) > 1 loops, best of 3: 25.1 us per loop >>> a = np.array([[1, np.nan], [np.nan, 4]]) >>> timeit ny.nansum(a) > 10 loops, best of 3: 3.11 us per loop >>> from nansum import nansum_2d_float64_axisNone >>> timeit nansum_2d_float64_axisNone(a) > 100 loops, best of 3: 395 ns per loop What's the speed advantage of nanny compared to np.nansum that you have if the arrays are larger, say (1000,10) or (1,100) axis=0 ? Josef > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 11:12 AM, Benjamin Root wrote: > That's why I use masked arrays. It is dtype agnostic. > > I am curious if there are any lessons that were learned in making Nanny that > could be applied to the masked array functions? I suppose you could write a cython function that operates on masked arrays. But other than that, I can't think of any lessons. All I can think about is speed: >> x = np.ma.array([[1, 2], [3, 4]], mask=[[0, 1], [1, 0]]) >> timeit np.sum(x) 1 loops, best of 3: 25.1 us per loop >> a = np.array([[1, np.nan], [np.nan, 4]]) >> timeit ny.nansum(a) 10 loops, best of 3: 3.11 us per loop >> from nansum import nansum_2d_float64_axisNone >> timeit nansum_2d_float64_axisNone(a) 100 loops, best of 3: 395 ns per loop ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 10:55 AM, Nathaniel Smith wrote: > On Fri, Nov 19, 2010 at 10:33 AM, Keith Goodman wrote: >> Nanny uses the magic of Cython to give you a faster, drop-in replacement for >> the NaN functions in NumPy and SciPy. > > Neat! > > Why not make this a patch to numpy/scipy instead? My guess is that having separate underlying functions for each dtype, ndim, and axis would be a nightmare for a large project like Numpy. But manageable for a focused project like nanny. >> Nanny uses a separate Cython function for each combination of ndim, dtype, >> and >> axis. You can get rid of a lot of overhead (useful in an inner loop, e.g.) by >> directly importing the function that matches your problem:: >> >> >> arr = np.random.rand(10, 10) >> >> from nansum import nansum_2d_float64_axis1 > > If this is really useful, then better to provide a function that finds > the correct function for you? > > best_nansum = ny.get_best_nansum(ary[0, :, :], axis=1) > for i in xrange(ary.shape[0]): > best_nansum(ary[i, :, :], axis=1) That would be useful. It is what nanny.nansum does but it returns the sum instead of the function. >> - functions: nansum >> - Operating systems: 64-bit (accumulator for int32 is hard coded to int64) >> - dtype: int32, int64, float64 >> - ndim: 1, 2, and 3 > > What does it even mean to do NaN operations on integers? (I'd > sometimes find it *really convenient* if there were a NaN value for > standard computer integers... but there isn't?) Well, sometimes you write functions without knowing the dtype of the input. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 12:55 PM, Nathaniel Smith wrote: > On Fri, Nov 19, 2010 at 10:33 AM, Keith Goodman > wrote: > > Nanny uses the magic of Cython to give you a faster, drop-in replacement > for > > the NaN functions in NumPy and SciPy. > > Neat! > > Why not make this a patch to numpy/scipy instead? > > > Nanny uses a separate Cython function for each combination of ndim, > dtype, and > > axis. You can get rid of a lot of overhead (useful in an inner loop, > e.g.) by > > directly importing the function that matches your problem:: > > > >>> arr = np.random.rand(10, 10) > >>> from nansum import nansum_2d_float64_axis1 > > If this is really useful, then better to provide a function that finds > the correct function for you? > > best_nansum = ny.get_best_nansum(ary[0, :, :], axis=1) > for i in xrange(ary.shape[0]): >best_nansum(ary[i, :, :], axis=1) > > > - functions: nansum > > - Operating systems: 64-bit (accumulator for int32 is hard coded to > int64) > > - dtype: int32, int64, float64 > > - ndim: 1, 2, and 3 > > What does it even mean to do NaN operations on integers? (I'd > sometimes find it *really convenient* if there were a NaN value for > standard computer integers... but there isn't?) > > -- Nathaniel > That's why I use masked arrays. It is dtype agnostic. I am curious if there are any lessons that were learned in making Nanny that could be applied to the masked array functions? Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions
On Fri, Nov 19, 2010 at 10:33 AM, Keith Goodman wrote: > Nanny uses the magic of Cython to give you a faster, drop-in replacement for > the NaN functions in NumPy and SciPy. Neat! Why not make this a patch to numpy/scipy instead? > Nanny uses a separate Cython function for each combination of ndim, dtype, and > axis. You can get rid of a lot of overhead (useful in an inner loop, e.g.) by > directly importing the function that matches your problem:: > > >> arr = np.random.rand(10, 10) > >> from nansum import nansum_2d_float64_axis1 If this is really useful, then better to provide a function that finds the correct function for you? best_nansum = ny.get_best_nansum(ary[0, :, :], axis=1) for i in xrange(ary.shape[0]): best_nansum(ary[i, :, :], axis=1) > - functions: nansum > - Operating systems: 64-bit (accumulator for int32 is hard coded to int64) > - dtype: int32, int64, float64 > - ndim: 1, 2, and 3 What does it even mean to do NaN operations on integers? (I'd sometimes find it *really convenient* if there were a NaN value for standard computer integers... but there isn't?) -- Nathaniel ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] [ANN] Nanny, faster NaN functions
= Nanny = Nanny uses the magic of Cython to give you a faster, drop-in replacement for the NaN functions in NumPy and SciPy. For example:: >> import nanny as ny >> import numpy as np >> arr = np.random.rand(100, 100) >> timeit np.nansum(arr) 1 loops, best of 3: 67.5 us per loop >> timeit ny.nansum(arr) 10 loops, best of 3: 18.2 us per loop Let's not forget to add some NaNs:: >> arr[arr > 0.5] = np.nan >> timeit np.nansum(arr) 1000 loops, best of 3: 411 us per loop >> timeit ny.nansum(arr) 1 loops, best of 3: 65 us per loop Nanny uses a separate Cython function for each combination of ndim, dtype, and axis. You can get rid of a lot of overhead (useful in an inner loop, e.g.) by directly importing the function that matches your problem:: >> arr = np.random.rand(10, 10) >> from nansum import nansum_2d_float64_axis1 >> timeit np.nansum(arr, axis=1) 1 loops, best of 3: 25.5 us per loop >> timeit ny.nansum(arr, axis=1) 10 loops, best of 3: 5.15 us per loop >> timeit nansum_2d_float64_axis1(arr) 100 loops, best of 3: 1.75 us per loop I put together Nanny as a way to learn Cython. It currently only supports: - functions: nansum - Operating systems: 64-bit (accumulator for int32 is hard coded to int64) - dtype: int32, int64, float64 - ndim: 1, 2, and 3 If there is interest in the project, I could continue adding the remaining NaN functions from NumPy and SciPy: nanmin, nanmax, nanmean, nanmedian (using a partial sort), nanstd. But why stop there? How about nancumsum or nanprod? Or anynan, which could short-circuit once a NaN is found? Feedback on the code or the direction of the project are welcomed. So is coding help---without that I doubt the package will ever be completed. Once nansum is complete, many of the remaining functions will be copy, paste, touch up operations. Remember, Nanny quickly protects your precious data from the corrupting influence of Mr. Nan. License === Nanny is distributed under a Simplified BSD license. Parts of NumPy, which has a BSD licenses, are included in Nanny. See the LICENSE file, which is distributed with Nanny, for details. Installation You can grab Nanny at http://github.com/kwgoodman/nanny. nansum of ints is only supported by 64-bit operating systems at the moment. **GNU/Linux, Mac OS X, et al.** To install Nanny:: $ python setup.py build $ sudo python setup.py install Or, if you wish to specify where Nanny is installed, for example inside ``/usr/local``:: $ python setup.py build $ sudo python setup.py install --prefix=/usr/local **Windows** In order to compile the C code in Nanny you need a Windows version of the gcc compiler. MinGW (Minimalist GNU for Windows) contains gcc and has been used to successfully compile Nanny on Windows. Install MinGW and add it to your system path. Then install Nanny with the commands:: python setup.py build --compiler=mingw32 python setup.py install **Post install** After you have installed Nanny, run the suite of unit tests:: >>> import nanny >>> nanny.test() Ran 1 tests in 0.008s OK ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] seeking advice on a fast string->array conversion
On Tue, Nov 16, 2010 at 10:31 AM, Darren Dale wrote:
> On Tue, Nov 16, 2010 at 9:55 AM, Pauli Virtanen wrote:
>> Tue, 16 Nov 2010 09:41:04 -0500, Darren Dale wrote:
>> [clip]
>>> That loop takes 0.33 seconds to execute, which is a good start. I need
>>> some help converting this example to return an actual numpy array. Could
>>> anyone please offer a suggestion?
>>
>> Easiest way is probably to use ndarray buffers and resize them when
>> needed. For example:
>>
>> https://github.com/pv/scipy-work/blob/enh/interpnd-smooth/scipy/spatial/qhull.pyx#L980
>
> Thank you Pauli. That makes it *incredibly* simple:
>
> import time
> cimport numpy as np
> import numpy as np
>
> cdef extern from 'stdlib.h':
> double atof(char*)
>
>
> def test():
> py_string = '100'
> cdef char* c_string = py_string
> cdef int i, j
> cdef double val
> i = 0
> j = 2048*1200
> cdef np.ndarray[np.float64_t, ndim=1] ret
>
> ret_arr = np.empty((2048*1200,), dtype=np.float64)
> ret = ret_arr
>
> d = time.time()
> while i c_string = py_string
> ret[i] = atof(c_string)
> i += 1
> ret_arr.shape = (1200, 2048)
> print ret_arr, ret_arr.shape, time.time()-d
>
> The loop now takes only 0.11 seconds to execute. Thanks again.
>
One follow-up issue: I can't cythonize this code for python-3. I've
installed numpy with the most recent changes to the 1.5.x maintenance
branch, then re-installed cython-0.13, but when I run "python3
setup.py build_ext --inplace" with this setup script:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [
Extension(
"test_open", ["test_open.pyx"], include_dirs=[numpy.get_include()]
)
]
)
I get the following error. Any suggestions what I need to fix, or
should I report it to the cython list?
$ python3 setup.py build_ext --inplace
running build_ext
cythoning test_open.pyx to test_open.c
Error converting Pyrex file to C:
...
# For use in situations where ndarray can't replace PyArrayObject*,
# like PyArrayObject**.
pass
ctypedef class numpy.ndarray [object PyArrayObject]:
cdef __cythonbufferdefaults__ = {"mode": "strided"}
^
/Users/darren/.local/lib/python3.1/site-packages/Cython/Includes/numpy.pxd:173:49:
"mode" is not a buffer option
Error converting Pyrex file to C:
...
cdef char* c_string = py_string
cdef int i, j
cdef double val
i = 0
j = 2048*1200
cdef np.ndarray[np.float64_t, ndim=1] ret
^
/Users/darren/temp/test/test_open.pyx:16:8: 'ndarray' is not a type identifier
building 'test_open' extension
/usr/bin/gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g
-fwrapv -O3 -Wall -Wstrict-prototypes
-I/Users/darren/.local/lib/python3.1/site-packages/numpy/core/include
-I/opt/local/Library/Frameworks/Python.framework/Versions/3.1/include/python3.1
-c test_open.c -o build/temp.macosx-10.6-x86_64-3.1/test_open.o
test_open.c:1:2: error: #error Do not use this file, it is the result
of a failed Cython compilation.
error: command '/usr/bin/gcc-4.2' failed with exit status 1
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Advise for numerical programming content (New python user)
Hi, On 19 November 2010 03:48, Sachin Kumar Sharma wrote: > Does graphic output like maps, histogram, crossplot, tornado charts is good > enough with basic installation or needs some additional packages? You might want to ask this question at the scipy mailig-list. For maps, you need basemap or PyNCL. I am presently using basemap. In the future, I will look at PyNCL, Python bindings to NCL. I think there are also Python bindings to GMT. Interestingly, both basemap and PyNCL are written by the same person (Jeff Whitaker). Probably those packages complement each other rather than doing the same thing. Others might correct me here. Gerrit. -- Exploring space at http://gerrit-explores.blogspot.com/ Personal homepage at http://www.topjaklont.org/ Asperger Syndroom: http://www.topjaklont.org/nl/asperger.html ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] seeking advice on a fast string->array conversion
Sorry, I accidentally hit send long before I was finished writing. But to answer your question, they contain many 2048-element multi-channel analyzer spectra. Darren On Tue, Nov 16, 2010 at 9:26 AM, william ratcliff wrote: > Actually, > I do use spec when I have synchotron experiments. But why are your files so > large? > > On Nov 16, 2010 9:20 AM, "Darren Dale" wrote: >> I am wrapping up a small package to parse a particular ascii-encoded >> file format generated by a program we use heavily here at the lab. (In >> the unlikely event that you work at a synchrotron, and use Certified >> Scientific's "spec" program, and are actually interested, the code is >> currently available at >> https://github.com/darrendale/praxes/tree/specformat/praxes/io/spec/ >> .) >> >> I have been benchmarking the project against another python package >> developed by a colleague, which is an extension module written in pure >> C. My python/cython project takes about twice as long to parse and >> index a file (~0.8 seconds for 100MB), which is acceptable. However, >> actually converting ascii strings to numpy arrays, which is done using >> numpy.fromstring, takes a factor of 10 longer than the extension >> module. So I am wondering about the performance of np.fromstring: >> >> import time >> import numpy as np >> s = b'1 ' * 2048 *1200 >> d = time.time() >> x = np.fromstring(s) >> print time.time() - d >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
