[Numpy-discussion] how to do this efficiently?
In a 1-d array, find the first point where all subsequent points have values less than a threshold, T. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
> In a 1-d array, find the first point where all subsequent points > have values > less than a threshold, T. Maybe something like: last_greater = numpy.arange(arr.shape)[arr >= T][-1] first_lower = last_greater + 1 There's probably a better way to do it, without the arange, though... ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
Zachary Pincus wrote: >> In a 1-d array, find the first point where all subsequent points >> have values >> less than a threshold, T. > > Maybe something like: > > last_greater = numpy.arange(arr.shape)[arr >= T][-1] > first_lower = last_greater + 1 > > There's probably a better way to do it, without the arange, though... I'm trying to find the first point in a power spectrum such that all subsequent points are below some level. I've started with: db is my power spectrum in dB, It is already reversed. mag = np.maximum.accumulate (db) - db[-1] Now all I need is to find the first point such that mag < -50. How to do this efficiently? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
>>> In a 1-d array, find the first point where all subsequent points >>> have values >>> less than a threshold, T. >> >> Maybe something like: >> >> last_greater = numpy.arange(arr.shape)[arr >= T][-1] >> first_lower = last_greater + 1 >> >> There's probably a better way to do it, without the arange, though... > > I'm trying to find the first point in a power spectrum such that all > subsequent > points are below some level. I've started with: > > db is my power spectrum in dB, It is already reversed. > > mag = np.maximum.accumulate (db) - db[-1] > > Now all I need is to find the first point such that mag < -50. How > to do this > efficiently? Right -- that's what I showed above. Find the last point in mag that is >= -50, and by definition the next point is the first point such that the remainder of mag is < -50. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
Zachary Pincus wrote: In a 1-d array, find the first point where all subsequent points have values less than a threshold, T. >>> >>> Maybe something like: >>> >>> last_greater = numpy.arange(arr.shape)[arr >= T][-1] >>> first_lower = last_greater + 1 >>> >>> There's probably a better way to do it, without the arange, though... >> >> I'm trying to find the first point in a power spectrum such that all >> subsequent >> points are below some level. I've started with: >> >> db is my power spectrum in dB, It is already reversed. >> >> mag = np.maximum.accumulate (db) - db[-1] >> >> Now all I need is to find the first point such that mag < -50. How >> to do this >> efficiently? > > Right -- that's what I showed above. Find the last point in mag that > is >= -50, and by definition the next point is the first point such > that the remainder of mag is < -50. But where is numpy's 'find_first' function? I can't seem to find it so I had to make my own in C++. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
On Feb 9, 2011, at 10:58 AM, Neal Becker wrote: > Zachary Pincus wrote: > > In a 1-d array, find the first point where all subsequent points > have values > less than a threshold, T. Maybe something like: last_greater = numpy.arange(arr.shape)[arr >= T][-1] first_lower = last_greater + 1 There's probably a better way to do it, without the arange, though... >>> >>> I'm trying to find the first point in a power spectrum such that all >>> subsequent >>> points are below some level. I've started with: >>> >>> db is my power spectrum in dB, It is already reversed. >>> >>> mag = np.maximum.accumulate (db) - db[-1] >>> >>> Now all I need is to find the first point such that mag < -50. How >>> to do this >>> efficiently? >> >> Right -- that's what I showed above. Find the last point in mag that >> is >= -50, and by definition the next point is the first point such >> that the remainder of mag is < -50. > > But where is numpy's 'find_first' function? I can't seem to find it > so I had to > make my own in C++. As before, the line below does what you said you need, though not maximally efficiently. (Try it in an interpreter...) There may be another way in numpy that doesn't rely on constructing the index array, but this is the first thing that came to mind. last_greater = numpy.arange(arr.shape)[arr >= T][-1] Let's unpack that dense line a bit: mask = arr >= T indices = numpy.arange(arr.shape) above_threshold_indices = indices[mask] last_above_threshold_index = above_threshold_indices[-1] Does this make sense? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
On 2/9/2011 10:58 AM, Neal Becker wrote: > But where is numpy's 'find_first' function? np.argmax(array>T) (Of course that constructs a boolean array...) Alan Isaac ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
On Wed, Feb 9, 2011 at 11:04 AM, Zachary Pincus wrote: > > On Feb 9, 2011, at 10:58 AM, Neal Becker wrote: > >> Zachary Pincus wrote: >> >> In a 1-d array, find the first point where all subsequent points >> have values >> less than a threshold, T. > > Maybe something like: > > last_greater = numpy.arange(arr.shape)[arr >= T][-1] > first_lower = last_greater + 1 > > There's probably a better way to do it, without the arange, > though... I'm trying to find the first point in a power spectrum such that all subsequent points are below some level. I've started with: db is my power spectrum in dB, It is already reversed. mag = np.maximum.accumulate (db) - db[-1] Now all I need is to find the first point such that mag < -50. How to do this efficiently? >>> >>> Right -- that's what I showed above. Find the last point in mag that >>> is >= -50, and by definition the next point is the first point such >>> that the remainder of mag is < -50. >> >> But where is numpy's 'find_first' function? I can't seem to find it >> so I had to >> make my own in C++. > > > As before, the line below does what you said you need, though not > maximally efficiently. (Try it in an interpreter...) There may be > another way in numpy that doesn't rely on constructing the index > array, but this is the first thing that came to mind. > > last_greater = numpy.arange(arr.shape)[arr >= T][-1] > > Let's unpack that dense line a bit: > > mask = arr >= T > indices = numpy.arange(arr.shape) > above_threshold_indices = indices[mask] > last_above_threshold_index = above_threshold_indices[-1] > > Does this make sense? This assumes monotonicity. Is that allowed? Josef > > > > > > > > > > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
>> As before, the line below does what you said you need, though not >> maximally efficiently. (Try it in an interpreter...) There may be >> another way in numpy that doesn't rely on constructing the index >> array, but this is the first thing that came to mind. >> >> last_greater = numpy.arange(arr.shape)[arr >= T][-1] >> >> Let's unpack that dense line a bit: >> >> mask = arr >= T >> indices = numpy.arange(arr.shape) >> above_threshold_indices = indices[mask] >> last_above_threshold_index = above_threshold_indices[-1] >> >> Does this make sense? > > This assumes monotonicity. Is that allowed? The twice-stated problem was: > In a 1-d array, find the first point where all subsequent points > have values less than a threshold, T. So that should do the trick... Though Alan's argmax solution is definitely a better one than indexing an indices array. Same logic and result though, just more compact. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
>> This assumes monotonicity. Is that allowed? > > The twice-stated problem was: [Note to avert email-miscommunications] BTW, I wasn't trying to snipe at you with that comment, Josef... I just meant to say that this solution solves the problem as Neal posed it, though that might not be the exact problem he has. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
On 02/09/2011 10:17 AM, Zachary Pincus wrote: >>> As before, the line below does what you said you need, though not >>> maximally efficiently. (Try it in an interpreter...) There may be >>> another way in numpy that doesn't rely on constructing the index >>> array, but this is the first thing that came to mind. >>> >>> last_greater = numpy.arange(arr.shape)[arr>= T][-1] >>> >>> Let's unpack that dense line a bit: >>> >>> mask = arr>= T >>> indices = numpy.arange(arr.shape) >>> above_threshold_indices = indices[mask] >>> last_above_threshold_index = above_threshold_indices[-1] >>> >>> Does this make sense? >> This assumes monotonicity. Is that allowed? > The twice-stated problem was: > >> In a 1-d array, find the first point where all subsequent points >> have values less than a threshold, T. > So that should do the trick... Though Alan's argmax solution is > definitely a better one than indexing an indices array. Same logic and > result though, just more compact. > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion This is similar to Zachary's as it just uses 'np.where'. >>> x=np.array([5,4,3,6,7,3,2,1]) >>> x array([5, 4, 3, 6, 7, 3, 2, 1]) >>> np.argmax(x>5) # doesn't appear to be correct 3 >>> np.where(x>5)[0][-1]#since np.where gives a tuple and you need the last element 4 This should give the index where all subsequent points are less than some threshold. However, this and Zachary's version fail when if all points are lower than the threshold (eg T=10 in this array). Bruce ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
Zachary Pincus wrote: > > On Feb 9, 2011, at 10:58 AM, Neal Becker wrote: > >> Zachary Pincus wrote: >> >> In a 1-d array, find the first point where all subsequent points >> have values >> less than a threshold, T. > > Maybe something like: > > last_greater = numpy.arange(arr.shape)[arr >= T][-1] > first_lower = last_greater + 1 > > There's probably a better way to do it, without the arange, > though... I'm trying to find the first point in a power spectrum such that all subsequent points are below some level. I've started with: db is my power spectrum in dB, It is already reversed. mag = np.maximum.accumulate (db) - db[-1] Now all I need is to find the first point such that mag < -50. How to do this efficiently? >>> >>> Right -- that's what I showed above. Find the last point in mag that >>> is >= -50, and by definition the next point is the first point such >>> that the remainder of mag is < -50. >> >> But where is numpy's 'find_first' function? I can't seem to find it >> so I had to >> make my own in C++. > > > As before, the line below does what you said you need, though not > maximally efficiently. (Try it in an interpreter...) There may be > another way in numpy that doesn't rely on constructing the index > array, but this is the first thing that came to mind. > > last_greater = numpy.arange(arr.shape)[arr >= T][-1] > > Let's unpack that dense line a bit: > > mask = arr >= T > indices = numpy.arange(arr.shape) > above_threshold_indices = indices[mask] > last_above_threshold_index = above_threshold_indices[-1] > > Does this make sense? Yes, thanks for the explanation. Still, I wonder if numpy would benefit from a 'find_first' function. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
On 2/9/2011 11:39 AM, Bruce Southey wrote: > np.argmax(x>5) # doesn't appear to be correct It was an answer to the particular question of how to do find_first, which it does (at the cost of a boolean array): it finds the first element greater than 5. >>> x array([5, 4, 3, 6, 7, 3, 2, 1]) >>> np.argmax(x>5) 3 >>> x[3] 6 You can also find the first element less than 4, etc. >>> np.argmax(x<4) 2 The condition should be chosen for the problem. As Josef pointed out, the usefulness of this and other approaches can depend on the monotonicity of the series. fwiw, Alan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
On Wed, Feb 9, 2011 at 10:48, Neal Becker wrote: > Still, I wonder if numpy would benefit from a 'find_first' function. I certainly could have used one many times. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
On Wed, Feb 9, 2011 at 11:25 AM, Zachary Pincus wrote: >>> This assumes monotonicity. Is that allowed? >> >> The twice-stated problem was: > > [Note to avert email-miscommunications] BTW, I wasn't trying to snipe > at you with that comment, Josef... > > I just meant to say that this solution solves the problem as Neal > posed it, though that might not be the exact problem he has. >> In a 1-d array, find the first point where all subsequent points have values >> less than a threshold. This doesn't imply monotonicity. Suppose with have a sin curve, and I want to find the last trough. Or a business cycle and I want to find the last recession. Unless my english deteriorated recently. :) Josef > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
> >>> In a 1-d array, find the first point where all subsequent points >>> have values >>> less than a threshold. > > This doesn't imply monotonicity. > Suppose with have a sin curve, and I want to find the last trough. Or > a business cycle and I want to find the last recession. > > Unless my english deteriorated recently. > Not sure that I follow? I read the statement as: find the smallest non-negative index i such that array[j] < t for all i <= j < len(array) for which the various solutions proposed work, except for the corner case where array.min() >= t. I'm not sure where monotonicity comes into play, unless one interprets the "all subsequent points" clause as "some number of subsequent points" or something... For those sort of problems, I usually wind up smoothing the array [optional], and then using scipy.ndimage.maximum to calculate peak values within a given window, and then find the points that equal the peak values -- this works pretty robustly for peak/trough detection in arbitrary dimension. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
On Wed, Feb 9, 2011 at 1:18 PM, Zachary Pincus wrote: >> In a 1-d array, find the first point where all subsequent points have values less than a threshold. >> >> This doesn't imply monotonicity. >> Suppose with have a sin curve, and I want to find the last trough. Or >> a business cycle and I want to find the last recession. >> >> Unless my english deteriorated recently. >> > > Not sure that I follow? I read the statement as: > find the smallest non-negative index i such that array[j] < t for all > i <= j < len(array) > > for which the various solutions proposed work, except for the corner > case where array.min() >= t. > > I'm not sure where monotonicity comes into play, unless one interprets > the "all subsequent points" clause as "some number of subsequent > points" or something... > > For those sort of problems, I usually wind up smoothing the array > [optional], and then using scipy.ndimage.maximum to calculate peak > values within a given window, and then find the points that equal the > peak values -- this works pretty robustly for peak/trough detection in > arbitrary dimension. (Sorry, I should reply when I'm not able to actually pay attention) your code works fine with a hump, I didn't read the example carefully enough >>> x = np.arange(10,0,-1) >>> x2 = 25-(x-5)**2 >>> x2 array([ 0, 9, 16, 21, 24, 25, 24, 21, 16, 9]) >>> T=20 >>> np.argmax(x2>> np.arange(x2.shape[0])[x2 >= T][-1]+1 8 >>> len(x)-np.argmax(x2[::]>T) + 1 8 Josef > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
I recently needed a find_first function and searched the list. It has been discussed a few times before. And I haven't seen an elegant solution yet. Maybe we should file a feature request, as it would be awfully nice to have? Mark > > > Still, I wonder if numpy would benefit from a 'find_first' function. > > I certainly could have used one many times. > > -- > Robert Kern > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] how to do this efficiently?
Mark Bakker wrote: > I recently needed a find_first function and searched the list. > It has been discussed a few times before. > And I haven't seen an elegant solution yet. > Maybe we should file a feature request, as it would be awfully nice to have? > Mark > >> >> > Still, I wonder if numpy would benefit from a 'find_first' function. >> >> I certainly could have used one many times. >> >> -- >> Robert Kern >> >> http://projects.scipy.org/numpy/ticket/1738 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
