[Numpy-discussion] Pre-allocate array
Hello, I have an array that I know will need to grow to X elements. However, I will need to work with it before it's completely filled. I see two ways of doing this: bigarray = np.empty(X) current_size = 0 for i in something: buf = produce_data(i) bigarray[current_size:current_size+len(buf)] = buf current_size += len(buf) # Do things with bigarray[:current_size] This avoids having to allocate new buffers and copying data around, but I have to separately manage the current array size. Alternatively, I could do bigarray = np.empty(0) current_size = 0 for i in something: buf = produce_data(i) bigarray.resize(len(bigarray)+len(buf)) bigarray[-len(buf):] = buf # Do things with bigarray this is much more elegant, but the resize() calls may have to copy data around. Is there any way to tell numpy to allocate all the required memory while using only a part of it for the array? Something like: bigarray = np.empty(50, will_grow_to=X) bigarray.resize(X) # Guaranteed to work without copying stuff around Thanks, -Nikolaus ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] List with numpy semantics
Gerrit Holl gerrit.h...@gmail.com writes: On 31 October 2010 17:10, Nikolaus Rath nikol...@rath.org wrote: Hello, I have a couple of numpy arrays which belong together. Unfortunately they have different dimensions, so I can't bundle them into a higher dimensional array. My solution was to put them into a Python list instead. But unfortunately this makes it impossible to use any ufuncs. Has someone else encountered a similar problem and found a nice solution? Something like a numpy list maybe? You could try a record array with a clever dtype, maybe? It seems that this requires more cleverness than I have... Could you give me an example? How do I replace l in the following code with a record array? l = list() l.append(np.arange(3)) l.append(np.arange(42)) l.append(np.arange(9)) for i in range(len(l)): l[i] += 32 Thanks, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] List with numpy semantics
Hello, I have a couple of numpy arrays which belong together. Unfortunately they have different dimensions, so I can't bundle them into a higher dimensional array. My solution was to put them into a Python list instead. But unfortunately this makes it impossible to use any ufuncs. Has someone else encountered a similar problem and found a nice solution? Something like a numpy list maybe? Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Find insertion point
Hello, I want to find the first i such that x[i] y and x[i+1] = y. Is there a way to do this without using a Python loop? I can't use np.searchsorted(), because my x array crosses y several times. Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Find insertion point
Lane Brooks l...@brooks.nu writes: On 08/17/2010 09:53 AM, Nikolaus Rath wrote: Hello, I want to find the first i such that x[i] y and x[i+1]= y. Is there a way to do this without using a Python loop? I can't use np.searchsorted(), because my x array crosses y several times. Best, -Nikolaus i = numpy.nonzero(x=y)[0][0] I'm afraid that also fails the x = [1,2,3,2,3,4], y=3 test. -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Find insertion point
Warren Weckesser warren.weckes...@enthought.com writes: Nikolaus Rath wrote: Hello, I want to find the first i such that x[i] y and x[i+1] = y. Is there a way to do this without using a Python loop? I can't use np.searchsorted(), because my x array crosses y several times. In [34]: np.where((x[:-1] y) (x[1:] y))[0][0] Out[34]: 2 Neat, thanks! -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] proposing a beware of [as]matrix() warning
Robert Kern robert.k...@gmail.com writes: Overloading '*' and '**' while convenient does have consequences. It would be nice if we could have a few more infix operators in Python to allow separation of element-by-element calculations and dot-product calculations. http://www.python.org/dev/peps/pep-0225/ was considered and rejected. But that was in 2000... While I don't have any spare cycles to push it forward and we are already far along on the NumPy to 3.0, I had wondered if we couldn't use the leverage of Python core developers wanting NumPy to be ported to Python 3 to actually add a few more infix operators to the language. I don't think that stands a chance: http://www.python.org/dev/peps/pep-3003/ Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Find indices of largest elements
eat e.antero.ta...@gmail.com writes: How do I best find out the indices of the largest x elements in an array? Just a= np.asarray([[1, 8, 2], [2, 1, 3]]) print np.where((a.T== a.max(axis= 1)).T) However, if any row contains more than 1 max entity, above will fail. Please let me to know if that's relevant for you. Not quite, because I'm interested in the n largest values over all elements, not the largest element in each row or column. But Keith's solution seems to work fine, even though I'm still struggling to understand what's going on there :-). Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Find indices of largest elements
Keith Goodman kwgood...@gmail.com writes: On Wed, Apr 14, 2010 at 12:39 PM, Nikolaus Rath nikol...@rath.org wrote: Keith Goodman kwgood...@gmail.com writes: On Wed, Apr 14, 2010 at 8:49 AM, Keith Goodman kwgood...@gmail.com wrote: On Wed, Apr 14, 2010 at 8:16 AM, Nikolaus Rath nikol...@rath.org wrote: Hello, How do I best find out the indices of the largest x elements in an array? Example: a = [ [1,8,2], [2,1,3] ] magic_function(a, 2) == [ (0,1), (1,2) ] Since the largest 2 elements are at positions (0,1) and (1,2). Here's a quick way to rank the data if there are no ties and no NaNs: ...or if you need the indices in order: shape = (3,2) x = np.random.rand(*shape) x array([[ 0.52420123, 0.43231286], [ 0.97995333, 0.87416228], [ 0.71604075, 0.66018382]]) r = x.reshape(-1).argsort().argsort() I don't understand why this works. Why do you call argsort() twice? Doesn't that give you the indices of the sorted indices? It is confusing. Let's look at an example: x = np.random.rand(4) x array([ 0.37412289, 0.68248559, 0.12935131, 0.42510212]) If we call argsort once we get the index that will sort x: idx = x.argsort() idx array([2, 0, 3, 1]) x[idx] array([ 0.12935131, 0.37412289, 0.42510212, 0.68248559]) Notice that the first element of idx is 2. That's because element x[2] is the min of x. But that's not what we want. I think that's exactly what I want, the index of the smallest element. It also seems to work: In [3]: x = np.random.rand(3,3) In [4]: x Out[4]: array([[ 0.49064281, 0.54989584, 0.05319183], [ 0.50510206, 0.39683101, 0.22801874], [ 0.04595144, 0.3329171 , 0.61156205]]) In [5]: idx = x.reshape(-1).argsort() In [6]: [ np.unravel_index(i, x.shape) for i in idx[-3:] ] Out[6]: [(1, 0), (0, 1), (2, 2)] So why the additional complication with the second argsort? I just don't get it... We want the first element to be the rank of the first element of x. I'm not quite sure why we want that...? Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Find indices of largest elements
eat e.antero.ta...@gmail.com writes: Nikolaus Rath Nikolaus at rath.org writes: [snip] Not quite, because I'm interested in the n largest values over all elements, not the largest element in each row or column. But Keith's solution seems to work fine, even though I'm still struggling to understand what's going on there . My bad. I just concentrated on your example, not the actual question. However, what's wrong with your above approach [ np.unravel_index(i, x.shape) for i in idx[-3:] ] ? Especially if your n largest elements are just a small fraction of all elements. The fact that it sorts the entire list. But since for my arrays it's not really an efficiency problem, I will use it anyway. Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Find indices of largest elements
Hello, How do I best find out the indices of the largest x elements in an array? Example: a = [ [1,8,2], [2,1,3] ] magic_function(a, 2) == [ (0,1), (1,2) ] Since the largest 2 elements are at positions (0,1) and (1,2). Best, -Niko -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Find indices of largest elements
Keith Goodman kwgood...@gmail.com writes: On Wed, Apr 14, 2010 at 8:49 AM, Keith Goodman kwgood...@gmail.com wrote: On Wed, Apr 14, 2010 at 8:16 AM, Nikolaus Rath nikol...@rath.org wrote: Hello, How do I best find out the indices of the largest x elements in an array? Example: a = [ [1,8,2], [2,1,3] ] magic_function(a, 2) == [ (0,1), (1,2) ] Since the largest 2 elements are at positions (0,1) and (1,2). Here's a quick way to rank the data if there are no ties and no NaNs: ...or if you need the indices in order: shape = (3,2) x = np.random.rand(*shape) x array([[ 0.52420123, 0.43231286], [ 0.97995333, 0.87416228], [ 0.71604075, 0.66018382]]) r = x.reshape(-1).argsort().argsort() I don't understand why this works. Why do you call argsort() twice? Doesn't that give you the indices of the sorted indices? Thanks, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
