Re: [Numpy-discussion] merge_arrays is very slow; alternatives?
On Fri, 26 Nov 2010 20:57:30 +0100, Gerrit Holl wrote: [clip] > I wonder, am I missing something or have I really written a significant > improvement in less than 10 LOC? Should I file a patch for this? The implementation of merge_arrays doesn't look optimal -- it seems to actually iterate over the data, which should not be needed. So yes, rewriting the function would be useful. The main difficulty in the rewrite seems to be appropriate mask handling, but slicing is a faster way to do that. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] merge_arrays is very slow; alternatives?
On 26 November 2010 20:16, Gerrit Holl wrote: > Hi, > > upon profiling my code, I found that > numpy.lib.recfunctions.merge_arrays is extremely slow; it does some > 7000 rows/second. This is not acceptable for me. ... > How can I do this in a faster way? Replying to my own code here. Either I have written a much faster implementation of this, or I am missing something. I consider it unlikely that I write a much faster implementation of an established numpy function with little effort, so I suspect I am missing something here. I wrote this implementation of the flattened version of merge_arrays: def merge_arrays(arr1, arr2): t1 = arr1.dtype t2 = arr2.dtype newdtype = numpy.dtype(t1.descr + t2.descr) newarray = numpy.empty(shape=arr1.shape, dtype=newdtype) for field in t1.names: newarray[field] = arr1[field] for field in t2.names: newarray[field] = arr2[field] return newarray and benchmarks show it's almost 100 times faster for a medium-sized array: In [211]: %timeit merged1 = numpy.lib.recfunctions.merge_arrays([metarows[:1], targetrows2[:1]], flatten=True) 1 loops, best of 3: 1.01 s per loop In [212]: %timeit merged2 = pyatmlab.tools.merge_arrays(metarows[:1], targetrows2[:1]) 100 loops, best of 3: 10.8 ms per loop In [214]: (merged1.view(dtype=uint64).reshape(-1, 100) == merged2.view(dtype=uint64).reshape(-1, 100)).all() Out[214]: True # and still 4 times faster for a small array: In [215]: %timeit merged1 = numpy.lib.recfunctions.merge_arrays([metarows[:10], targetrows2[:10]], flatten=True) 1000 loops, best of 3: 1.31 ms per loop In [216]: %timeit merged2 = pyatmlab.tools.merge_arrays(metarows[:10], targetrows2[:10]) 1000 loops, best of 3: 344 us per loop # and 15 times faster for a large array (1.5 million elements): In [218]: %timeit merged1 = numpy.lib.recfunctions.merge_arrays([metarows, targetrows2], flatten=True) 1 loops, best of 3: 110 s per loop In [217]: %timeit merged2 = pyatmlab.tools.merge_arrays(metarows, targetrows2) 1 loops, best of 3: 7.26 s per loop I wonder, am I missing something or have I really written a significant improvement in less than 10 LOC? Should I file a patch for this? Gerrit. -- Exploring space at http://gerrit-explores.blogspot.com/ Personal homepage at http://www.topjaklont.org/ Asperger Syndroom: http://www.topjaklont.org/nl/asperger.html ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] merge_arrays is very slow; alternatives?
Hi,
upon profiling my code, I found that
numpy.lib.recfunctions.merge_arrays is extremely slow; it does some
7000 rows/second. This is not acceptable for me.
I have two large record arrays, or arrays with a complicated dtype.
All I want to do is to merge them into one. I don't think that should
have to be a very slow operation, I don't need to copy anything, I
just want to view the two record arrays as one.
How can I do this in a faster way?
In [45]: cProfile.runctx("numpy.lib.recfunctions.merge_arrays([metarows,
targetrows2], flatten=True)", globals(), locals())
225381902 function calls (150254635 primitive calls) in
166.620 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.031 0.031 166.620 166.620 :1()
68/1 0.000 0.000 0.000 0.000 _internal.py:82(_array_descr)
2 0.000 0.000 0.000 0.000 numeric.py:286(asanyarray)
2 0.000 0.000 0.000 0.000 recfunctions.py:135(flatten_descr)
1 0.000 0.000 0.001 0.001 recfunctions.py:161(zip_descr)
149165600/74038400 117.195 0.000 139.701 0.000
recfunctions.py:235(_izip_fields_flat)
1088801 12.146 0.000 151.847 0.000 recfunctions.py:263(izip_records)
3 0.000 0.000 0.000 0.000 recfunctions.py:277(sentinel)
1 4.599 4.599 166.589 166.589 recfunctions.py:328(merge_arrays)
3 0.000 0.000 0.000 0.000 recfunctions.py:406()
75127201 22.506 0.000 22.506 0.000 {isinstance}
69 0.000 0.000 0.000 0.000 {len}
1 0.000 0.000 0.000 0.000 {map}
1 0.000 0.000 0.000 0.000 {max}
2 0.000 0.000 0.000 0.000 {method '__array__' of
'numpy.ndarray' objects}
136 0.000 0.000 0.000 0.000 {method 'append' of
'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of
'_lsprof.Profiler' objects}
2 0.000 0.000 0.000 0.000 {method 'extend' of
'list' objects}
2 0.000 0.000 0.000 0.000 {method 'pop' of 'list' objects}
2 0.000 0.000 0.000 0.000 {method 'ravel' of
'numpy.ndarray' objects}
2 0.000 0.000 0.000 0.000 {numpy.core.multiarray.array}
1 10.142 10.142 10.142 10.142 {numpy.core.multiarray.fromiter}
Gerrit.
--
Gerrit Holl
PhD student at Department of Space Science, Luleå University of
Technology, Kiruna, Sweden
http://www.sat.ltu.se/members/gerrit/
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
