Jake VanderPlas wrote: > It sounds like all of this could be done very simply without going to > C, using a class based on numpy.ndarray. The following works for 1D > arrays, behaves like a regular 1D numpy array, and could be easily > improved with a little care. Is this what you had in mind? > > import numpy > > #easy scalable array class > class scarray: > def __init__(self,*args,**kwargs): > self.__data = numpy.ndarray(*args,**kwargs) > > def append(self,val): > tmp = self.__data > self.__data = numpy.ndarray(tmp.size+1) > self.__data[:-1] = tmp > self.__data[-1] = val > del tmp > > def __getattr__(self,attr): > return getattr(self.__data,attr)
The problem here is that it's re-allocating memory with every single addition of an element. It's pretty common to pre-allocate some extra space for this kind of thing (python lists, std vectors, etc, etc, etc), so I assumed that it would be performance killer. However, you know what they say about premature optimization, so a test or two is in order. This is incrementally adding 10000 integers, one point at a time: Using the suggested code: In [21]: timeit p.scarray1(10000) 10 loops, best of 3: 1.71 s per loop Using my accumulator code: In [23]: timeit p.accum1(10000) 10 loops, best of 3: 25.6 ms per loop So all that memory re-allocation really does kill performance. In [24]: timeit p.list1(10000) 100 loops, best of 3: 9.96 ms per loop But, of course, lists are still faster. I think this is because I'm adding python integers, which are already python objects, so that's exactly what lists are for -- you can't beat them. This wouldn't apply to using them from C, however. Also, I see a change when we add chunks of data already in a numpy array, with .extend(): # adding a sequence of ten integers at a time: In [40]: timeit profile_accumulator.accum_extend1(10000) 100 loops, best of 3: 6.36 ms per loop In [41]: timeit profile_accumulator.accum_extend1(10000) 100 loops, best of 3: 6.22 ms per loop # about the same speed # but when I add 100 elements at a time: In [46]: timeit profile_accumulator.list_extend1(10000) 10 loops, best of 3: 56.6 ms per loop In [47]: timeit profile_accumulator.accum_extend1(10000) 100 loops, best of 3: 13.3 ms per loop # I start to see a real advantage to the numpy accumulator approach. Is # that a real use-case? I'm not sure. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [email protected] _______________________________________________ NumPy-Discussion mailing list [email protected] http://mail.scipy.org/mailman/listinfo/numpy-discussion
