On Tue, Oct 23, 2012 at 8:06 PM, Oscar Benjamin <oscar.j.benja...@gmail.com> wrote: > On 23 October 2012 15:31, Virgil Stokes <v...@it.uu.se> wrote: >> I am working with some rather large data files (>100GB) that contain time >> series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII >> format. I perform various types of processing on these data (e.g. moving >> median, moving average, and Kalman-filter, Kalman-smoother) in a sequential >> manner and only a small number of these data need be stored in RAM when >> being processed. When performing Kalman-filtering (forward in time pass, k = >> 0,1,...,N) I need to save to an external file several variables (e.g. 11*32 >> bytes) for each (t_k, y(t_k)). These are inputs to the Kalman-smoother >> (backward in time pass, k = N,N-1,...,0). Thus, I will need to input these >> variables saved to an external file from the forward pass, in reverse order >> --- from last written to first written. >> >> Finally, to my question --- What is a fast way to write these variables to >> an external file and then read them in backwards? > > You mentioned elsewhere that you are using numpy. I'll assume that the > data you want to read/write are numpy arrays.
If that is the case always timeit. The following is an example of 3 functions, with repetitions of time that give an average: import timeit #3 dimensional matrix x_dim = -1 y_dim = -1 z_dim = -1 s = """\ x_dim = -1 y_dim = -1 z_dim = -1 dict_1 = {} for i in xrange(0,6): x_dim = 1 y_dim = 1 z_dim = 1 dict_1['%s' % (i) ] = ['x = %i' % (x_dim), 'y = %i' % (y_dim), 'z = %i' % (z_dim)] """ t = """\ import numpy numpy.array([[ 1., 0., 0.], [ 0., 1., 2.]]) """ u = """\ list_count = 0 an_array = [] for i in range(0,10): if list_count > 3: break if i % 3 != 0: an_array.append(i) if i % 3 == 0: list_count += 1 """ print timeit.timeit(stmt=s, number=100000) print timeit.timeit(stmt=t, number=100000) print timeit.timeit(stmt=u, number=100000) -- Best Regards, David Hutto CEO: http://www.hitwebdevelopment.com -- http://mail.python.org/mailman/listinfo/python-list