Re: Most efficient way to "pre-grow" a list?

2009-11-11 Thread Steven D'Aprano
On Wed, 11 Nov 2009 08:13:54 -0800, Francis Carr wrote: > Hmmm. I am trying some algorithms, and timeit reports that a > list.extend variant is fastest... WTH?! Really this seems like it must > be a "bug" in implementing the "[None]*x" idiom. > The straightforward "pythonic" solution is better

Re: Most efficient way to "pre-grow" a list?

2009-11-11 Thread Francis Carr
Hmmm. I am trying some algorithms, and timeit reports that a list.extend variant is fastest... WTH?! Really this seems like it must be a "bug" in implementing the "[None]*x" idiom. As expected, appending one-at-a-time is slowest (by an order of magnitude): % python -m timeit -s "N=100" \

Re: Most efficient way to "pre-grow" a list?

2009-11-10 Thread gil_johnson
On Nov 9, 10:56 pm, "Gabriel Genellina" wrote: > > [much cutting] > > def method3a(): >    newArray = array.array('I', [INITIAL_VALUE]) * SIZE >    assert len(newArray)==SIZE >    assert newArray[SIZE-1]==INITIAL_VALUE > > [more cutting] > > So arrays are faster than lists, and in both cases one_it

Re: Most efficient way to "pre-grow" a list?

2009-11-09 Thread Gabriel Genellina
En Sun, 08 Nov 2009 10:08:35 -0300, gil_johnson escribió: On Nov 6, 8:46 pm, gil_johnson wrote: The problem I was solving was this: I wanted an array of 32-bit integers to be used as a bit array, and I wanted it initialized with all bits set, that is, each member of the array had to be set to

Re: Most efficient way to "pre-grow" a list?

2009-11-08 Thread Emile van Sebille
On 11/7/2009 5:18 PM Carl Banks said... I think the top one is O(N log N), and I'm suspicious that it's even possible to grow a list in less than O(N log N) time without knowing the final size in advance. Citation? Quoting Tim -- Python uses mildly exponential over-allocation, sufficient so

Re: Most efficient way to "pre-grow" a list?

2009-11-08 Thread gb345
In Luis Alberto Zarrabeitia Gomez writes: >¿Have you ever tried to read list/matrix that you know it is not sparse, but >you jdon't know the size, and it may not be in order? A "grow-able" array would just >be the right thing to use - currently I have to settle with either hacking >together my

Re: Most efficient way to "pre-grow" a list?

2009-11-08 Thread gil_johnson
On Nov 6, 8:46 pm, gil_johnson wrote: > I don't have the code with me, but for huge arrays, I have used > something like: > > >>> arr[0] = initializer > >>> for i in range N: > >>>      arr.extend(arr) > > This doubles the array every time through the loop, and you can add > the powers of 2 to ge

Re: Most efficient way to "pre-grow" a list?

2009-11-08 Thread gil_johnson
On Nov 6, 8:46 pm, gil_johnson wrote: > >>> arr[0] = initializer > >>> for i in range N: > >>>      arr.extend(arr) > > This doubles the array every time through the loop, and you can add > the powers of 2 to get the desired result. > Gil To all who asked about my previous post, I'm sorry I pos

Re: Most efficient way to "pre-grow" a list?

2009-11-07 Thread Paul Rudin
kj writes: > The best I can come up with is this: > > arr = [None] * 100 > > Is this the most efficient way to achieve this result? > Depends on what you take as given. You can do it with ctypes more efficiently, but you can also shoot yourself in the foot. Another possibility is to use a n

Re: Most efficient way to "pre-grow" a list?

2009-11-07 Thread exarkun
On 01:18 am, pavlovevide...@gmail.com wrote: On Nov 7, 5:05�pm, sturlamolden wrote: On 7 Nov, 03:46, gil_johnson wrote:> > I don't have the code with me, but for huge arrays, I have used > something like: > >>> arr[0] = initializer > >>> for i in range N: > >>> � � �arr.extend(arr) > This d

Re: Most efficient way to "pre-grow" a list?

2009-11-07 Thread Carl Banks
On Nov 7, 5:05 pm, sturlamolden wrote: > On 7 Nov, 03:46, gil_johnson wrote:> > > > I don't have the code with me, but for huge arrays, I have used > > something like: > > > >>> arr[0] = initializer > > >>> for i in range N: > > >>>      arr.extend(arr) > > > This doubles the array every time thr

Re: Most efficient way to "pre-grow" a list?

2009-11-07 Thread sturlamolden
On 6 Nov, 22:03, kj wrote: > As I said, this is considered an optimization, at least in Perl, > because it lets the interpreter allocate all the required memory > in one fell swoop, instead of having to reallocate it repeatedly > as the array grows. Python does not need to reallocate repeatedly

Re: Most efficient way to "pre-grow" a list?

2009-11-07 Thread sturlamolden
On 7 Nov, 03:46, gil_johnson wrote:> > I don't have the code with me, but for huge arrays, I have used > something like: > > >>> arr[0] = initializer > >>> for i in range N: > >>>      arr.extend(arr) > > This doubles the array every time through the loop, and you can add > the powers of 2 to get

Re: Most efficient way to "pre-grow" a list?

2009-11-07 Thread sturlamolden
On 6 Nov, 13:12, kj wrote: > > The best I can come up with is this: > > arr = [None] * 100 > > Is this the most efficient way to achieve this result? Yes, but why would you want to? Appending to a Python list has amortized O(1) complexity. I am not sure about Perl, but in MATLAB arrays are p

Re: Most efficient way to "pre-grow" a list?

2009-11-07 Thread kj
In Luis Alberto Zarrabeitia Gomez writes: >Quoting Andre Engels : >> On Sat, Nov 7, 2009 at 8:25 PM, Luis Alberto Zarrabeitia Gomez >> wrote: >> > >> > Ok, he has a dict. >> > >> > Now what? He needs a non-sparse array. >> >> Let d be your dict. >> >> Call the zeroeth place in your array d

Re: Most efficient way to "pre-grow" a list?

2009-11-07 Thread Ivan Illarionov
On Nov 6, 3:12 pm, kj wrote: > The best I can come up with is this: > > arr = [None] * 100 > > Is this the most efficient way to achieve this result? It is the most efficient SAFE way to achieve this result. In fact, there IS the more efficient way, but it's dangerous, unsafe, unpythonic and

Re: Most efficient way to "pre-grow" a list?

2009-11-07 Thread Terry Reedy
Steven D'Aprano wrote: On Fri, 06 Nov 2009 18:46:33 -0800, gil_johnson wrote: I don't have the code with me, but for huge arrays, I have used something like: arr[0] = initializer for i in range N: arr.extend(arr) This doubles the array every time through the loop, and you can add the po

Re: Most efficient way to "pre-grow" a list?

2009-11-07 Thread Luis Alberto Zarrabeitia Gomez
Quoting Andre Engels : > On Sat, Nov 7, 2009 at 8:25 PM, Luis Alberto Zarrabeitia Gomez > wrote: > > > > Ok, he has a dict. > > > > Now what? He needs a non-sparse array. > > Let d be your dict. > > Call the zeroeth place in your array d[0], the first d[1], the 1th > d[10]. Following

Re: Most efficient way to "pre-grow" a list?

2009-11-07 Thread Andre Engels
On Sat, Nov 7, 2009 at 8:25 PM, Luis Alberto Zarrabeitia Gomez wrote: > > Quoting Bruno Desthuilliers : > >> > Another situation where one may want to do this is if one needs to >> > initialize a non-sparse array in a non-sequential order, >> >> Then use a dict. > > Ok, he has a dict. > > Now what

Re: Most efficient way to "pre-grow" a list?

2009-11-07 Thread Luis Alberto Zarrabeitia Gomez
Quoting Bruno Desthuilliers : > > Another situation where one may want to do this is if one needs to > > initialize a non-sparse array in a non-sequential order, > > Then use a dict. Ok, he has a dict. Now what? He needs a non-sparse array. -- Luis Zarrabeitia Facultad de Matemática y Comput

Re: Most efficient way to "pre-grow" a list?

2009-11-07 Thread Bruno Desthuilliers
kj a écrit : > > As I said, this is considered an optimization, at least in Perl, > because it lets the interpreter allocate all the required memory > in one fell swoop, instead of having to reallocate it repeatedly > as the array grows. IIRC, CPython has it's own way to optimize list growth. >

Re: Most efficient way to "pre-grow" a list?

2009-11-07 Thread Steven D'Aprano
On Fri, 06 Nov 2009 18:46:33 -0800, gil_johnson wrote: > I don't have the code with me, but for huge arrays, I have used > something like: > arr[0] = initializer for i in range N: arr.extend(arr) > > This doubles the array every time through the loop, and you can add the > po