Re: Optimizing list processing

2013-12-12 Thread Stefan Behnel
Terry Reedy, 12.12.2013 03:26: from itertools import count table = sorted(t for t in zip(iterable, count)) This is a useless use of a generator expression. sorted(zip(...)) is enough (and likely to be substantially faster also). Stefan -- https://mail.python.org/mailman/listinfo/python-list

Re: Optimizing list processing

2013-12-12 Thread Steven D'Aprano
On Wed, 11 Dec 2013 21:26:51 -0500, Terry Reedy wrote: On 12/11/2013 6:54 PM, Steven D'Aprano wrote: I have some code which produces a list from an iterable using at least one temporary list, using a Decorate-Sort-Undecorate idiom. It is a non-standard version thereof, as DSU usually means

Re: Optimizing list processing

2013-12-12 Thread Chris Angelico
On Thu, Dec 12, 2013 at 11:08 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: P.S. The algorithm I'm working on is a way of generating index and rank tables. Not that it really matters -- what matters is determining whether or not to shift from make a copy of the list to modify

Re: Optimizing list processing

2013-12-12 Thread MRAB
On 12/12/2013 12:25, Chris Angelico wrote: On Thu, Dec 12, 2013 at 11:08 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: P.S. The algorithm I'm working on is a way of generating index and rank tables. Not that it really matters -- what matters is determining whether or not to

Re: Optimizing list processing

2013-12-12 Thread Chris Angelico
On Fri, Dec 13, 2013 at 12:32 AM, MRAB pyt...@mrabarnett.plus.com wrote: table[:] = [i for x,i in table] # Does slice assignment get optimized? [snip] If you're trying that, you could also try: table[:] = (i for x,i in table) Right, a tweak which could be applied also to the split

Re: Optimizing list processing

2013-12-12 Thread Peter Otten
Steven D'Aprano wrote: I have some code which produces a list from an iterable using at least one temporary list, using a Decorate-Sort-Undecorate idiom. The algorithm looks something like this (simplified): table = sorted([(x, i) for i,x in enumerate(iterable)]) table = [i for x,i in

Re: Optimizing list processing

2013-12-12 Thread Terry Reedy
On 12/12/2013 6:09 AM, Stefan Behnel wrote: Terry Reedy, 12.12.2013 03:26: from itertools import count table = sorted(t for t in zip(iterable, count)) This is a useless use of a generator expression. sorted(zip(...)) is enough (and likely to be substantially faster also). Yes, definitely,

Re: Optimizing list processing

2013-12-12 Thread Terry Reedy
On 12/12/2013 7:08 AM, Steven D'Aprano wrote: Please don't focus on the algorithm I gave. Focus on the fact that I could write it like this: if some condition to do with the computer's available memory: make modifications in place else: make a copy of the data containing the

Re: Optimizing list processing

2013-12-12 Thread Steven D'Aprano
as a proxy for whether or not I should switch algorithms. That's why this post is called Optimizing list processing rather than Finding out how much memory is available. If there's another way to solve the same problem, I'm happy to hear it. [...] I'm not terribly fussed about micro-optimizations

Re: Optimizing list processing

2013-12-12 Thread Chris Angelico
On Fri, Dec 13, 2013 at 11:14 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Back in the 1980s, when I was a Mac user and occasional programmer, there were memory manager routines which (if I recall correctly) would tell you whether or not an allocation would succeed or not.

Re: Optimizing list processing

2013-12-12 Thread Steven D'Aprano
On Thu, 12 Dec 2013 16:08:33 +0100, Peter Otten wrote: Steven D'Aprano wrote: [...] So, ideally I'd like to write my code like this: table = [(x, i) for i,x in enumerate(iterable)] table.sort() if len(table) ?: table = [i for x,i in table] else: for x, i in table:

Re: Optimizing list processing

2013-12-12 Thread rusi
On Friday, December 13, 2013 8:31:37 AM UTC+5:30, Steven D'Aprano wrote: I don't know of any reasonable way to tell at runtime which of the two algorithms I ought to take. Hard-coding an arbitrary value (if len(table) 500) is not the worst idea I've ever had, but I'm hoping for

Optimizing list processing

2013-12-11 Thread Steven D'Aprano
I have some code which produces a list from an iterable using at least one temporary list, using a Decorate-Sort-Undecorate idiom. The algorithm looks something like this (simplified): table = sorted([(x, i) for i,x in enumerate(iterable)]) table = [i for x,i in table] The problem here is that

Re: Optimizing list processing

2013-12-11 Thread MRAB
On 11/12/2013 23:54, Steven D'Aprano wrote: I have some code which produces a list from an iterable using at least one temporary list, using a Decorate-Sort-Undecorate idiom. The algorithm looks something like this (simplified): table = sorted([(x, i) for i,x in enumerate(iterable)]) table = [i

Re: Optimizing list processing

2013-12-11 Thread duncan smith
On 11/12/13 23:54, Steven D'Aprano wrote: I have some code which produces a list from an iterable using at least one temporary list, using a Decorate-Sort-Undecorate idiom. The algorithm looks something like this (simplified): table = sorted([(x, i) for i,x in enumerate(iterable)]) table = [i

Re: Optimizing list processing

2013-12-11 Thread Ben Finney
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info writes: For giant iterables (ten million items), this version is a big improvement, about three times faster than the list comp version. […] Except that for more reasonably sized iterables, it's a pessimization. With one million items,

Re: Optimizing list processing

2013-12-11 Thread Steven D'Aprano
On Thu, 12 Dec 2013 00:59:42 +, MRAB wrote: table = [(x, i) for i,x in enumerate(iterable)] table.sort() This looks wrong to me: for x, i in table: table[i] = x Yes, you're right, I over-simplified the example, and in doing so introduced a bug. What I actually use is: for i,

Re: Optimizing list processing

2013-12-11 Thread MRAB
On 12/12/2013 01:43, Steven D'Aprano wrote: On Thu, 12 Dec 2013 00:59:42 +, MRAB wrote: table = [(x, i) for i,x in enumerate(iterable)] table.sort() This looks wrong to me: for x, i in table: table[i] = x Yes, you're right, I over-simplified the example, and in doing so

Re: Optimizing list processing

2013-12-11 Thread Terry Reedy
On 12/11/2013 6:54 PM, Steven D'Aprano wrote: I have some code which produces a list from an iterable using at least one temporary list, using a Decorate-Sort-Undecorate idiom. It is a non-standard version thereof, as DSU usually means to decorate with a key that gets discarded. A couple of