Re: fastest way to find the intersection of n lists of sets

2007-04-30 Thread Prateek
On Apr 30, 12:37 pm, Raymond Hettinger <[EMAIL PROTECTED]> wrote: > [Prateek] > > > The reason why I'm casting to a list first is because I found that > > creating a long list which I convert to a set in a single operation is > > faster (although probably less memory efficient - which I can deal >

Re: fastest way to find the intersection of n lists of sets

2007-04-30 Thread Raymond Hettinger
[Prateek] > The reason why I'm casting to a list first is because I found that > creating a long list which I convert to a set in a single operation is > faster (although probably less memory efficient - which I can deal > with) than doing all the unions. That would be a surprising result because

Re: fastest way to find the intersection of n lists of sets

2007-04-30 Thread Raymond Hettinger
[Prateek] > I have 3 variable length lists of sets. I need to find the common > elements in each list (across sets) really really quickly. . . . > l1 = reduce(operator.add, list(x) for x in l1) > l2 = reduce(operator.add, list(x) for x in l2) > l3 = reduce(operator.add, list(x) for x in l3) > s = f

Re: fastest way to find the intersection of n lists of sets

2007-04-29 Thread Alex Martelli
Prateek <[EMAIL PROTECTED]> wrote: > > For the above example, it's worth sorting lists_of_sets by the > > length of the sets, and doing the short ones first. > > Thanks. I thought so - I'm doing just that using a simple Decorate- > Sort-Undecorate idiom. Use, instead, the DSU that is now bu

Re: fastest way to find the intersection of n lists of sets

2007-04-29 Thread Paul Rubin
Prateek <[EMAIL PROTECTED]> writes: > The big set does stay around for a while - I've implemented an LRU > based caching algorithm on the code that does the I/O. Since the db is > transactioned, I keep one copy in the current transaction cache (which > is a simple dictionary) and one in the main re

Re: fastest way to find the intersection of n lists of sets

2007-04-29 Thread Prateek
On Apr 30, 5:08 am, John Nagle <[EMAIL PROTECTED]> wrote: > Prateek wrote: > >> For the above example, it's worth sorting lists_of_sets by the > >>length of the sets, and doing the short ones first. > > > Thanks. I thought so - I'm doing just that using a simple Decorate- > > Sort-Undecorate id

Re: fastest way to find the intersection of n lists of sets

2007-04-29 Thread John Nagle
Prateek wrote: >> For the above example, it's worth sorting lists_of_sets by the >>length of the sets, and doing the short ones first. > > > Thanks. I thought so - I'm doing just that using a simple Decorate- > Sort-Undecorate idiom. > > >> How big are the sets? If they're small, but y

Re: fastest way to find the intersection of n lists of sets

2007-04-29 Thread Prateek
On Apr 30, 3:48 am, James Stroud <[EMAIL PROTECTED]> wrote: > Prateek wrote: > > I have 3 variable length lists of sets. I need to find the common > > elements in each list (across sets) really really quickly. > > > Here is some sample code: > > > # Doesn't make sense to union the sets - we're goin

Re: fastest way to find the intersection of n lists of sets

2007-04-29 Thread Prateek
> For the above example, it's worth sorting lists_of_sets by the > length of the sets, and doing the short ones first. Thanks. I thought so - I'm doing just that using a simple Decorate- Sort-Undecorate idiom. > How big are the sets? If they're small, but you have a lot of > them, you

Re: fastest way to find the intersection of n lists of sets

2007-04-29 Thread John Nagle
James Stroud wrote: > Prateek wrote: > >> I have 3 variable length lists of sets. I need to find the common >> elements in each list (across sets) really really quickly. >> >> Here is some sample code: >> >> # Doesn't make sense to union the sets - we're going to do >> intersections later anyway >

Re: fastest way to find the intersection of n lists of sets

2007-04-29 Thread James Stroud
Prateek wrote: > I have 3 variable length lists of sets. I need to find the common > elements in each list (across sets) really really quickly. > > Here is some sample code: > > # Doesn't make sense to union the sets - we're going to do > intersections later anyway > l1 = reduce(operator.add, lis

fastest way to find the intersection of n lists of sets

2007-04-29 Thread Prateek
I have 3 variable length lists of sets. I need to find the common elements in each list (across sets) really really quickly. Here is some sample code: # Doesn't make sense to union the sets - we're going to do intersections later anyway l1 = reduce(operator.add, list(x) for x in l1) l2 = reduce(o