Re: speeding up iterating over query set

2011-07-12 Thread Andre Terra
Have you looked at haystack? http://haystacksearch.org/ Cheers, André On Tue, Jul 12, 2011 at 10:49 AM, Michel30 wrote: > I have tried and I think I have it mostly working: it returns ALL > unique docid's. > > What is left is that my code is part of a search function. Originally > I got norma

Re: speeding up iterating over query set

2011-07-12 Thread bruno desthuilliers
On Jul 12, 3:49 pm, Michel30 wrote: > I have tried and I think I have it mostly working: it returns ALL > unique docid's. > > What is left is that my code is part of a search function. Originally > I got normalized keywords from a user and used those Q-objects to look > for keywords in a selected

Re: speeding up iterating over query set

2011-07-12 Thread Michel30
I have tried and I think I have it mostly working: it returns ALL unique docid's. What is left is that my code is part of a search function. Originally I got normalized keywords from a user and used those Q-objects to look for keywords in a selected set of columns. I still have to figure out how

Re: speeding up iterating over query set

2011-07-12 Thread bruno desthuilliers
On Jul 12, 12:26 pm, Michel30 wrote: > Hi guys, > > I've been trying your suggestions but I'm afraid I'm stretching the > limits of my Python/Django abilities ;-) > > Bruno got it right: what I want is a queryset of "model" with distinct > docid having the highest version number, sorted by revisio

Re: speeding up iterating over query set

2011-07-12 Thread Michel30
Hi guys, I've been trying your suggestions but I'm afraid I'm stretching the limits of my Python/Django abilities ;-) Bruno got it right: what I want is a queryset of "model" with distinct docid having the highest version number, sorted by revisiondate. If have the following result of my found_e

Re: speeding up iterating over query set

2011-07-12 Thread Michel30
Hi guys, I've been trying your suggestions but I'm afraid I'm stretching the limits of my Python/Django abilities ;-) Bruno got it right: what I want is a queryset of "model" with distinct docid having the highest version number, sorted by revisiondate. If have the following result of my found_e

Re: speeding up iterating over query set

2011-07-12 Thread Michel30
Hi guys, I've been trying your suggestions but I'm afraid I'm stretching the limits of my Python/Django abilities ;-) Bruno got it right: what I want is a queryset of "model" with distinct docid having the highest version number, sorted by revisiondate. If have the following result of my found_e

Re: speeding up iterating over query set

2011-07-11 Thread bruno desthuilliers
On 11 juil, 15:57, Michel30 wrote: > Hi all, > > I have a basic search function that uses Q objects. > After profiling it I found that the actual (mysql) database query > finishes in fractions of seconds but the iterating after this can take > up to 50 seconds per 10.000 results. > > I have been t

Re: speeding up iterating over query set

2011-07-11 Thread Andre Terra
You seem to be hauling the same data from one side to the other, and I can't see why. Try: MyModel.objects.values_list('id', flat=True) You can use that to build a dictionary if you add a second field, or using list comprehensions which I believe are faster than for loops, dict(MyModel.objects.

Re: speeding up iterating over query set

2011-07-11 Thread Michel30
I didn't try this approach yet, I'll give it a go and let you know my mileage. Thanks On Jul 11, 4:06 pm, "Cal Leeming [Simplicity Media Ltd]" wrote: > On Mon, Jul 11, 2011 at 3:04 PM, Cal Leeming [Simplicity Media Ltd] < > > > > cal.leem...@simplicitymedialtd.co.uk> wrote: > > > On Mon, Jul 11,

Re: speeding up iterating over query set

2011-07-11 Thread Cal Leeming [Simplicity Media Ltd]
On Mon, Jul 11, 2011 at 3:06 PM, Cal Leeming [Simplicity Media Ltd] < cal.leem...@simplicitymedialtd.co.uk> wrote: > > > On Mon, Jul 11, 2011 at 3:04 PM, Cal Leeming [Simplicity Media Ltd] < > cal.leem...@simplicitymedialtd.co.uk> wrote: > >> >> >> On Mon, Jul 11, 2011 at 2:57 PM, Michel30 wrote:

Re: speeding up iterating over query set

2011-07-11 Thread Cal Leeming [Simplicity Media Ltd]
On Mon, Jul 11, 2011 at 3:04 PM, Cal Leeming [Simplicity Media Ltd] < cal.leem...@simplicitymedialtd.co.uk> wrote: > > > On Mon, Jul 11, 2011 at 2:57 PM, Michel30 wrote: > >> Hi all, >> >> I have a basic search function that uses Q objects. >> After profiling it I found that the actual (mysql) da

Re: speeding up iterating over query set

2011-07-11 Thread Cal Leeming [Simplicity Media Ltd]
On Mon, Jul 11, 2011 at 2:57 PM, Michel30 wrote: > Hi all, > > I have a basic search function that uses Q objects. > After profiling it I found that the actual (mysql) database query > finishes in fractions of seconds but the iterating after this can take > up to 50 seconds per 10.000 results. >

speeding up iterating over query set

2011-07-11 Thread Michel30
Hi all, I have a basic search function that uses Q objects. After profiling it I found that the actual (mysql) database query finishes in fractions of seconds but the iterating after this can take up to 50 seconds per 10.000 results. I have been trying to speed it up but I have had not much resul