[google-appengine] Re: Query with >1000 matches

2009-06-23 Thread Nick Johnson (Google)
On Tue, Jun 23, 2009 at 3:27 PM, herbie <4whi...@o2.co.uk> wrote: > > Oh, really? That limits my app somewhat. I asume if I have no > inequality filtres and order by date I will get the latest entities? > > I could then filter these in memory for x > threshold Correct. -Nick Johnson > > > >

[google-appengine] Re: Query with >1000 matches

2009-06-23 Thread herbie
Oh, really? That limits my app somewhat. I asume if I have no inequality filtres and order by date I will get the latest entities? I could then filter these in memory for x > threshold On Jun 23, 1:50 pm, "Nick Johnson (Google)" wrote: > Hi herbie, > > If your query includes an inequality (s

[google-appengine] Re: Query with >1000 matches

2009-06-23 Thread Nick Johnson (Google)
Hi herbie, If your query includes an inequality (such as x>50), then your first sort order has to be on the same property as that inequality, which means you can't (directly) fetch the most recent 200 results with x>50. You either need to change your query to use only equality filters, or you need

[google-appengine] Re: Query with >1000 matches

2009-06-23 Thread herbie
Thanks for your help Nick. No my threshold value 'x' isn't constant. I still havn't got my head round this yet! Can you tell me how to get the latest entities (assuming I don't want all of them) out of the datastore and filter on another property? For example: Get the latest 200 entities

[google-appengine] Re: Query with >1000 matches

2009-06-23 Thread Nick Johnson (Google)
On Tue, Jun 23, 2009 at 12:42 PM, herbie <4whi...@o2.co.uk> wrote: > > > So will this : > query = Foo.all().filter("property_x >" 50).order("property_x") .order > ("-timestamp") > results = query.fetch(200) > > ..get the latest entities where property_x > 50 ? Or will it get the > 200 properties

[google-appengine] Re: Query with >1000 matches

2009-06-23 Thread herbie
So will this : query = Foo.all().filter("property_x >" 50).order("property_x") .order ("-timestamp") results = query.fetch(200) ..get the latest entities where property_x > 50 ? Or will it get the 200 properties with the largest 'property_x' which are then ordered by 'timestamp' ? A subtle b

[google-appengine] Re: Query with >1000 matches

2009-06-22 Thread Tony
Yes, that is what it means. I forgot about that restriction. I see what you mean about changing 'x' values. Perhaps consider keeping two counts - a running sum and a running count (of the # of x properties). If a user modifies an 'x' value, you can adjust the sum up or down accordingly. On Ju

[google-appengine] Re: Query with >1000 matches

2009-06-22 Thread herbie
I tried your query below but I get "BadArgumentError: First ordering property must be the same as inequality filter property, if specified for this query;" Does this mean I have to order on 'x' first, then order on 'date'? Will this still return the latest 200 of all entities with x > 50 if I cal

[google-appengine] Re: Query with >1000 matches

2009-06-22 Thread Tony
I should clarify, I'm not saying move the 1000-entity fetch to the write process, but instead keep a running total of sum and count that you can increment and use to calculate the average, rather than having to fetch entities. This doesn't solve the use case of "average of last x entities", thoug

[google-appengine] Re: Query with >1000 matches

2009-06-22 Thread Tony
You could accomplish this task like so: xlist = [] query = Foo.all().filter("property_x >" 50).order("-timestamp") for q in query: xlist.append(q.property_x) avg = sum(xlist) / len(xlist) What Nick is saying, I think, is that fetching 1000 entities is going to be very resource-intensive, so a

[google-appengine] Re: Query with >1000 matches

2009-06-22 Thread herbie
Ok. Say I have many (>1000) Model entities with two properties 'x' and 'date'.What is the most efficient query to fetch say the latest 200 entities where x > 50. I don't care what their 'date's are as long as I get the latest and x > 50 Thanks again for your help. On Jun 22, 4:11 pm, "N

[google-appengine] Re: Query with >1000 matches

2009-06-22 Thread Nick Johnson (Google)
Consider precalculating this data and storing it against another entity. This will save a lot of work on requests. -Nick Johnson On Mon, Jun 22, 2009 at 3:55 PM, herbie <4whi...@o2.co.uk> wrote: > > No the users won't need to read 1000 entities, but I want to calculate > the average of a proper

[google-appengine] Re: Query with >1000 matches

2009-06-22 Thread herbie
No the users won't need to read 1000 entities, but I want to calculate the average of a property from the latest 1000 entities. On Jun 22, 3:30 pm, "Nick Johnson (Google)" wrote: > Correct. Are you sure you need 1000 entities, though? Your users probably > won't read through all 1000. > > -Nic

[google-appengine] Re: Query with >1000 matches

2009-06-22 Thread Nick Johnson (Google)
Correct. Are you sure you need 1000 entities, though? Your users probably won't read through all 1000. -Nick Johnson On Mon, Jun 22, 2009 at 3:23 PM, herbie <4whi...@o2.co.uk> wrote: > > > So to be sure to get the latest 1000 entities I should add a datetime > property to my entitie model and fi

[google-appengine] Re: Query with >1000 matches

2009-06-22 Thread herbie
So to be sure to get the latest 1000 entities I should add a datetime property to my entitie model and filter and sort on that? On Jun 22, 1:42 pm, herbie <4whi...@o2.co.uk> wrote: > I know that if there are more than 1000 entities that match a query, > then only 1000 will  be return by fetch(

[google-appengine] Re: Query with >1000 matches

2009-06-22 Thread Nick Johnson (Google)
Hi herbie, The first 1000 results of a query are the ones returned. If you do not specify a sort order, entities are returned sorted by their keys. -Nick Johnson On Mon, Jun 22, 2009 at 1:42 PM, herbie <4whi...@o2.co.uk> wrote: > > I know that if there are more than 1000 entities that match a q