Well the Stephen's explanation seems more logical... So that means that the limit count influences the success of the query ? I had a situation where my query (with 3 params) failed in my app (using low-level api) but succeeded in the App console (datastore viewer), using GQL... How can that be ?
On 14 April 2011 21:59, Alexandru Farcaş <alex.far...@expert-group.biz> wrote: > In this presentation, > http://dl.google.com/io/2009/pres/W_0415_Building_Scalable_Complex_App_Engines.pdf, > at slide 53, google say that "to many overlapping values = lots of zig > - zagging" > > > I test my app with queries with individual results (90000, <500) and I > didn't receive "Datastore need index exception...". > When I get this exception the query time is ~80ms. It looks like it > didn't try to zig zag anything... > > > On Apr 14, 8:34 pm, Robert Kluin <robert.kl...@gmail.com> wrote: >> Hi Stephen, >> Yeah, I didn't explain very that clearly at all; I think you did a >> significantly better job: "you have too few overlapping entities in >> the two sets." That is exactly what I was trying to say. ;) >> >> For those interested, the merge-join algorithm is explained here: >> http://sites.google.com/site/io/under-the-covers-of-the-google-app-en... >> >> Robert >> >> 2011/4/15 Stephen Johnson <onepagewo...@gmail.com>: >> >> >> >> >> >> >> >> > Hi Alexander, >> > I'm sorry to disagree with Robert, but the zig zag merge-join doesn't >> > return >> > the various rows for each index back to the application code to combine >> > them >> > in the application. It's a difficult algorithm to explain in an email but >> > basically the problem looks to be that with the 90000 entities and < 500 >> > entites case you have too few overlapping entites in the two sets. >> > Basically, think of it as you start off with 90000 possible candidates that >> > could be part of the result set. Now, the zig zag merge join goes and >> > checks >> > the other list (the less than 500 list), but it basically does this one at >> > a >> > time. Every time it doesn't find a match it throws that candidate away and >> > goes on to the next. It has to do this for each equality filter so if you >> > have 4 then it has to check for indexes for each candidate. You can see >> > that >> > this can take some time. Now if it finds enough matches fast enough to >> > satisfy the query limit that is requested to be returned then all is fine, >> > but if it has to keep throwing candidates away and trying new ones than it >> > times out and reports that it needs an index. So with your queries that >> > have >> > large individual results (90000, 50000 and 50000) you obviously have a >> > large >> > overlapping set and it finds enough matches very quickly, but when you >> > have >> > large sets combined with small sets (90000 and <500) then you have few >> > overlaps but lots of checking to do. >> > I hope this helps and if I misinterpreted Robert's post then I apologize to >> > him. >> > Stephen >> >> > On Thu, Apr 14, 2011 at 8:36 AM, Alexandru Farcaş >> > <alex.far...@expert-group.biz> wrote: >> >> >> Hi Robert, >> >> Thanks for your answer. >> >> If the number of results returned by each filter individually is >> >> smaller than, let's say 1000, the "final" number of results will be >> >> returned correctly? >> >> I want to know if there are some known limits for this type of query >> >> on a single list property (maybe for the size of the list, for the >> >> number of the results returned by each filter, for the number of >> >> filters, etc...). I didn't find the answer for this question. >> >> In all queries that returns datastore need index exception I have one >> >> filter that returns almost 90000 entities and one that returns < 500 >> >> results. >> >> But I have also many queries with 3 filters (90000 results, 50000 >> >> results, 50000 results) that returns the correct results. >> >> >> Hi Andrei, >> >> Yes, almost the same queries: the difference is that in the second >> >> table I have a single list property. This list contains the rest of >> >> fields from the first table. >> >> I tried to limit the number of the results from datastore viewer but >> >> with the same result :(. >> >> >> On Apr 14, 5:03 pm, Andrei Cosmin Fifiiţă <andrei.fifi...@gmail.com> >> >> wrote: >> >> > And also, i limit the number of results (for ex when implementing >> >> > pagination)... Does that affect the success of the query ? >> >> >> > 2011/4/14 Andrei Cosmin Fifiiţă <andrei.fifi...@gmail.com>: >> >> >> > > I don't want to create custom indices cause they will fail... but does >> >> > > the >> >> > > order of the equality filters matter? >> >> >> > > On Apr 14, 2011 3:38 AM, "Robert Kluin" <robert.kl...@gmail.com> >> >> > > wrote: >> >> >> -- >> >> You received this message because you are subscribed to the Google Groups >> >> "Google App Engine" group. >> >> To post to this group, send email to google-appengine@googlegroups.com. >> >> To unsubscribe from this group, send email to >> >> google-appengine+unsubscr...@googlegroups.com. >> >> For more options, visit this group at >> >>http://groups.google.com/group/google-appengine?hl=en. >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to google-appengine@googlegroups.com. >> > To unsubscribe from this group, send email to >> > google-appengine+unsubscr...@googlegroups.com. >> > For more options, visit this group at >> >http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
