In this presentation, http://dl.google.com/io/2009/pres/W_0415_Building_Scalable_Complex_App_Engines.pdf, at slide 53, google say that "to many overlapping values = lots of zig - zagging"
I test my app with queries with individual results (90000, <500) and I didn't receive "Datastore need index exception...". When I get this exception the query time is ~80ms. It looks like it didn't try to zig zag anything... On Apr 14, 8:34 pm, Robert Kluin <robert.kl...@gmail.com> wrote: > Hi Stephen, > Yeah, I didn't explain very that clearly at all; I think you did a > significantly better job: "you have too few overlapping entities in > the two sets." That is exactly what I was trying to say. ;) > > For those interested, the merge-join algorithm is explained here: > http://sites.google.com/site/io/under-the-covers-of-the-google-app-en... > > Robert > > 2011/4/15 Stephen Johnson <onepagewo...@gmail.com>: > > > > > > > > > Hi Alexander, > > I'm sorry to disagree with Robert, but the zig zag merge-join doesn't return > > the various rows for each index back to the application code to combine them > > in the application. It's a difficult algorithm to explain in an email but > > basically the problem looks to be that with the 90000 entities and < 500 > > entites case you have too few overlapping entites in the two sets. > > Basically, think of it as you start off with 90000 possible candidates that > > could be part of the result set. Now, the zig zag merge join goes and checks > > the other list (the less than 500 list), but it basically does this one at a > > time. Every time it doesn't find a match it throws that candidate away and > > goes on to the next. It has to do this for each equality filter so if you > > have 4 then it has to check for indexes for each candidate. You can see that > > this can take some time. Now if it finds enough matches fast enough to > > satisfy the query limit that is requested to be returned then all is fine, > > but if it has to keep throwing candidates away and trying new ones than it > > times out and reports that it needs an index. So with your queries that have > > large individual results (90000, 50000 and 50000) you obviously have a large > > overlapping set and it finds enough matches very quickly, but when you have > > large sets combined with small sets (90000 and <500) then you have few > > overlaps but lots of checking to do. > > I hope this helps and if I misinterpreted Robert's post then I apologize to > > him. > > Stephen > > > On Thu, Apr 14, 2011 at 8:36 AM, Alexandru Farcaş > > <alex.far...@expert-group.biz> wrote: > > >> Hi Robert, > >> Thanks for your answer. > >> If the number of results returned by each filter individually is > >> smaller than, let's say 1000, the "final" number of results will be > >> returned correctly? > >> I want to know if there are some known limits for this type of query > >> on a single list property (maybe for the size of the list, for the > >> number of the results returned by each filter, for the number of > >> filters, etc...). I didn't find the answer for this question. > >> In all queries that returns datastore need index exception I have one > >> filter that returns almost 90000 entities and one that returns < 500 > >> results. > >> But I have also many queries with 3 filters (90000 results, 50000 > >> results, 50000 results) that returns the correct results. > > >> Hi Andrei, > >> Yes, almost the same queries: the difference is that in the second > >> table I have a single list property. This list contains the rest of > >> fields from the first table. > >> I tried to limit the number of the results from datastore viewer but > >> with the same result :(. > > >> On Apr 14, 5:03 pm, Andrei Cosmin Fifiiţă <andrei.fifi...@gmail.com> > >> wrote: > >> > And also, i limit the number of results (for ex when implementing > >> > pagination)... Does that affect the success of the query ? > > >> > 2011/4/14 Andrei Cosmin Fifiiţă <andrei.fifi...@gmail.com>: > > >> > > I don't want to create custom indices cause they will fail... but does > >> > > the > >> > > order of the equality filters matter? > > >> > > On Apr 14, 2011 3:38 AM, "Robert Kluin" <robert.kl...@gmail.com> > >> > > wrote: > > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Google App Engine" group. > >> To post to this group, send email to google-appengine@googlegroups.com. > >> To unsubscribe from this group, send email to > >> google-appengine+unsubscr...@googlegroups.com. > >> For more options, visit this group at > >>http://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
