Thx Robert. Your reference is always good materials for me to study. Would study it and see if can come up with any solution to solve my problem following Alfred's optimization! And if still cannot, would bring up the qeustion again ;)
Best Regards, Eric On 12 October 2010 00:48, Robert Kluin <robert.kl...@gmail.com> wrote: > Hey Eric, > Perhaps you can come up with a method to identify the 'best' search > parameters when deciding how to filter your items. > > You should also check out Alfred Fuller's IO talk, next gen queries > on app engine. > > http://code.google.com/events/io/2010/sessions/next-gen-queries-appengine.html > > They are now starting to support some of those items. From the > 1.3.8 (pre)release notes: > - Removed limits on zigzag merge-join queries. Therefore the > error "The built-in > indices are not efficient enough for this query and your data. > Please add a > composite index for this query." will no longer be thrown in many > cases, > enabling more types of exploratory queries without indexes. > > > Maybe some of Alfred's optimizations can help you. > > > Robert > > > > > > On Mon, Oct 11, 2010 at 05:35, Eric Ka Ka Ng <ngk...@gmail.com> wrote: > > From the slide "Building scalable, complex apps on App Engine" by Brett > > Slatkin > > > http://dl.google.com/io/2009/pres/W_0415_Building_Scalable_Complex_App_Engines.pdf > > > > we learn techniques for building scalable services by using list and self > > merge-join. self merge-join is very useful to me as it does not require > > building additional custom indexes (the hard limit of 200 custom indexes > is > > always a headache for me with my application, and always have to use many > > different techniques to reduce the custom indexes usage) > > > > however, recently i hit a problem > > NeedIndexError: The built-in indices are not efficient enough for this > query > > and your data. Please add a composite index for this query. > > > > My model is very simple, just like this > > class Searchable(db.Model): > > phrases = db.StringListProperty() > > > > the query is > > query = searchable.all().filter('phrases =', word1).filter('phrases =', > > word2).fetch(1000) > > > > It works fine before. But now fail with some cases. > > I re-read slides, and find that about merge-join performance > > "Scales with number of filters and size of result set Best for queries > with > > fewer results (less than 100)" > > > > i guess very likely, now when my application scales with more data (let > say > > Searchable now stores 1 million records), when doing the query and if the > > result set is large (how large is large? > 100?), i would then hit this > > 'built-in indices are not efficient enough' problem. the query works fine > if > > the returned result set is much smaller > > > > this is my guessing. so my questions are > > 1. is my guess correct / making sense? what other cases would > > cause 'built-in indices are not efficient enough' problem? are there any > > others share similar experiences? > > 2. is there any hard number, or formula, we can use to estimate by how > large > > the result set is, would hit the 'built-in indices are not efficient > enough' > > problem? other than size of result set, would there be other factor > matter? > > (number of self merge-join, number of properties, number of indexed > > properties etc.) > > 3. the most important question, any solution / workaround for that ? > (other > > than building extra custom indexes which is not quite feasible for me as > i'm > > already having 200 indexes) > > > > > > appreciate for any comments / suggestions. > > - eric ng > > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > > For more options, visit this group at > > http://groups.google.com/group/google-appengine?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
