>From the slide "Building scalable, complex apps on App Engine" by Brett
Slatkin
http://dl.google.com/io/2009/pres/W_0415_Building_Scalable_Complex_App_Engines.pdf
we learn techniques for building scalable services by using list and self
merge-join. self merge-join is very useful to me as it does not require
building additional custom indexes (the hard limit of 200 custom indexes is
always a headache for me with my application, and always have to use many
different techniques to reduce the custom indexes usage)
however, recently i hit a problem
*NeedIndexError: The built-in indices are not efficient enough for this
query and your data. Please add a composite index for this query.*
My model is very simple, just like this
class Searchable(db.Model):
phrases = db.StringListProperty()
the query is
query = searchable.all().filter('phrases =', word1).filter('phrases =',
word2).fetch(1000)
It works fine before. But now fail with some cases.
I re-read slides, and find that about merge-join performance
*"**Scales with number of filters and size of result set Best for queries
with fewer results (less than 100)"*
i guess very likely, now when my application scales with more data (let say
Searchable now stores 1 million records), when doing the query and if the
result set is large (how large is large? > 100?), i would then hit
this 'built-in
indices are not efficient enough' problem. the query works fine if the
returned result set is much smaller
this is my guessing. so my questions are
1. is my guess correct / making sense? what other cases would cause 'built-in
indices are not efficient enough' problem? are there any others share
similar experiences?
2. is there any hard number, or formula, we can use to estimate by how large
the result set is, would hit the 'built-in indices are not efficient enough'
problem? other than size of result set, would there be other factor matter?
(number of self merge-join, number of properties, number of indexed
properties etc.)
3. the most important question, any solution / workaround for that ? (other
than building extra custom indexes which is not quite feasible for me as i'm
already having 200 indexes)
appreciate for any comments / suggestions.
- eric ng
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
