[google-appengine] Re: Trouble with NeedIndexError when calling count()
I thought I would post our findings and solution. Our issue seems slightly different. We had entities similar to the following: class Account(db.Model): ... class AccountEntry(db.Model): account = db.ReferenceProperty(Account) tags = db.StringListProperty() utime = db.DateTimeProperty() ... We had the following index in index.yaml, which was generated by dev_appserver (for another gql query) and deployed to appspot successfully: - kind: AccountEntry properties: - name: account - name: tags - name: utime direction: desc However, once enough 'AccountEntry' entities where created, the following gql query: AccountEntry.gql(WHERE account = :1 AND tags = :2, account, tag).count(100) raised the NeedIndexError: The built-in indices are not efficient enough for this query and your data. Please add a composite index for this query. error. The problem was solved by manually adding the following index to the index.yaml: - kind: AccountEntry properties: - name: account - name: tags since dev_appserver did not automatically generate this index for us when the above query is executed. Cheers Tony On Jan 9, 3:32 pm, Ryan W rwilli...@gmail.com wrote: Thanks for the explanation. I was trying to avoid any __search_text_index indexes as you suggest, to get around the 1 per each number of keywords, so I didn't use any equality or sorts. It usually works fine, and is again today as far as I can tell. It must've been been something with the particular search query being done. I've tried to set it up to capture/log errors better. Hopefully the full text solution is on its way soon. On Jan 9, 10:30 am, ryan ryanb+appeng...@google.com wrote: hi guys! merge join queries like these, ie queries that have equals filters on multiple properties but no inequality filters or sort orders, *usually* don't require an index. that's described inhttp://code.google.com/appengine/docs/datastore/queriesandindexes.htm... . however, in cases with unusually shaped data, these queries can sometimes require an index. these cases are exceedingly rare, but they're somewhat more common with apphosting.ext.search. also, you need an index for each number of keywords that you expect in a query. for example, if you support three-word search queries, you need this index: - kind: Post properties: - name: __searchable_text_index - name: __searchable_text_index - name: __searchable_text_index - name: approved more in the ext.search docstring,http://code.google.com/p/googleappengine/source/browse/trunk/google/a... . also, unfortunately, these kinds of indices are likely to explode, as described inhttp://code.google.com/appengine/docs/datastore/queriesandindexes.htm... . the upshot of all of this, as we've mentioned before, is that appengine.ext.search is a hack. it's not scalable, mature, or anywhere near industrial strength, which is why it's not officially supported or documented. we're well aware that it's not good enough, and we dearly hope to offer a real full text search solution in the future. i don't have any details or timeline, though. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Trouble with NeedIndexError when calling count()
We saw this error yesterday too, but only for one of our users. The query succeeds for everyone else... (We were doing a fetch instead of a count, but otherwise it seems pretty similar) On Jan 8, 11:48 am, Ryan W rwilli...@gmail.com wrote: All of a sudden this morning, I began getting this error in my app, when doing a search inside of an internal process (searching for duplicates) and then taking a count() of the results. It was working fine before. I have this same query on a search page as well, and it's still fine. The search page does not call count() though. Perhaps it has something to do with the number of results being returned, I'll try to instrument to get a apples to apples search string comparison. Any ideas on this though? This is the stack trace: File /base/data/home/apps/..., line 196, in approvePost if qPosts.count() 0: File /base/python_lib/versions/1/google/appengine/ext/db/ __init__.py, line 1353, in count return self._get_query().Count(limit=limit) File /base/python_lib/versions/1/google/appengine/api/ datastore.py, line 960, in Count raise _ToDatastoreError(err) File /base/python_lib/versions/1/google/appengine/api/ datastore.py, line 1637, in _ToDatastoreError raise errors[err.application_error](err.error_detail) NeedIndexError: The built-in indices are not efficient enough for this query and your data. Please add a composite index for this query. This is the offending code: qPosts = datamodel.Post.all().search(search_string).filter(approved = , True) if qPosts.count() 0: This is the model: class Post(search.SearchableModel): user = db.ReferenceProperty(User, collection_name='posts') status_id = db.IntegerProperty(required=True) entry = db.TextProperty(required=True) entry_title = db.StringProperty(required=True,multiline=True) entry_link = db.LinkProperty(required=True) approved = db.BooleanProperty(required=True, default=False) reject_reason = db.StringListProperty(default=None) retweet_status_id = db.IntegerProperty(required=False) in_reply_to_status_id = db.IntegerProperty(required=False) short_url = db.LinkProperty(required=False) hash = db.StringProperty(required=False) source = db.StringProperty(required=False) view_count = db.IntegerProperty(required=True, default=0) reply_count = db.IntegerProperty(required=True, default=0) like_count = db.IntegerProperty(required=True, default=0) published = db.DateTimeProperty() created = db.DateTimeProperty(auto_now_add=True) I tried adding this index, but was still getting the error: - kind: Post properties: - name: __searchable_text_index - name: approved --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Trouble with NeedIndexError when calling count()
hi guys! merge join queries like these, ie queries that have equals filters on multiple properties but no inequality filters or sort orders, *usually* don't require an index. that's described in http://code.google.com/appengine/docs/datastore/queriesandindexes.html#Defining_Indexes_With_index_yaml . however, in cases with unusually shaped data, these queries can sometimes require an index. these cases are exceedingly rare, but they're somewhat more common with apphosting.ext.search. also, you need an index for each number of keywords that you expect in a query. for example, if you support three-word search queries, you need this index: - kind: Post properties: - name: __searchable_text_index - name: __searchable_text_index - name: __searchable_text_index - name: approved more in the ext.search docstring, http://code.google.com/p/googleappengine/source/browse/trunk/google/appengine/ext/search/__init__.py . also, unfortunately, these kinds of indices are likely to explode, as described in http://code.google.com/appengine/docs/datastore/queriesandindexes.html#Big_Entities_and_Exploding_Indexes . the upshot of all of this, as we've mentioned before, is that appengine.ext.search is a hack. it's not scalable, mature, or anywhere near industrial strength, which is why it's not officially supported or documented. we're well aware that it's not good enough, and we dearly hope to offer a real full text search solution in the future. i don't have any details or timeline, though. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
