I am not sure why this problem is unattended. I think it is quite serious: "Indexing seems to fail during heavy entity writes. There after the query simply does not return correct results"
The only way we were able to get this working was to read all the 1 million entities out the datastore and write it back on a live "production server". If someone is interested to look into this problem, pls let us know. thanks, mani On Nov 13, 12:09 pm, mani doraisamy <mdorais...@orangescape.com> wrote: > Martin, > In this case, we have a property called "ModifiedAt". This property > has neither been added recently nor is it missing in one of the > entities. But it still does not return consistent results. In fact, > when an additional AND condition is added to the query (for testing), > the entity starts appearing in the result, even though these entities > satisfy both the conditions. > > thanks, > mani > > On Nov 9, 6:31 pm, Martin Trummer <martin.trummer...@gmail.com> wrote: > > > > > I can imagine just one cenario that would lead to > > this behaviour: > > > say, i have an entity that has a property A > > I create 10 of these entities > > > the I update my application and I added a property B > > now I store another 10 of those entities to the datastore > > > in the datastore there are now 20 of these entities > > 10 have only property A > > 10 have properties: A, B > > > when you now build an index on property B, it will only > > include the 10 entities that actually have the property B > > > now a query on all entities would give 20 results > > the same query ordered by B would give only 10 results > > > when you now edit one of the old entities that only have > > property A and save this entity, the saved entity will > > also include entity B, so that > > > now a query on all entities would give 20 results > > the same query ordered by B would give only 11 results > > > On Nov 9, 11:49 am, Adhi <adhi.ramanat...@orangescape.com> wrote: > > > > Martin > > > Thanks for the info. But I explicitly created all the necessary > > > indexes. > > > > When debugging this issue I felt there might be a problem in building > > > the indexes, > > > if I just open and save the missing record then that record is also > > > included > > > in the resultset. > > > > So I think when I the record updation has to do something with the > > > query execution. > > > Any clues..? > > > > Adhi > > > > On Nov 9, 10:28 am, Jason Smith <j...@proven-corporation.com> wrote: > > > > > I have the same problem, which I wrote about on Stack Overflow but > > > > received no response. > > > > >http://stackoverflow.com/questions/1691792/query-gqlquery-order-restr... > > > > > My models require the property in question and I manually confirmed > > > > that they are all present, so it is not an issue of queries not > > > > returning entities with missing properties. I am stuck with this > > > > problem, and currently I am working around it by fetching all data and > > > > sorting in memory. Fortunately I can get away with that as it's a > > > > small data set and in infrequent query. > > > > > On Nov 7, 12:53 am, Adhi <adhi.ramanat...@orangescape.com> wrote: > > > > > > Yes, I've tried using order by also. But its giving different > > > > > resultset. > > > > > When using order by I got only 842 records, but with out order by I > > > > > got 1251 > > > > > where as my actual records will be >1260. and when I change the fetch > > > > > size > > > > > I'm getting different count. > > > > > > Here is my code... > > > > > > def get_serialized_data(entityClass, params): > > > > > query = entityClass.all() > > > > > query.order('__key__') > > > > > > for filterColumn, filterValue in params.iteritems(): > > > > > query.filter(filterColumn, filterValue) > > > > > limit = 400 > > > > > offset = 0 > > > > > totalLimit = 800 > > > > > lastRecordKey = None > > > > > n = 0 > > > > > entities = query.fetch(limit, offset) > > > > > while entities and offset <= (totalLimit-limit): > > > > > lastRecordKey = entities[-1].key() > > > > > n += len(entities) > > > > > # My serialization code here > > > > > offset+=limit > > > > > if len(entities)==limit: > > > > > entities = query.fetch(limit, offset) > > > > > else: > > > > > entities = None > > > > > entities = None > > > > > return (n>=totalLimit, lastRecordKey) > > > > > > def download_data(): > > > > > params = {'ApplicationId':applicationId, 'Deleted':False, > > > > > 'SheetMetadataId':'Sheet003'} > > > > > (moreRecords, lastRecordKey) = get_serialized_data(PrimaryData, > > > > > params) > > > > > while moreRecords: > > > > > params['__key__ >'] = lastRecordKey > > > > > (moreRecords, lastRecordKey) = get_serialized_data > > > > > (PrimaryData, params) > > > > > > download_data() > > > > > > Each batch will fetch 800 records if I use q.fetch(800) its giving > > > > > Timeout so I've used offset. > > > > > As per the documentation > > > > > inhttp://code.google.com/appengine/articles/remote_api.html > > > > > they haven't specified > > > > > to add order by for __key__ so I thought its implicit. Thats why I > > > > > initially tried with out order by. > > > > > Am I doing anything wrong? > > > > > > Now I'm trying to delete and recreating the indexes because of this > > > > > problem, but it still in deleting state. > > > > > > Adhi > > > > > > On Nov 6, 7:13 pm, Eli Jones <eli.jo...@gmail.com> wrote: > > > > > > > Always post a full code snippet. > > > > > > > Aren't you supposed to use Order By when paging by key? > > > > > > > On 11/6/09, Adhi <adhi.ramanat...@orangescape.com> wrote: > > > > > > > > Hi, > > > > > > > Sometimes I am not getting the complete resultset fromgqleven > > > > > > > though > > > > > > > the missing records satisfies the condition. I've proper indexes. > > > > > > > Total records for that query will be around 1300. So, I'm not > > > > > > > fetching > > > > > > > the records in a single fetch, I'm using __key__ > > > > > > > > last_record_key to > > > > > > > get in batches. > > > > > > > > Why is this anomaly..? anything I am missing here. > > > > > > > > Adhi > > > > > > > -- > > > > > > Sent from my mobile device -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
