Hi Ikai see the <a href="http://i52.tinypic.com/28hndxk.jpg"; target="_blank">app stats</a>
@905ms datastore_v3.RunQuery real=1090ms api=8225ms @2504ms datastore_v3.Next real=339ms api=4711ms my q.fetch(2000) method takes ~3.5 seconds On Mar 23, 12:42 am, "Ikai Lan (Google)" <ika...@google.com> wrote: > I think you should run AppStats and see what's going on. 2000 entities is > never going to be particularly fast, but 3 seconds might be a bit on the > high side. Run the tool to be sure. > > Ikai Lan > Developer Programs Engineer, Google App Engine > Blog:http://googleappengine.blogspot.com > Twitter:http://twitter.com/app_engine > Reddit:http://www.reddit.com/r/appengine > > > > > > > > On Tue, Mar 22, 2011 at 7:25 AM, adhi <adhinaraya...@gmail.com> wrote: > > Hi Ikai, is there a measure to say the entities are large? is it about > > the number of > > columns or blob or text? if so in my entities I have string and number > > properties only. > > > Yes. I need all the 2000 (some times bit more) entities at once for > > reporting. > > Currently I'm not seeing a way where I can split the keys and fetch. > > > On Mar 21, 11:54 pm, "Ikai Lan (Google)" <ika...@google.com> wrote: > > > If that's the performance, then I'm inclined to say the entities are > > large > > > and that's about the performance you can expect. One more thing to keep > > in > > > mind: the cost of deserializing each entity is non-trivial. > > > > Is there a reason you need all 2000 entities? Perhaps there is a > > different > > > way to accomplish what you want without querying a (relatively) large > > number > > > of entities at once. > > > > One other possible solution that will at least make your user facing code > > > run faster is to split the query up by key, perform multiple async > > queries > > > and join them at the end. It'll still consume the same (or more) CPU > > time, > > > but the user ms should be lower since you are doing things in parallel. > > The > > > tradeoff here, of course, is that you'd need to know how you can split up > > > the keyspace. > > > > Ikai Lan > > > Developer Programs Engineer, Google App Engine > > > Blog:http://googleappengine.blogspot.com > > > Twitter:http://twitter.com/app_engine > > > Reddit:http://www.reddit.com/r/appengine > > > > On Mon, Mar 21, 2011 at 12:17 AM, adhi <adhinaraya...@gmail.com> wrote: > > > > q = db.Query(PrimaryData) > > > > q.filter('SheetMetadataId', metadata.getSheetId()) > > > > return q.fetch(2000) > > > > > Hi Ikai, even the above query takes 3.3 ~ 3.6 seconds. The number of > > > > entities returned by the query is 1261. > > > > I'm using Expando model, the total number columns for this particular > > > > set of entities are 25. > > > > Yes. I've created composite index. Here is the index definition. > > > > > - kind: PrimaryData > > > > properties: > > > > - name: SheetMetadataId > > > > - name: InstanceAssignedTo > > > > > please let me know if you need more information. > > > > > On Mar 17, 11:50 pm, "Ikai Lan (Google)" <ika...@google.com> wrote: > > > > > If you have a composite index, the performance of this query should > > be > > > > > mostly linear (find start location of index and just return min(2000, > > > > number > > > > > of entities that satisfy query from there on out) number of keys and > > > > items. > > > > > If you only have individual indexes for SheetMetadataId as well as > > > > > InstanceAssignedTo, this uses zigzag merge join, where performance > > can > > > > > greatly vary depending on the shape of the data. > > > > > > Just out of curiosity, what is the performance of this query? > > > > > > q = db.Query(PrimaryData) > > > > > q.filter('SheetMetadataId', metadata.getSheetId()) > > > > > return q.fetch(2000) > > > > > > If it's in a similar ballpark, my other guess is that the query takes > > > > that > > > > > amount of time because the entities are large. > > > > > > You mentioned created a composite index. Can you post that? > > > > > > I'll talk to the datastore team to see what we can do to provide > > > > something > > > > > similar to EXPLAIN plans for queries. > > > > > > Ikai Lan > > > > > Developer Programs Engineer, Google App Engine > > > > > Blog:http://googleappengine.blogspot.com > > > > > Twitter:http://twitter.com/app_engine > > > > > Reddit:http://www.reddit.com/r/appengine > > > > > > On Thu, Mar 17, 2011 at 10:07 AM, adhi <adhinaraya...@gmail.com> > > wrote: > > > > > > Hi Robert, > > > > > > I've not used reference properties. Its a simple query and it'll be > > > > > > like this. > > > > > > > q = db.Query(PrimaryData) > > > > > > q.filter('SheetMetadataId', metadata.getSheetId()) > > > > > > q.filter('InstanceAssignedTo IN', > > > > > > [u'User_c42e8919_448e_11e0_b87b_f58d20c6e2c3', > > > > > > u'User_1fd87ac5_073d_11e0_8ba1_c122a5867c4a']) > > > > > > return q.fetch(2000) > > > > > > > and I replaced the IN filter to '=' with single value. Still its > > > > > > taking same time. > > > > > > > On Mar 17, 6:27 am, Robert Kluin <robert.kl...@gmail.com> wrote: > > > > > > > Use Appstats. It may not be the query that is slow. If you are > > using > > > > > > > reference properties, perhaps you are dereferencing them. If you > > > > > > > should us the query and how you're using the results we might be > > abel > > > > > > > to give more suggestions. > > > > > > > >http://code.google.com/appengine/docs/python/tools/appstats.html > > > > > > > > Robert > > > > > > > > On Mon, Mar 14, 2011 at 10:54, adhi <adhinaraya...@gmail.com> > > wrote: > > > > > > > > Hi, I'm running a query in appengine which is returning just > > 1200 > > > > > > > > entities, but its taking 3.5 seconds. The query doesn't > > contains > > > > > > > > inequality filters, but anyway I added index for that. Can > > anyone > > > > tell > > > > > > > > me how to analyse this and improve theperformance? > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the > > Google > > > > > > Groups "Google App Engine" group. > > > > > > > > To post to this group, send email to > > > > google-appengine@googlegroups.com > > > > > > . > > > > > > > > To unsubscribe from this group, send email to > > > > > > google-appengine+unsubscr...@googlegroups.com. > > > > > > > > For more options, visit this group athttp:// > > > > > > groups.google.com/group/google-appengine?hl=en. > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > Groups > > > > > > "Google App Engine" group. > > > > > > To post to this group, send email to > > google-appengine@googlegroups.com > > > > . > > > > > > To unsubscribe from this group, send email to > > > > > > google-appengine+unsubscr...@googlegroups.com. > > > > > > For more options, visit this group at > > > > > >http://groups.google.com/group/google-appengine?hl=en. > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to google-appengine@googlegroups.com > > . > > > > To unsubscribe from this group, send email to > > > > google-appengine+unsubscr...@googlegroups.com. > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
![http://i52.tinypic.com/28hndxk.jpg"](http://i52.tinypic.com/28hndxk.jpg"){kind=link}