Hi Ikai, is there a measure to say the entities are large? is it about the number of columns or blob or text? if so in my entities I have string and number properties only.
Yes. I need all the 2000 (some times bit more) entities at once for reporting. Currently I'm not seeing a way where I can split the keys and fetch. On Mar 21, 11:54 pm, "Ikai Lan (Google)" <ika...@google.com> wrote: > If that's the performance, then I'm inclined to say the entities are large > and that's about the performance you can expect. One more thing to keep in > mind: the cost of deserializing each entity is non-trivial. > > Is there a reason you need all 2000 entities? Perhaps there is a different > way to accomplish what you want without querying a (relatively) large number > of entities at once. > > One other possible solution that will at least make your user facing code > run faster is to split the query up by key, perform multiple async queries > and join them at the end. It'll still consume the same (or more) CPU time, > but the user ms should be lower since you are doing things in parallel. The > tradeoff here, of course, is that you'd need to know how you can split up > the keyspace. > > Ikai Lan > Developer Programs Engineer, Google App Engine > Blog:http://googleappengine.blogspot.com > Twitter:http://twitter.com/app_engine > Reddit:http://www.reddit.com/r/appengine > > > > > > > > On Mon, Mar 21, 2011 at 12:17 AM, adhi <adhinaraya...@gmail.com> wrote: > > q = db.Query(PrimaryData) > > q.filter('SheetMetadataId', metadata.getSheetId()) > > return q.fetch(2000) > > > Hi Ikai, even the above query takes 3.3 ~ 3.6 seconds. The number of > > entities returned by the query is 1261. > > I'm using Expando model, the total number columns for this particular > > set of entities are 25. > > Yes. I've created composite index. Here is the index definition. > > > - kind: PrimaryData > > properties: > > - name: SheetMetadataId > > - name: InstanceAssignedTo > > > please let me know if you need more information. > > > On Mar 17, 11:50 pm, "Ikai Lan (Google)" <ika...@google.com> wrote: > > > If you have a composite index, the performance of this query should be > > > mostly linear (find start location of index and just return min(2000, > > number > > > of entities that satisfy query from there on out) number of keys and > > items. > > > If you only have individual indexes for SheetMetadataId as well as > > > InstanceAssignedTo, this uses zigzag merge join, where performance can > > > greatly vary depending on the shape of the data. > > > > Just out of curiosity, what is the performance of this query? > > > > q = db.Query(PrimaryData) > > > q.filter('SheetMetadataId', metadata.getSheetId()) > > > return q.fetch(2000) > > > > If it's in a similar ballpark, my other guess is that the query takes > > that > > > amount of time because the entities are large. > > > > You mentioned created a composite index. Can you post that? > > > > I'll talk to the datastore team to see what we can do to provide > > something > > > similar to EXPLAIN plans for queries. > > > > Ikai Lan > > > Developer Programs Engineer, Google App Engine > > > Blog:http://googleappengine.blogspot.com > > > Twitter:http://twitter.com/app_engine > > > Reddit:http://www.reddit.com/r/appengine > > > > On Thu, Mar 17, 2011 at 10:07 AM, adhi <adhinaraya...@gmail.com> wrote: > > > > Hi Robert, > > > > I've not used reference properties. Its a simple query and it'll be > > > > like this. > > > > > q = db.Query(PrimaryData) > > > > q.filter('SheetMetadataId', metadata.getSheetId()) > > > > q.filter('InstanceAssignedTo IN', > > > > [u'User_c42e8919_448e_11e0_b87b_f58d20c6e2c3', > > > > u'User_1fd87ac5_073d_11e0_8ba1_c122a5867c4a']) > > > > return q.fetch(2000) > > > > > and I replaced the IN filter to '=' with single value. Still its > > > > taking same time. > > > > > On Mar 17, 6:27 am, Robert Kluin <robert.kl...@gmail.com> wrote: > > > > > Use Appstats. It may not be the query that is slow. If you are using > > > > > reference properties, perhaps you are dereferencing them. If you > > > > > should us the query and how you're using the results we might be abel > > > > > to give more suggestions. > > > > > >http://code.google.com/appengine/docs/python/tools/appstats.html > > > > > > Robert > > > > > > On Mon, Mar 14, 2011 at 10:54, adhi <adhinaraya...@gmail.com> wrote: > > > > > > Hi, I'm running a query in appengine which is returning just 1200 > > > > > > entities, but its taking 3.5 seconds. The query doesn't contains > > > > > > inequality filters, but anyway I added index for that. Can anyone > > tell > > > > > > me how to analyse this and improve theperformance? > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > Groups "Google App Engine" group. > > > > > > To post to this group, send email to > > google-appengine@googlegroups.com > > > > . > > > > > > To unsubscribe from this group, send email to > > > > google-appengine+unsubscr...@googlegroups.com. > > > > > > For more options, visit this group athttp:// > > > > groups.google.com/group/google-appengine?hl=en. > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Google App Engine" group. > > > > To post to this group, send email to google-appengine@googlegroups.com > > . > > > > To unsubscribe from this group, send email to > > > > google-appengine+unsubscr...@googlegroups.com. > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
