[google-appengine] Re: GQL performance slow

2011-03-23 Thread adhi
Hi Ikai
see the a href=http://i52.tinypic.com/28hndxk.jpg;
target=_blankapp stats/a

@905ms datastore_v3.RunQuery real=1090ms api=8225ms
@2504ms datastore_v3.Next real=339ms api=4711ms

my q.fetch(2000) method takes ~3.5 seconds

On Mar 23, 12:42 am, Ikai Lan (Google) ika...@google.com wrote:
 I think you should run AppStats and see what's going on. 2000 entities is
 never going to be particularly fast, but 3 seconds might be a bit on the
 high side. Run the tool to be sure.

 Ikai Lan
 Developer Programs Engineer, Google App Engine
 Blog:http://googleappengine.blogspot.com
 Twitter:http://twitter.com/app_engine
 Reddit:http://www.reddit.com/r/appengine







 On Tue, Mar 22, 2011 at 7:25 AM, adhi adhinaraya...@gmail.com wrote:
  Hi Ikai, is there a measure to say the entities are large? is it about
  the number of
  columns or blob or text? if so in my entities I have string and number
  properties only.

  Yes. I need all the 2000 (some times bit more) entities at once for
  reporting.
  Currently I'm not seeing a way where I can split the keys and fetch.

  On Mar 21, 11:54 pm, Ikai Lan (Google) ika...@google.com wrote:
   If that's the performance, then I'm inclined to say the entities are
  large
   and that's about the performance you can expect. One more thing to keep
  in
   mind: the cost of deserializing each entity is non-trivial.

   Is there a reason you need all 2000 entities? Perhaps there is a
  different
   way to accomplish what you want without querying a (relatively) large
  number
   of entities at once.

   One other possible solution that will at least make your user facing code
   run faster is to split the query up by key, perform multiple async
  queries
   and join them at the end. It'll still consume the same (or more) CPU
  time,
   but the user ms should be lower since you are doing things in parallel.
  The
   tradeoff here, of course, is that you'd need to know how you can split up
   the keyspace.

   Ikai Lan
   Developer Programs Engineer, Google App Engine
   Blog:http://googleappengine.blogspot.com
   Twitter:http://twitter.com/app_engine
   Reddit:http://www.reddit.com/r/appengine

   On Mon, Mar 21, 2011 at 12:17 AM, adhi adhinaraya...@gmail.com wrote:
q = db.Query(PrimaryData)
q.filter('SheetMetadataId', metadata.getSheetId())
return q.fetch(2000)

Hi Ikai, even the above query takes 3.3 ~ 3.6 seconds. The number of
entities returned by the query is 1261.
I'm using Expando model, the total number columns for this particular
set of entities are 25.
Yes. I've created composite index. Here is the index definition.

- kind: PrimaryData
 properties:
 - name: SheetMetadataId
 - name: InstanceAssignedTo

please let me know if you need more information.

On Mar 17, 11:50 pm, Ikai Lan (Google) ika...@google.com wrote:
 If you have a composite index, the performance of this query should
  be
 mostly linear (find start location of index and just return min(2000,
number
 of entities that satisfy query from there on out) number of keys and
items.
 If you only have individual indexes for SheetMetadataId as well as
 InstanceAssignedTo, this uses zigzag merge join, where performance
  can
 greatly vary depending on the shape of the data.

 Just out of curiosity, what is the performance of this query?

 q = db.Query(PrimaryData)
 q.filter('SheetMetadataId', metadata.getSheetId())
 return q.fetch(2000)

 If it's in a similar ballpark, my other guess is that the query takes
that
 amount of time because the entities are large.

 You mentioned created a composite index. Can you post that?

 I'll talk to the datastore team to see what we can do to provide
something
 similar to EXPLAIN plans for queries.

 Ikai Lan
 Developer Programs Engineer, Google App Engine
 Blog:http://googleappengine.blogspot.com
 Twitter:http://twitter.com/app_engine
 Reddit:http://www.reddit.com/r/appengine

 On Thu, Mar 17, 2011 at 10:07 AM, adhi adhinaraya...@gmail.com
  wrote:
  Hi Robert,
  I've not used reference properties. Its a simple query and it'll be
  like this.

  q = db.Query(PrimaryData)
  q.filter('SheetMetadataId', metadata.getSheetId())
  q.filter('InstanceAssignedTo IN',
  [u'User_c42e8919_448e_11e0_b87b_f58d20c6e2c3',
  u'User_1fd87ac5_073d_11e0_8ba1_c122a5867c4a'])
  return q.fetch(2000)

  and I replaced the IN filter to '=' with single value. Still its
  taking same time.

  On Mar 17, 6:27 am, Robert Kluin robert.kl...@gmail.com wrote:
   Use Appstats. It may not be the query that is slow.  If you are
  using
   reference properties, perhaps you are dereferencing them.  If you
   should us the query and how you're using the results we might be
  abel
   to give more suggestions.

  http://code.google.com/appengine/docs/python/tools/appstats.html

   Robert

   

[google-appengine] Re: GQL performance slow

2011-03-22 Thread adhi
Hi Ikai, is there a measure to say the entities are large? is it about
the number of
columns or blob or text? if so in my entities I have string and number
properties only.

Yes. I need all the 2000 (some times bit more) entities at once for
reporting.
Currently I'm not seeing a way where I can split the keys and fetch.


On Mar 21, 11:54 pm, Ikai Lan (Google) ika...@google.com wrote:
 If that's the performance, then I'm inclined to say the entities are large
 and that's about the performance you can expect. One more thing to keep in
 mind: the cost of deserializing each entity is non-trivial.

 Is there a reason you need all 2000 entities? Perhaps there is a different
 way to accomplish what you want without querying a (relatively) large number
 of entities at once.

 One other possible solution that will at least make your user facing code
 run faster is to split the query up by key, perform multiple async queries
 and join them at the end. It'll still consume the same (or more) CPU time,
 but the user ms should be lower since you are doing things in parallel. The
 tradeoff here, of course, is that you'd need to know how you can split up
 the keyspace.

 Ikai Lan
 Developer Programs Engineer, Google App Engine
 Blog:http://googleappengine.blogspot.com
 Twitter:http://twitter.com/app_engine
 Reddit:http://www.reddit.com/r/appengine







 On Mon, Mar 21, 2011 at 12:17 AM, adhi adhinaraya...@gmail.com wrote:
  q = db.Query(PrimaryData)
  q.filter('SheetMetadataId', metadata.getSheetId())
  return q.fetch(2000)

  Hi Ikai, even the above query takes 3.3 ~ 3.6 seconds. The number of
  entities returned by the query is 1261.
  I'm using Expando model, the total number columns for this particular
  set of entities are 25.
  Yes. I've created composite index. Here is the index definition.

  - kind: PrimaryData
   properties:
   - name: SheetMetadataId
   - name: InstanceAssignedTo

  please let me know if you need more information.

  On Mar 17, 11:50 pm, Ikai Lan (Google) ika...@google.com wrote:
   If you have a composite index, the performance of this query should be
   mostly linear (find start location of index and just return min(2000,
  number
   of entities that satisfy query from there on out) number of keys and
  items.
   If you only have individual indexes for SheetMetadataId as well as
   InstanceAssignedTo, this uses zigzag merge join, where performance can
   greatly vary depending on the shape of the data.

   Just out of curiosity, what is the performance of this query?

   q = db.Query(PrimaryData)
   q.filter('SheetMetadataId', metadata.getSheetId())
   return q.fetch(2000)

   If it's in a similar ballpark, my other guess is that the query takes
  that
   amount of time because the entities are large.

   You mentioned created a composite index. Can you post that?

   I'll talk to the datastore team to see what we can do to provide
  something
   similar to EXPLAIN plans for queries.

   Ikai Lan
   Developer Programs Engineer, Google App Engine
   Blog:http://googleappengine.blogspot.com
   Twitter:http://twitter.com/app_engine
   Reddit:http://www.reddit.com/r/appengine

   On Thu, Mar 17, 2011 at 10:07 AM, adhi adhinaraya...@gmail.com wrote:
Hi Robert,
I've not used reference properties. Its a simple query and it'll be
like this.

q = db.Query(PrimaryData)
q.filter('SheetMetadataId', metadata.getSheetId())
q.filter('InstanceAssignedTo IN',
[u'User_c42e8919_448e_11e0_b87b_f58d20c6e2c3',
u'User_1fd87ac5_073d_11e0_8ba1_c122a5867c4a'])
return q.fetch(2000)

and I replaced the IN filter to '=' with single value. Still its
taking same time.

On Mar 17, 6:27 am, Robert Kluin robert.kl...@gmail.com wrote:
 Use Appstats. It may not be the query that is slow.  If you are using
 reference properties, perhaps you are dereferencing them.  If you
 should us the query and how you're using the results we might be abel
 to give more suggestions.

http://code.google.com/appengine/docs/python/tools/appstats.html

 Robert

 On Mon, Mar 14, 2011 at 10:54, adhi adhinaraya...@gmail.com wrote:
  Hi, I'm running a query in appengine which is returning just 1200
  entities, but its taking 3.5 seconds. The query doesn't contains
  inequality filters, but anyway I added index for that. Can anyone
  tell
  me how to analyse this and improve theperformance?

  --
  You received this message because you are subscribed to the Google
Groups Google App Engine group.
  To post to this group, send email to
  google-appengine@googlegroups.com
.
  To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group athttp://
groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google
  Groups
Google App Engine group.
To post to this group, send email to 

Re: [google-appengine] Re: GQL performance slow

2011-03-22 Thread Ikai Lan (Google)
I think you should run AppStats and see what's going on. 2000 entities is
never going to be particularly fast, but 3 seconds might be a bit on the
high side. Run the tool to be sure.

Ikai Lan
Developer Programs Engineer, Google App Engine
Blog: http://googleappengine.blogspot.com
Twitter: http://twitter.com/app_engine
Reddit: http://www.reddit.com/r/appengine



On Tue, Mar 22, 2011 at 7:25 AM, adhi adhinaraya...@gmail.com wrote:

 Hi Ikai, is there a measure to say the entities are large? is it about
 the number of
 columns or blob or text? if so in my entities I have string and number
 properties only.

 Yes. I need all the 2000 (some times bit more) entities at once for
 reporting.
 Currently I'm not seeing a way where I can split the keys and fetch.


 On Mar 21, 11:54 pm, Ikai Lan (Google) ika...@google.com wrote:
  If that's the performance, then I'm inclined to say the entities are
 large
  and that's about the performance you can expect. One more thing to keep
 in
  mind: the cost of deserializing each entity is non-trivial.
 
  Is there a reason you need all 2000 entities? Perhaps there is a
 different
  way to accomplish what you want without querying a (relatively) large
 number
  of entities at once.
 
  One other possible solution that will at least make your user facing code
  run faster is to split the query up by key, perform multiple async
 queries
  and join them at the end. It'll still consume the same (or more) CPU
 time,
  but the user ms should be lower since you are doing things in parallel.
 The
  tradeoff here, of course, is that you'd need to know how you can split up
  the keyspace.
 
  Ikai Lan
  Developer Programs Engineer, Google App Engine
  Blog:http://googleappengine.blogspot.com
  Twitter:http://twitter.com/app_engine
  Reddit:http://www.reddit.com/r/appengine
 
 
 
 
 
 
 
  On Mon, Mar 21, 2011 at 12:17 AM, adhi adhinaraya...@gmail.com wrote:
   q = db.Query(PrimaryData)
   q.filter('SheetMetadataId', metadata.getSheetId())
   return q.fetch(2000)
 
   Hi Ikai, even the above query takes 3.3 ~ 3.6 seconds. The number of
   entities returned by the query is 1261.
   I'm using Expando model, the total number columns for this particular
   set of entities are 25.
   Yes. I've created composite index. Here is the index definition.
 
   - kind: PrimaryData
properties:
- name: SheetMetadataId
- name: InstanceAssignedTo
 
   please let me know if you need more information.
 
   On Mar 17, 11:50 pm, Ikai Lan (Google) ika...@google.com wrote:
If you have a composite index, the performance of this query should
 be
mostly linear (find start location of index and just return min(2000,
   number
of entities that satisfy query from there on out) number of keys and
   items.
If you only have individual indexes for SheetMetadataId as well as
InstanceAssignedTo, this uses zigzag merge join, where performance
 can
greatly vary depending on the shape of the data.
 
Just out of curiosity, what is the performance of this query?
 
q = db.Query(PrimaryData)
q.filter('SheetMetadataId', metadata.getSheetId())
return q.fetch(2000)
 
If it's in a similar ballpark, my other guess is that the query takes
   that
amount of time because the entities are large.
 
You mentioned created a composite index. Can you post that?
 
I'll talk to the datastore team to see what we can do to provide
   something
similar to EXPLAIN plans for queries.
 
Ikai Lan
Developer Programs Engineer, Google App Engine
Blog:http://googleappengine.blogspot.com
Twitter:http://twitter.com/app_engine
Reddit:http://www.reddit.com/r/appengine
 
On Thu, Mar 17, 2011 at 10:07 AM, adhi adhinaraya...@gmail.com
 wrote:
 Hi Robert,
 I've not used reference properties. Its a simple query and it'll be
 like this.
 
 q = db.Query(PrimaryData)
 q.filter('SheetMetadataId', metadata.getSheetId())
 q.filter('InstanceAssignedTo IN',
 [u'User_c42e8919_448e_11e0_b87b_f58d20c6e2c3',
 u'User_1fd87ac5_073d_11e0_8ba1_c122a5867c4a'])
 return q.fetch(2000)
 
 and I replaced the IN filter to '=' with single value. Still its
 taking same time.
 
 On Mar 17, 6:27 am, Robert Kluin robert.kl...@gmail.com wrote:
  Use Appstats. It may not be the query that is slow.  If you are
 using
  reference properties, perhaps you are dereferencing them.  If you
  should us the query and how you're using the results we might be
 abel
  to give more suggestions.
 
 http://code.google.com/appengine/docs/python/tools/appstats.html
 
  Robert
 
  On Mon, Mar 14, 2011 at 10:54, adhi adhinaraya...@gmail.com
 wrote:
   Hi, I'm running a query in appengine which is returning just
 1200
   entities, but its taking 3.5 seconds. The query doesn't
 contains
   inequality filters, but anyway I added index for that. Can
 anyone
   tell
   me how to analyse this and improve theperformance?
 
   

[google-appengine] Re: GQL performance slow

2011-03-21 Thread adhi
q = db.Query(PrimaryData)
q.filter('SheetMetadataId', metadata.getSheetId())
return q.fetch(2000)

Hi Ikai, even the above query takes 3.3 ~ 3.6 seconds. The number of
entities returned by the query is 1261.
I'm using Expando model, the total number columns for this particular
set of entities are 25.
Yes. I've created composite index. Here is the index definition.

- kind: PrimaryData
  properties:
  - name: SheetMetadataId
  - name: InstanceAssignedTo

please let me know if you need more information.


On Mar 17, 11:50 pm, Ikai Lan (Google) ika...@google.com wrote:
 If you have a composite index, the performance of this query should be
 mostly linear (find start location of index and just return min(2000, number
 of entities that satisfy query from there on out) number of keys and items.
 If you only have individual indexes for SheetMetadataId as well as
 InstanceAssignedTo, this uses zigzag merge join, where performance can
 greatly vary depending on the shape of the data.

 Just out of curiosity, what is the performance of this query?

 q = db.Query(PrimaryData)
 q.filter('SheetMetadataId', metadata.getSheetId())
 return q.fetch(2000)

 If it's in a similar ballpark, my other guess is that the query takes that
 amount of time because the entities are large.

 You mentioned created a composite index. Can you post that?

 I'll talk to the datastore team to see what we can do to provide something
 similar to EXPLAIN plans for queries.

 Ikai Lan
 Developer Programs Engineer, Google App Engine
 Blog:http://googleappengine.blogspot.com
 Twitter:http://twitter.com/app_engine
 Reddit:http://www.reddit.com/r/appengine







 On Thu, Mar 17, 2011 at 10:07 AM, adhi adhinaraya...@gmail.com wrote:
  Hi Robert,
  I've not used reference properties. Its a simple query and it'll be
  like this.

  q = db.Query(PrimaryData)
  q.filter('SheetMetadataId', metadata.getSheetId())
  q.filter('InstanceAssignedTo IN',
  [u'User_c42e8919_448e_11e0_b87b_f58d20c6e2c3',
  u'User_1fd87ac5_073d_11e0_8ba1_c122a5867c4a'])
  return q.fetch(2000)

  and I replaced the IN filter to '=' with single value. Still its
  taking same time.

  On Mar 17, 6:27 am, Robert Kluin robert.kl...@gmail.com wrote:
   Use Appstats. It may not be the query that is slow.  If you are using
   reference properties, perhaps you are dereferencing them.  If you
   should us the query and how you're using the results we might be abel
   to give more suggestions.

  http://code.google.com/appengine/docs/python/tools/appstats.html

   Robert

   On Mon, Mar 14, 2011 at 10:54, adhi adhinaraya...@gmail.com wrote:
Hi, I'm running a query in appengine which is returning just 1200
entities, but its taking 3.5 seconds. The query doesn't contains
inequality filters, but anyway I added index for that. Can anyone tell
me how to analyse this and improve theperformance?

--
You received this message because you are subscribed to the Google
  Groups Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
  .
To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group athttp://
  groups.google.com/group/google-appengine?hl=en.

  --
  You received this message because you are subscribed to the Google Groups
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



Re: [google-appengine] Re: GQL performance slow

2011-03-21 Thread Ikai Lan (Google)
If that's the performance, then I'm inclined to say the entities are large
and that's about the performance you can expect. One more thing to keep in
mind: the cost of deserializing each entity is non-trivial.

Is there a reason you need all 2000 entities? Perhaps there is a different
way to accomplish what you want without querying a (relatively) large number
of entities at once.

One other possible solution that will at least make your user facing code
run faster is to split the query up by key, perform multiple async queries
and join them at the end. It'll still consume the same (or more) CPU time,
but the user ms should be lower since you are doing things in parallel. The
tradeoff here, of course, is that you'd need to know how you can split up
the keyspace.

Ikai Lan
Developer Programs Engineer, Google App Engine
Blog: http://googleappengine.blogspot.com
Twitter: http://twitter.com/app_engine
Reddit: http://www.reddit.com/r/appengine



On Mon, Mar 21, 2011 at 12:17 AM, adhi adhinaraya...@gmail.com wrote:

 q = db.Query(PrimaryData)
 q.filter('SheetMetadataId', metadata.getSheetId())
 return q.fetch(2000)

 Hi Ikai, even the above query takes 3.3 ~ 3.6 seconds. The number of
 entities returned by the query is 1261.
 I'm using Expando model, the total number columns for this particular
 set of entities are 25.
 Yes. I've created composite index. Here is the index definition.

 - kind: PrimaryData
  properties:
  - name: SheetMetadataId
  - name: InstanceAssignedTo

 please let me know if you need more information.


 On Mar 17, 11:50 pm, Ikai Lan (Google) ika...@google.com wrote:
  If you have a composite index, the performance of this query should be
  mostly linear (find start location of index and just return min(2000,
 number
  of entities that satisfy query from there on out) number of keys and
 items.
  If you only have individual indexes for SheetMetadataId as well as
  InstanceAssignedTo, this uses zigzag merge join, where performance can
  greatly vary depending on the shape of the data.
 
  Just out of curiosity, what is the performance of this query?
 
  q = db.Query(PrimaryData)
  q.filter('SheetMetadataId', metadata.getSheetId())
  return q.fetch(2000)
 
  If it's in a similar ballpark, my other guess is that the query takes
 that
  amount of time because the entities are large.
 
  You mentioned created a composite index. Can you post that?
 
  I'll talk to the datastore team to see what we can do to provide
 something
  similar to EXPLAIN plans for queries.
 
  Ikai Lan
  Developer Programs Engineer, Google App Engine
  Blog:http://googleappengine.blogspot.com
  Twitter:http://twitter.com/app_engine
  Reddit:http://www.reddit.com/r/appengine
 
 
 
 
 
 
 
  On Thu, Mar 17, 2011 at 10:07 AM, adhi adhinaraya...@gmail.com wrote:
   Hi Robert,
   I've not used reference properties. Its a simple query and it'll be
   like this.
 
   q = db.Query(PrimaryData)
   q.filter('SheetMetadataId', metadata.getSheetId())
   q.filter('InstanceAssignedTo IN',
   [u'User_c42e8919_448e_11e0_b87b_f58d20c6e2c3',
   u'User_1fd87ac5_073d_11e0_8ba1_c122a5867c4a'])
   return q.fetch(2000)
 
   and I replaced the IN filter to '=' with single value. Still its
   taking same time.
 
   On Mar 17, 6:27 am, Robert Kluin robert.kl...@gmail.com wrote:
Use Appstats. It may not be the query that is slow.  If you are using
reference properties, perhaps you are dereferencing them.  If you
should us the query and how you're using the results we might be abel
to give more suggestions.
 
   http://code.google.com/appengine/docs/python/tools/appstats.html
 
Robert
 
On Mon, Mar 14, 2011 at 10:54, adhi adhinaraya...@gmail.com wrote:
 Hi, I'm running a query in appengine which is returning just 1200
 entities, but its taking 3.5 seconds. The query doesn't contains
 inequality filters, but anyway I added index for that. Can anyone
 tell
 me how to analyse this and improve theperformance?
 
 --
 You received this message because you are subscribed to the Google
   Groups Google App Engine group.
 To post to this group, send email to
 google-appengine@googlegroups.com
   .
 To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group athttp://
   groups.google.com/group/google-appengine?hl=en.
 
   --
   You received this message because you are subscribed to the Google
 Groups
   Google App Engine group.
   To post to this group, send email to google-appengine@googlegroups.com
 .
   To unsubscribe from this group, send email to
   google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group at
  http://groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 

[google-appengine] Re: GQL performance slow

2011-03-17 Thread adhi
Hi Robert,
I've not used reference properties. Its a simple query and it'll be
like this.

q = db.Query(PrimaryData)
q.filter('SheetMetadataId', metadata.getSheetId())
q.filter('InstanceAssignedTo IN',
[u'User_c42e8919_448e_11e0_b87b_f58d20c6e2c3',
u'User_1fd87ac5_073d_11e0_8ba1_c122a5867c4a'])
return q.fetch(2000)

and I replaced the IN filter to '=' with single value. Still its
taking same time.



On Mar 17, 6:27 am, Robert Kluin robert.kl...@gmail.com wrote:
 Use Appstats. It may not be the query that is slow.  If you are using
 reference properties, perhaps you are dereferencing them.  If you
 should us the query and how you're using the results we might be abel
 to give more suggestions.

 http://code.google.com/appengine/docs/python/tools/appstats.html

 Robert







 On Mon, Mar 14, 2011 at 10:54, adhi adhinaraya...@gmail.com wrote:
  Hi, I'm running a query in appengine which is returning just 1200
  entities, but its taking 3.5 seconds. The query doesn't contains
  inequality filters, but anyway I added index for that. Can anyone tell
  me how to analyse this and improve theperformance?

  --
  You received this message because you are subscribed to the Google Groups 
  Google App Engine group.
  To post to this group, send email to google-appengine@googlegroups.com.
  To unsubscribe from this group, send email to 
  google-appengine+unsubscr...@googlegroups.com.
  For more options, visit this group 
  athttp://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



Re: [google-appengine] Re: GQL performance slow

2011-03-17 Thread Ikai Lan (Google)
If you have a composite index, the performance of this query should be
mostly linear (find start location of index and just return min(2000, number
of entities that satisfy query from there on out) number of keys and items.
If you only have individual indexes for SheetMetadataId as well as
InstanceAssignedTo, this uses zigzag merge join, where performance can
greatly vary depending on the shape of the data.

Just out of curiosity, what is the performance of this query?

q = db.Query(PrimaryData)
q.filter('SheetMetadataId', metadata.getSheetId())
return q.fetch(2000)

If it's in a similar ballpark, my other guess is that the query takes that
amount of time because the entities are large.

You mentioned created a composite index. Can you post that?

I'll talk to the datastore team to see what we can do to provide something
similar to EXPLAIN plans for queries.

Ikai Lan
Developer Programs Engineer, Google App Engine
Blog: http://googleappengine.blogspot.com
Twitter: http://twitter.com/app_engine
Reddit: http://www.reddit.com/r/appengine



On Thu, Mar 17, 2011 at 10:07 AM, adhi adhinaraya...@gmail.com wrote:

 Hi Robert,
 I've not used reference properties. Its a simple query and it'll be
 like this.

 q = db.Query(PrimaryData)
 q.filter('SheetMetadataId', metadata.getSheetId())
 q.filter('InstanceAssignedTo IN',
 [u'User_c42e8919_448e_11e0_b87b_f58d20c6e2c3',
 u'User_1fd87ac5_073d_11e0_8ba1_c122a5867c4a'])
 return q.fetch(2000)

 and I replaced the IN filter to '=' with single value. Still its
 taking same time.



 On Mar 17, 6:27 am, Robert Kluin robert.kl...@gmail.com wrote:
  Use Appstats. It may not be the query that is slow.  If you are using
  reference properties, perhaps you are dereferencing them.  If you
  should us the query and how you're using the results we might be abel
  to give more suggestions.
 
  http://code.google.com/appengine/docs/python/tools/appstats.html
 
  Robert
 
 
 
 
 
 
 
  On Mon, Mar 14, 2011 at 10:54, adhi adhinaraya...@gmail.com wrote:
   Hi, I'm running a query in appengine which is returning just 1200
   entities, but its taking 3.5 seconds. The query doesn't contains
   inequality filters, but anyway I added index for that. Can anyone tell
   me how to analyse this and improve theperformance?
 
   --
   You received this message because you are subscribed to the Google
 Groups Google App Engine group.
   To post to this group, send email to google-appengine@googlegroups.com
 .
   To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
   For more options, visit this group athttp://
 groups.google.com/group/google-appengine?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
 Google App Engine group.
 To post to this group, send email to google-appengine@googlegroups.com.
 To unsubscribe from this group, send email to
 google-appengine+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/google-appengine?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.