[google-appengine] Re: Datastore delete taking long time
Hi Anders, How are you deleting the entities? Are you fetching them first, or deleting them by providing the keys? And are you deleting them in a single batch, or one at a time? -Nick Johnson On Wed, Jun 24, 2009 at 7:44 AM, Anders i...@blabline.com wrote: Has the delete function of records in the datastore started to take much longer time? One Ajax call I have that deletes up to 20 records often (more often than not perhaps) takes a very long time, where a single request consumes around 8000cpu_ms and around 8000api_cpu_ms. The records deleted are simple with only 10 properties of which only one is a ReferenceProperty and the rest are single-line no-reference properties. And no complicated indexes are used (as far as I know). -- Nick Johnson, App Engine Developer Programs Engineer Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore delete taking long time
First I get the entities using a query, something like:
entities = entity.Entity.gql('WHERE account_key = :1 AND topic = :2
ORDER BY message_count DESC LIMIT 20', account_key, topic)
Then a loop calling delete() on each entity:
for entity in entities:
entity.delete()
On Jun 24, 12:29 pm, Nick Johnson (Google) nick.john...@google.com
wrote:
Hi Anders,
How are you deleting the entities? Are you fetching them first, or deleting
them by providing the keys? And are you deleting them in a single batch, or
one at a time?
-Nick Johnson
On Wed, Jun 24, 2009 at 7:44 AM, Anders i...@blabline.com wrote:
Has the delete function of records in the datastore started to take
much longer time? One Ajax call I have that deletes up to 20 records
often (more often than not perhaps) takes a very long time, where a
single request consumes around 8000cpu_ms and around 8000api_cpu_ms.
The records deleted are simple with only 10 properties of which only
one is a ReferenceProperty and the rest are single-line no-reference
properties. And no complicated indexes are used (as far as I know).
--
Nick Johnson, App Engine Developer Programs Engineer
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
368047
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Datastore delete taking long time
Hi Anders,
You don't actually need the entities in order to delete them - only their
keys. Also, you can do all the deletes in a single batch operation. This
ought to be substantially faster:
db.delete(db.GqlQuery(SELECT __key__ FROM Entity WHERE account_key = :1 AND
topic = :2 ORDER BY message_count DESC, account_key, topic).fetch(20))
or equivalently with the Query api:
db.delete(entity.Entity.all(keys_only=True).filter('account_key',
account_key).filter('topic', topic).order('-message_count').fetch(20))
-Nick Johnson
On Wed, Jun 24, 2009 at 11:45 AM, Anders i...@blabline.com wrote:
First I get the entities using a query, something like:
entities = entity.Entity.gql('WHERE account_key = :1 AND topic = :2
ORDER BY message_count DESC LIMIT 20', account_key, topic)
Then a loop calling delete() on each entity:
for entity in entities:
entity.delete()
On Jun 24, 12:29 pm, Nick Johnson (Google) nick.john...@google.com
wrote:
Hi Anders,
How are you deleting the entities? Are you fetching them first, or
deleting
them by providing the keys? And are you deleting them in a single batch,
or
one at a time?
-Nick Johnson
On Wed, Jun 24, 2009 at 7:44 AM, Anders i...@blabline.com wrote:
Has the delete function of records in the datastore started to take
much longer time? One Ajax call I have that deletes up to 20 records
often (more often than not perhaps) takes a very long time, where a
single request consumes around 8000cpu_ms and around 8000api_cpu_ms.
The records deleted are simple with only 10 properties of which only
one is a ReferenceProperty and the rest are single-line no-reference
properties. And no complicated indexes are used (as far as I know).
--
Nick Johnson, App Engine Developer Programs Engineer
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration
Number:
368047
--
Nick Johnson, App Engine Developer Programs Engineer
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
368047
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Datastore delete taking long time
I have now changed it to the second variant, with the Query api, and
the result for 3 test calls was:
6775cpu_ms 6687api_cpu_ms
6757cpu_ms 6687api_cpu_ms
6765cpu_ms 6687api_cpu_ms
That's perhaps a slight improvement, but I have also done
vacuum_indexes which could perhaps have improved the performance a
bit. In either case, it's still a lot of CPU time for a single Ajax
call. If these calls are made frequently it will consume vast amounts
of CPU time.
On Jun 24, 12:53 pm, Nick Johnson (Google) nick.john...@google.com
wrote:
Hi Anders,
You don't actually need the entities in order to delete them - only their
keys. Also, you can do all the deletes in a single batch operation. This
ought to be substantially faster:
db.delete(db.GqlQuery(SELECT __key__ FROM Entity WHERE account_key = :1 AND
topic = :2 ORDER BY message_count DESC, account_key, topic).fetch(20))
or equivalently with the Query api:
db.delete(entity.Entity.all(keys_only=True).filter('account_key',
account_key).filter('topic', topic).order('-message_count').fetch(20))
-Nick Johnson
On Wed, Jun 24, 2009 at 11:45 AM, Anders i...@blabline.com wrote:
First I get the entities using a query, something like:
entities = entity.Entity.gql('WHERE account_key = :1 AND topic = :2
ORDER BY message_count DESC LIMIT 20', account_key, topic)
Then a loop calling delete() on each entity:
for entity in entities:
entity.delete()
On Jun 24, 12:29 pm, Nick Johnson (Google) nick.john...@google.com
wrote:
Hi Anders,
How are you deleting the entities? Are you fetching them first, or
deleting
them by providing the keys? And are you deleting them in a single batch,
or
one at a time?
-Nick Johnson
On Wed, Jun 24, 2009 at 7:44 AM, Anders i...@blabline.com wrote:
Has the delete function of records in the datastore started to take
much longer time? One Ajax call I have that deletes up to 20 records
often (more often than not perhaps) takes a very long time, where a
single request consumes around 8000cpu_ms and around 8000api_cpu_ms.
The records deleted are simple with only 10 properties of which only
one is a ReferenceProperty and the rest are single-line no-reference
properties. And no complicated indexes are used (as far as I know).
--
Nick Johnson, App Engine Developer Programs Engineer
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration
Number:
368047
--
Nick Johnson, App Engine Developer Programs Engineer
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
368047
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Datastore delete taking long time
Sometimes delete operations are implemented to be allowed to take more time than put operations. Could it be that the delete operation in GAE has been allowed to be slow in order to make put operations and/or queries faster? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore delete taking long time
Hmm... On a second thought, the put operation is perhaps also costly in terms of CPU time. I haven't tried doing 20 puts in one page access. Maybe that would also consume a lot of CPU time. On a practical level, this is usually not a big problem when the traffic to the application is low, but imagine an application like Twitter being implemented using GAE and with that kind of huge traffic Twitter has. Millions of datastore updates every minute. If an application like that would run on GAE, the Quota costs for CPU time alone would become astronomical! P.S. Twitter is actually using the Amazon Elastic Compute Cloud at the moment I think. ;-) On Jun 24, 6:36 pm, Anders i...@blabline.com wrote: Sometimes delete operations are implemented to be allowed to take more time than put operations. Could it be that the delete operation in GAE has been allowed to be slow in order to make put operations and/or queries faster? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
