Re: [google-appengine] Re: API call datastore_v3.Put() required more quota ?
If you delete many entities it's faster to use a cursor. An entity is
marked as deleted and will physically be removed in the datastore at a
later time. It takes time to skip the marked entities when executing a
query.
The loop can be modified as such, no need to use count() as it uses
CPU and time also, if the result set is empty there are no entities.
try:
cursor = None
while True:
q = Documents.all(keys_only=True)
if cursor:
q.with_cursor(cursor)
result = q.fetch(200)
if not result:
break
cursor = q.cursor()
db.delete(result)
except Exception, e:
self.response.out.write(repr(e)+'\n')
pass
Op 16 september 2011 18:24 heeft sofia sofiacard...@gmail.com het
volgende geschreven:
I have the same problem.. Datastore quota at 100% and unable to delete an
entity either through admin or using map/reduce. What I did is set up a
script to delete x records at a time. I've managed to decrease data by 45%
in 2 days but I then hit cpu quota so I'm guessing it's gonna take a few
more days until I'm able to delete all the data. What i did was set up this
script, bulkdelete.py:
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
#
#
import time
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.ext import db
from lib.model import Documents
class BulkDelete(webapp.RequestHandler):
def get(self):
self.response.headers['Content-Type'] = 'text/plain'
mod = self.request.get('m')
if not mod:
exit
try:
while True:
q = db.GqlQuery(SELECT __key__ FROM Documents ORDER BY date
ASC)
assert q.count()
db.delete(q.fetch(200))
time.sleep(0.5)
except Exception, e:
self.response.out.write(repr(e)+'\n')
pass
# init
application = webapp.WSGIApplication([('/bulkdelete',
BulkDelete)],debug=True)
def main():
run_wsgi_app(application)
if __name__ == '__main__':
main()
and then call it each 5 min through cron.yaml.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: API call datastore_v3.Put() required more quota ?
Any official engineer present on this group ? On Sep 14, 7:21 am, @zghanv/- azgha...@gmail.com wrote: I'm trying to delete testing data from my app from datastore admin ... but it's giving following error. My High Replication Data is 100%, now i want to reset all data. How can i remove that. Thanks. --- Delete Job Status There was a problem kicking off the jobs. The error was: The API call datastore_v3.Put() required more quota than is available. --- Dashboard status ... CPU Time 30% 1.95 of 6.50 CPU hours Outgoing Bandwidth 2% 0.02 of 1.00 GBytes Incoming Bandwidth 0% 0.00 of 1.00 GBytes Total Stored Data 0% 0.00 of 1.00 GBytes Recipients Emailed 0% 0 of 2,000 High Replication Data 100% 0.50 of 0.50 GBytes This resource is currently experiencing a short-term quota limit. Backend Usage 0% $0.00 of $0.72 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: API call datastore_v3.Put() required more quota ?
Sleep in non-GAE Environments says Do other things and check back on me in X time With instance Hour billing Sleep would only save you money if you have concurrency to handle multiple tasks, and this freed CPU Cycles, AND you weren't sleeping longer than the thing you were waiting to have happen. From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Gerald Tan Sent: Friday, September 16, 2011 9:51 AM To: google-appengine@googlegroups.com Subject: [google-appengine] Re: API call datastore_v3.Put() required more quota ? From my experience (at least with Java) I believe sleep() continues to eat cpu time despite the cpu not really doing anything. My cpu times skyrocketed when I used sleep() with long-held http connections to implement http push back before channel api was available. I recommend breaking the work into taskqueue tasks instead. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/vJC56brjHYwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: API call datastore_v3.Put() required more quota ?
I have the same problem.. Datastore quota at 100% and unable to delete an
entity either through admin or using map/reduce. What I did is set up a
script to delete x records at a time. I've managed to decrease data by 45%
in 2 days but I then hit cpu quota so I'm guessing it's gonna take a few
more days until I'm able to delete all the data. What i did was set up this
script, bulkdelete.py:
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
#
#
import time
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.ext import db
from lib.model import Documents
class BulkDelete(webapp.RequestHandler):
def get(self):
self.response.headers['Content-Type'] = 'text/plain'
mod = self.request.get('m')
if not mod:
exit
try:
while True:
q = db.GqlQuery(SELECT __key__ FROM Documents ORDER BY date
ASC)
assert q.count()
db.delete(q.fetch(200))
time.sleep(0.5)
except Exception, e:
self.response.out.write(repr(e)+'\n')
pass
# init
application = webapp.WSGIApplication([('/bulkdelete',
BulkDelete)],debug=True)
def main():
run_wsgi_app(application)
if __name__ == '__main__':
main()
and then call it each 5 min through cron.yaml.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/0fgWMvYsVdsJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: API call datastore_v3.Put() required more quota ?
From my experience (at least with Java) I believe sleep() continues to eat cpu time despite the cpu not really doing anything. My cpu times skyrocketed when I used sleep() with long-held http connections to implement http push back before channel api was available. I recommend breaking the work into taskqueue tasks instead. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/vJC56brjHYwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: API call datastore_v3.Put() required more quota ?
any expert ? On Sep 14, 7:21 am, @zghanv/- azgha...@gmail.com wrote: I'm trying to delete testing data from my app from datastore admin ... but it's giving following error. My High Replication Data is 100%, now i want to reset all data. How can i remove that. Thanks. --- Delete Job Status There was a problem kicking off the jobs. The error was: The API call datastore_v3.Put() required more quota than is available. --- Dashboard status ... CPU Time 30% 1.95 of 6.50 CPU hours Outgoing Bandwidth 2% 0.02 of 1.00 GBytes Incoming Bandwidth 0% 0.00 of 1.00 GBytes Total Stored Data 0% 0.00 of 1.00 GBytes Recipients Emailed 0% 0 of 2,000 High Replication Data 100% 0.50 of 0.50 GBytes This resource is currently experiencing a short-term quota limit. Backend Usage 0% $0.00 of $0.72 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
