Re: [google-appengine] Re: API call datastore_v3.Put() required more quota ?

2011-09-29 Thread djidjadji
If you delete many entities it's faster to use a cursor. An entity is
marked as deleted and will physically be removed in the datastore at a
later time. It takes time to skip the marked entities when executing a
query.
The loop can be modified as such, no need to use count() as it uses
CPU and time also, if the result set is empty there are no entities.

try:
cursor = None
while True:
q = Documents.all(keys_only=True)
if cursor:
   q.with_cursor(cursor)
result = q.fetch(200)
if not result:
   break
cursor = q.cursor()
db.delete(result)
except Exception, e:
self.response.out.write(repr(e)+'\n')
pass

Op 16 september 2011 18:24 heeft sofia sofiacard...@gmail.com het
volgende geschreven:
 I have the same problem.. Datastore quota at 100% and unable to delete an
 entity either through admin or using map/reduce. What I did is set up a
 script to delete x records at a time. I've managed to decrease data by 45%
 in 2 days but I then hit cpu quota so I'm guessing it's gonna take a few
 more days until I'm able to delete all the data. What i did was set up this
 script, bulkdelete.py:
 #!/usr/bin/env python
 # -*- coding: UTF-8 -*-
 #
 #
 import time
 from google.appengine.ext import webapp
 from google.appengine.ext.webapp.util import run_wsgi_app
 from google.appengine.ext import db
 from lib.model import Documents
 class BulkDelete(webapp.RequestHandler):
     def get(self):
         self.response.headers['Content-Type'] = 'text/plain'
         mod = self.request.get('m')
         if not mod:
             exit
         try:
             while True:
                 q = db.GqlQuery(SELECT __key__ FROM Documents ORDER BY date
 ASC)
                 assert q.count()
                 db.delete(q.fetch(200))
                 time.sleep(0.5)
         except Exception, e:
             self.response.out.write(repr(e)+'\n')
             pass
 # init
 application = webapp.WSGIApplication([('/bulkdelete',
 BulkDelete)],debug=True)
 def main():
         run_wsgi_app(application)
 if __name__ == '__main__':
         main()
 and then call it each 5 min through cron.yaml.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.



RE: [google-appengine] Re: API call datastore_v3.Put() required more quota ?

2011-09-24 Thread Brandon Wirtz
Sleep in non-GAE Environments says Do other things and check back on me in
X time 

With instance Hour billing Sleep would only save you money if you have
concurrency to handle multiple tasks, and this freed CPU Cycles, AND you
weren't sleeping longer than the thing you were waiting to have happen.

 

From: google-appengine@googlegroups.com
[mailto:google-appengine@googlegroups.com] On Behalf Of Gerald Tan
Sent: Friday, September 16, 2011 9:51 AM
To: google-appengine@googlegroups.com
Subject: [google-appengine] Re: API call datastore_v3.Put() required more
quota ?

 

From my experience (at least with Java) I believe sleep() continues to eat
cpu time despite the cpu not really doing anything. My cpu times skyrocketed
when I used sleep() with long-held http connections to implement http push
back before channel api was available. I recommend breaking the work into
taskqueue tasks instead.

-- 
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/vJC56brjHYwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to 
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/google-appengine?hl=en.