[google-appengine] Re: Quota exceeded... although it doesn't seem to be
Hi Shrinand, The bulk uploader lets you set a rate at which you want to upload your data - see the command line reference in the documentation: http://code.google.com/appengine/docs/python/tools/uploadingdata.html#Command_Line_Arguments -Nick Johnson On Wed, Sep 2, 2009 at 5:45 PM, Shrinand Javadekar shrin...@gmail.comwrote: You need to ratelimit the upload to a slower speed in order to not exhaust your short-term quotas. -Nick Johnson Thanks a ton Nick. I do delete the data in batches. The problem is not so much with delete. I can always restart the deletion process and clear out all the data. The problem is with uploading. I tried uploading in batches of 100 (lines in the csv file). Then with 50. None of them worked. How do I rate-limit the upload to a slower speed? Also, I think the appcfg.py with the upload data option tackles this problem of restarting the upload process from wherever it crashes last. I am getting some errors in that too. Can you help me with that? http://groups.google.com/group/google-appengine/browse_thread/thread/795f1030500440f0# Thanks in advance. -Shri -- Nick Johnson, Developer Programs Engineer, App Engine --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Quota exceeded... although it doesn't seem to be
Hi ShriJ, On Wed, Sep 2, 2009 at 2:44 AM, ShriJ shrin...@gmail.com wrote: Hi, I've been having a very hard time since the last few days w.r.t. deleting existing data and uploading new data in my datastore. For deleting: I get a row using the key_name and then delete it. dbRow = modelObject.get_by_key_name(key_name) if dbRow: dbRow.delete() My datastore has roughly 60K rows and I delete them one by one. Many times when doing this, I get a quota exceeded exception. But when I login to appspot.com for seeing whats wrong, I don't see any error. It's possible to run out of short-term quota without exhausting your daily quota. This limit exists in order to prevent a burst of traffic taking your site offline for the rest of the day. Is there a reason you're not deleting records in batches? This will be much easier on your quotas. Similarly, when uploading data, I am using the bulkload_client.py script for uploading the data. There again I see the same issues. There are times when after inserting 500 rows, I get the error and thereafter no matter how many times I start again, I see the same error. You need to ratelimit the upload to a slower speed in order to not exhaust your short-term quotas. -Nick Johnson I've almost come to the conclusion that downloading and uploading the data on the same day is kinda impossible. Am I doing something fundamentally incorrect? If not, is there any ETA on when will things get better. My website has been down since the last 4 days because of these issues :(. -Shri -- Nick Johnson, Developer Programs Engineer, App Engine --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Quota exceeded... although it doesn't seem to be
You need to ratelimit the upload to a slower speed in order to not exhaust your short-term quotas. -Nick Johnson Thanks a ton Nick. I do delete the data in batches. The problem is not so much with delete. I can always restart the deletion process and clear out all the data. The problem is with uploading. I tried uploading in batches of 100 (lines in the csv file). Then with 50. None of them worked. How do I rate-limit the upload to a slower speed? Also, I think the appcfg.py with the upload data option tackles this problem of restarting the upload process from wherever it crashes last. I am getting some errors in that too. Can you help me with that? http://groups.google.com/group/google-appengine/browse_thread/thread/795f1030500440f0# Thanks in advance. -Shri --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
