[google-appengine] Re: Datastore errors, timeouts, general difficulties
That should read, after deleting, I'm now using 99% of my data storage limit with no additional puts. On Mar 6, 6:00 pm, Jonathan Ultis wrote: > I created a simple page that simply instantiates and stores a model > with about 9 primitive properties. The model is filled with the same > data each time, and that data is less than 100 bytes. I don't have > indices on any of the properties. > > I did some load tests, storing approximately half a million of the > objects at most. Some of the tests were done during the recent latency > problems. Some were done after latency was stabilized. All tests with > significant concurrency (50) generated fairly high timeout and quota > exceeded error rates. So, I got significant log spew. > > My total data size should be around 50 megs of real user data. > > Today, my data usage jumped from about 0.5% of my data storage limit > to 67% of my data storage limit without any intervening tests. That > was surprising since that's about 10x what I really should be using. > But, I figured that data limit might count all replicas or something > odd. And, perhaps there's a latency between the data store and the > data size calculation. So, I wrote a delete all page that deletes 50 > items, then redirects to itself, to delete the next 50. I just delete > the first 50 items returned. I'm not paging or anything. The script > works locally. > > I ran that delete page for a while, periodically getting annoying > timeouts and quota exceeded. I did not run the page in parallel. > > Now that I'm using 99% of my data storage limit. I'm afraid I may have > to buy data storage before I can delete more items. > > Any ideas? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore errors, timeouts, general difficulties
These may be relevant: http://groups.google.com/group/google-appengine/browse_thread/thread/517c7da78454da66/939a67117246d120?lnk=gst&q=log+storage#939a67117246d120 http://code.google.com/p/googleappengine/issues/detail?id=452 Basically you need to get the entity before you delete or it will be gone without reducing your storage quota. On Mar 7, 2:58 am, Jonathan Ultis wrote: > That should read, after deleting, I'm now using 99% of my data storage > limit with no additional puts. > > On Mar 6, 6:00 pm, Jonathan Ultis wrote: > > > I created a simple page that simply instantiates and stores a model > > with about 9 primitive properties. The model is filled with the same > > data each time, and that data is less than 100 bytes. I don't have > > indices on any of the properties. > > > I did some load tests, storing approximately half a million of the > > objects at most. Some of the tests were done during the recent latency > > problems. Some were done after latency was stabilized. All tests with > > significant concurrency (50) generated fairly high timeout and quota > > exceeded error rates. So, I got significant log spew. > > > My total data size should be around 50 megs of real user data. > > > Today, my data usage jumped from about 0.5% of my data storage limit > > to 67% of my data storage limit without any intervening tests. That > > was surprising since that's about 10x what I really should be using. > > But, I figured that data limit might count all replicas or something > > odd. And, perhaps there's a latency between the data store and the > > data size calculation. So, I wrote a delete all page that deletes 50 > > items, then redirects to itself, to delete the next 50. I just delete > > the first 50 items returned. I'm not paging or anything. The script > > works locally. > > > I ran that delete page for a while, periodically getting annoying > > timeouts and quota exceeded. I did not run the page in parallel. > > > Now that I'm using 99% of my data storage limit. I'm afraid I may have > > to buy data storage before I can delete more items. > > > Any ideas? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore errors, timeouts, general difficulties
Hi, My apologies for not closing 452. With the launch of billing, this issue was fixed, we now run a process that correctly accounts for the amount of datastore usage. Deletes may not immediately be reflected, but they will be eventually. So if you did delete all of your data this should be accounted for, minus some delay. What is likely is that you are seeing we now include indexes when accounting for storage usage, which we did not previously do: http://code.google.com/appengine/kb/billing.html#indexes -Marzia On Sat, Mar 7, 2009 at 7:42 AM, Brandon Thomson wrote: > > These may be relevant: > > > http://groups.google.com/group/google-appengine/browse_thread/thread/517c7da78454da66/939a67117246d120?lnk=gst&q=log+storage#939a67117246d120 > > http://code.google.com/p/googleappengine/issues/detail?id=452 > > Basically you need to get the entity before you delete or it will be > gone without reducing your storage quota. > > On Mar 7, 2:58 am, Jonathan Ultis wrote: > > That should read, after deleting, I'm now using 99% of my data storage > > limit with no additional puts. > > > > On Mar 6, 6:00 pm, Jonathan Ultis wrote: > > > > > I created a simple page that simply instantiates and stores a model > > > with about 9 primitive properties. The model is filled with the same > > > data each time, and that data is less than 100 bytes. I don't have > > > indices on any of the properties. > > > > > I did some load tests, storing approximately half a million of the > > > objects at most. Some of the tests were done during the recent latency > > > problems. Some were done after latency was stabilized. All tests with > > > significant concurrency (50) generated fairly high timeout and quota > > > exceeded error rates. So, I got significant log spew. > > > > > My total data size should be around 50 megs of real user data. > > > > > Today, my data usage jumped from about 0.5% of my data storage limit > > > to 67% of my data storage limit without any intervening tests. That > > > was surprising since that's about 10x what I really should be using. > > > But, I figured that data limit might count all replicas or something > > > odd. And, perhaps there's a latency between the data store and the > > > data size calculation. So, I wrote a delete all page that deletes 50 > > > items, then redirects to itself, to delete the next 50. I just delete > > > the first 50 items returned. I'm not paging or anything. The script > > > works locally. > > > > > I ran that delete page for a while, periodically getting annoying > > > timeouts and quota exceeded. I did not run the page in parallel. > > > > > Now that I'm using 99% of my data storage limit. I'm afraid I may have > > > to buy data storage before I can delete more items. > > > > > Any ideas? > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore errors, timeouts, general difficulties
Thanks for the feedback. If indices started to count against quota, that would certainly account for the change in storage size. Although, the page I see at http://code.google.com/appengine/kb/billing.html#indexes still shows that indices do *not* count against the quota. This will end up being an expensive storage solution if indices are billed and there's no way to explicitly exclude properties from the default indices. On Mar 9, 11:36 am, Marzia Niccolai wrote: > Hi, > > My apologies for not closing 452. With the launch of billing, this issue > was fixed, we now run a process that correctly accounts for the amount of > datastore usage. Deletes may not immediately be reflected, but they will be > eventually. So if you did delete all of your data this should be accounted > for, minus some delay. > > What is likely is that you are seeing we now include indexes when accounting > for storage usage, which we did not previously > do:http://code.google.com/appengine/kb/billing.html#indexes > > -Marzia > > On Sat, Mar 7, 2009 at 7:42 AM, Brandon Thomson wrote: > > > These may be relevant: > > >http://groups.google.com/group/google-appengine/browse_thread/thread/... > > >http://code.google.com/p/googleappengine/issues/detail?id=452 > > > Basically you need to get the entity before you delete or it will be > > gone without reducing your storage quota. > > > On Mar 7, 2:58 am, Jonathan Ultis wrote: > > > That should read, after deleting, I'm now using 99% of my data storage > > > limit with no additional puts. > > > > On Mar 6, 6:00 pm, Jonathan Ultis wrote: > > > > > I created a simple page that simply instantiates and stores a model > > > > with about 9 primitive properties. The model is filled with the same > > > > data each time, and that data is less than 100 bytes. I don't have > > > > indices on any of the properties. > > > > > I did some load tests, storing approximately half a million of the > > > > objects at most. Some of the tests were done during the recent latency > > > > problems. Some were done after latency was stabilized. All tests with > > > > significant concurrency (50) generated fairly high timeout and quota > > > > exceeded error rates. So, I got significant log spew. > > > > > My total data size should be around 50 megs of real user data. > > > > > Today, my data usage jumped from about 0.5% of my data storage limit > > > > to 67% of my data storage limit without any intervening tests. That > > > > was surprising since that's about 10x what I really should be using. > > > > But, I figured that data limit might count all replicas or something > > > > odd. And, perhaps there's a latency between the data store and the > > > > data size calculation. So, I wrote a delete all page that deletes 50 > > > > items, then redirects to itself, to delete the next 50. I just delete > > > > the first 50 items returned. I'm not paging or anything. The script > > > > works locally. > > > > > I ran that delete page for a while, periodically getting annoying > > > > timeouts and quota exceeded. I did not run the page in parallel. > > > > > Now that I'm using 99% of my data storage limit. I'm afraid I may have > > > > to buy data storage before I can delete more items. > > > > > Any ideas? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
