[google-appengine] Data Storage Size - multiplier per object
I created a model with fixed content that requires ~250b serialized, including all field names, the key, and the kind name, and parent (None). I added 312000 of those to the datastore, for 75 megs of raw data. There are 8 indexable fields, The indices should require no more than 176 megs of additional space, if the indices don't do any sort of column compression. That's 250 megs of raw space. But, the data store reports 1GB of space used. That suggests perhaps 2x redundancy, plus a 50% fill rate in big table. Or, maybe just 4x redundancy. No idea. Anyhow, for now, take your raw object size including kind, key, field names, and field content, and multiply by 10x-15x, depending on how many indexable properties you have, to get your final storage size. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Data Storage Size - multiplier per object
Or better, take your raw data size excluding BlobProperty and TextProperty and multiply by 15x. I'm not sure what the multiplier is on the unindexed properties yet. On Mar 10, 9:28 am, Jonathan Ultis jonathan.ul...@gmail.com wrote: I created a model with fixed content that requires ~250b serialized, including all field names, the key, and the kind name, and parent (None). I added 312000 of those to the datastore, for 75 megs of raw data. There are 8 indexable fields, The indices should require no more than 176 megs of additional space, if the indices don't do any sort of column compression. That's 250 megs of raw space. But, the data store reports 1GB of space used. That suggests perhaps 2x redundancy, plus a 50% fill rate in big table. Or, maybe just 4x redundancy. No idea. Anyhow, for now, take your raw object size including kind, key, field names, and field content, and multiply by 10x-15x, depending on how many indexable properties you have, to get your final storage size. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Unbelieveable jump in stored data.
According to this message, google may be counting index sizes in the quota now. http://groups.google.com/group/google-appengine/msg/a16edddcc6db424f On Mar 9, 8:25 pm, Josh Cronemeyer joshuacroneme...@gmail.com wrote: This afternoon I had less than 1GB stored data, which took me several months to accumulate. This evening the dashboard is telling me I have used nearly 3GB stored data! It doesn't look like my traffic levels or number of users have changed significantly. This has to be wrong. Could it be related to the recent hardware changes mentioned on the app engine system status page? Has anyone else noticed anything with stored data reporting? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Datastore errors, timeouts, general difficulties
I created a simple page that simply instantiates and stores a model with about 9 primitive properties. The model is filled with the same data each time, and that data is less than 100 bytes. I don't have indices on any of the properties. I did some load tests, storing approximately half a million of the objects at most. Some of the tests were done during the recent latency problems. Some were done after latency was stabilized. All tests with significant concurrency (50) generated fairly high timeout and quota exceeded error rates. So, I got significant log spew. My total data size should be around 50 megs of real user data. Today, my data usage jumped from about 0.5% of my data storage limit to 67% of my data storage limit without any intervening tests. That was surprising since that's about 10x what I really should be using. But, I figured that data limit might count all replicas or something odd. And, perhaps there's a latency between the data store and the data size calculation. So, I wrote a delete all page that deletes 50 items, then redirects to itself, to delete the next 50. I just delete the first 50 items returned. I'm not paging or anything. The script works locally. I ran that delete page for a while, periodically getting annoying timeouts and quota exceeded. I did not run the page in parallel. Now that I'm using 99% of my data storage limit. I'm afraid I may have to buy data storage before I can delete more items. Any ideas? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore errors, timeouts, general difficulties
That should read, after deleting, I'm now using 99% of my data storage limit with no additional puts. On Mar 6, 6:00 pm, Jonathan Ultis jonathan.ul...@gmail.com wrote: I created a simple page that simply instantiates and stores a model with about 9 primitive properties. The model is filled with the same data each time, and that data is less than 100 bytes. I don't have indices on any of the properties. I did some load tests, storing approximately half a million of the objects at most. Some of the tests were done during the recent latency problems. Some were done after latency was stabilized. All tests with significant concurrency (50) generated fairly high timeout and quota exceeded error rates. So, I got significant log spew. My total data size should be around 50 megs of real user data. Today, my data usage jumped from about 0.5% of my data storage limit to 67% of my data storage limit without any intervening tests. That was surprising since that's about 10x what I really should be using. But, I figured that data limit might count all replicas or something odd. And, perhaps there's a latency between the data store and the data size calculation. So, I wrote a delete all page that deletes 50 items, then redirects to itself, to delete the next 50. I just delete the first 50 items returned. I'm not paging or anything. The script works locally. I ran that delete page for a while, periodically getting annoying timeouts and quota exceeded. I did not run the page in parallel. Now that I'm using 99% of my data storage limit. I'm afraid I may have to buy data storage before I can delete more items. Any ideas? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Django 1.0 + app-engine-patch
It looks like you really are missing the PIL module. syncdb isn't necessary with app-engine-patch. But, $ python manage.py shell will probably fail with the same error until PIL gets installed. Which platform are you developing on? On Jan 30, 10:44 am, metametagirl metametag...@gmail.com wrote: I'm trying to start a new app using Django 1.0 with the app-engine- patch. I'm used to using MySQL but I'm new to this datastore. The Django tutorial says to run 'python manage.py syncdb' to create the tables, but this doesn't work for me in my development environment. Is this step necessary when using this combination of technologies (Django 1.0 + app-engine-patch)? Also, how do you invoke the Django admin in the development environment? My understanding is that it won't work once you upload the app to GAE, but that there should be a way to make it work in the dev environment. python manage.py syncdb WARNING:root:Could not read datastore data from /var/folders/bz/ bzgZyG4mE5adBq+imZYFdk+++TI/-Tmp-/django_myapp.datastore WARNING:root:Could not initialize images API; you are likely missing the Python PIL module. ImportError: No module named PIL Unknown command: 'syncdb' Type 'manage.py help' for usage. Thanks in advance, Kelly --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
