[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
I have also been puzzled at times on where the space is going. I filed this request today: "More granular accounting of how datastore space is used" http://code.google.com/p/googleappengine/issues/detail?id=1396 Please browse to the issue and add your vote/star if you want to see this feature implemented. Panos --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
On Apr 23, 4:47 am, Panos wrote: > I have also been puzzled at times on where the space is going. I filed > this request today: > > "More granular accounting of how datastore space is > used"http://code.google.com/p/googleappengine/issues/detail?id=1396 > > Please browse to the issue and add your vote/star if you want to see > this feature implemented. > > Panos I also think there is something wrong. I have 2.3M Domain records and the source CSV is only 63 megabytes, no composite index. The dashboard claims I am using 3GB !?! (3.03 of 101.00 GBytes) This is my base expando model: class Domain(db.Expando): name = db.StringProperty(required=True, verbose_name='FQDN') revname = db.StringProperty(verbose_name='Reverse FQDN') since = db.DateTimeProperty(auto_now_add=True) I am ready to upload 102M more records, I guess I am going to wait until this issue is resolved. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
Can you both provide your application IDs so I can investigate a bit? Thanks, - Jason On Sat, Apr 25, 2009 at 12:33 PM, Kugutsumen wrote: > > > > On Apr 23, 4:47 am, Panos wrote: > > I have also been puzzled at times on where the space is going. I filed > > this request today: > > > > "More granular accounting of how datastore space is used" > http://code.google.com/p/googleappengine/issues/detail?id=1396 > > > > Please browse to the issue and add your vote/star if you want to see > > this feature implemented. > > > > Panos > > I also think there is something wrong. > > I have 2.3M Domain records and the source CSV is only 63 megabytes, > no composite index. The dashboard claims I am using 3GB !?! > (3.03 of 101.00 GBytes) > > This is my base expando model: > > class Domain(db.Expando): > name = db.StringProperty(required=True, verbose_name='FQDN') > revname = db.StringProperty(verbose_name='Reverse FQDN') > since = db.DateTimeProperty(auto_now_add=True) > > I am ready to upload 102M more records, I guess I am going to wait > until this issue is resolved. > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
I've sent you my ID. Thanks for looking into this. On Apr 29, 4:06 am, "Jason (Google)" wrote: > Can you both provide your application IDs so I can investigate a bit? > > Thanks, > - Jason > > On Sat, Apr 25, 2009 at 12:33 PM, Kugutsumen wrote: > > > On Apr 23, 4:47 am, Panos wrote: > > > I have also been puzzled at times on where the space is going. I filed > > > this request today: > > > > "More granular accounting of how datastore space is used" > >http://code.google.com/p/googleappengine/issues/detail?id=1396 > > > > Please browse to the issue and add your vote/star if you want to see > > > this feature implemented. > > > > Panos > > > I also think there is something wrong. > > > I have 2.3M Domain records and the source CSV is only 63 megabytes, > > no composite index. The dashboard claims I am using 3GB !?! > > (3.03 of 101.00 GBytes) > > > This is my base expando model: > > > class Domain(db.Expando): > > name = db.StringProperty(required=True, verbose_name='FQDN') > > revname = db.StringProperty(verbose_name='Reverse FQDN') > > since = db.DateTimeProperty(auto_now_add=True) > > > I am ready to upload 102M more records, I guess I am going to wait > > until this issue is resolved. > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
I've created the following issue: http://code.google.com/p/googleappengine/issues/detail?id=1436 On Apr 29, 4:06 am, "Jason (Google)" wrote: > Can you both provide your application IDs so I can investigate a bit? > > Thanks, > - Jason > > On Sat, Apr 25, 2009 at 12:33 PM, Kugutsumen wrote: > > > On Apr 23, 4:47 am, Panos wrote: > > > I have also been puzzled at times on where the space is going. I filed > > > this request today: > > > > "More granular accounting of how datastore space is used" > >http://code.google.com/p/googleappengine/issues/detail?id=1396 > > > > Please browse to the issue and add your vote/star if you want to see > > > this feature implemented. > > > > Panos > > > I also think there is something wrong. > > > I have 2.3M Domain records and the source CSV is only 63 megabytes, > > no composite index. The dashboard claims I am using 3GB !?! > > (3.03 of 101.00 GBytes) > > > This is my base expando model: > > > class Domain(db.Expando): > > name = db.StringProperty(required=True, verbose_name='FQDN') > > revname = db.StringProperty(verbose_name='Reverse FQDN') > > since = db.DateTimeProperty(auto_now_add=True) > > > I am ready to upload 102M more records, I guess I am going to wait > > until this issue is resolved. > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
Two weeks ago, I've sent my applications ID to both you and Nick and I haven't heard from you since then. Thanks --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
Hi Anthony. I'm very sorry for the late reply, and thank you for bearing with me. I've discussed this with the datastore team and it's evident that the CSV file's size is not a great indicator of how much storage your entities will consume. On top of the size of the raw data, each entity has associated metadata, as you've already mentioned, but I'd bet that the indexes are consuming the greatest space. If you don't ever query on one or more of these 15 string properties, you may consider changing their property types to Text or declaring indexed=false in your model. If you can do this with one of your properties and re-build your indexes, I'd be interested in seeing how much your storage usage decreases since you'll need one less index. (Note that single-property indexes are present but not listed in the Admin Console.) - Jason On Sat, May 9, 2009 at 4:34 PM, Kugutsumen wrote: > > Two weeks ago, I've sent my applications ID to both you and Nick and I > haven't heard from you since then. > > Thanks > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
Since index space can be significant, can we get some additional information? For example, does an indexed db.ListProperty(db.Key) with three elements take significantly more or less space than an indexed db.StringListProperty with three elements whose value is str() of the same keys? (The pickle of keys seems to be significantly larger than the pickle of the equivalent strings.) On May 11, 5:04 pm, "Jason (Google)" wrote: > Hi Anthony. I'm very sorry for the late reply, and thank you for bearing > with me. I've discussed this with the datastore team and it's evident that > the CSV file's size is not a great indicator of how much storage your > entities will consume. On top of the size of the raw data, each entity has > associated metadata, as you've already mentioned, but I'd bet that the > indexes are consuming the greatest space. If you don't ever query on one or > more of these 15 string properties, you may consider changing their property > types to Text or declaring indexed=false in your model. If you can do this > with one of your properties and re-build your indexes, I'd be interested in > seeing how much your storage usage decreases since you'll need one less > index. > > (Note that single-property indexes are present but not listed in the Admin > Console.) > > - Jason > > > > On Sat, May 9, 2009 at 4:34 PM, Kugutsumen wrote: > > > Two weeks ago, I've sent my applications ID to both you and Nick and I > > haven't heard from you since then. > > > Thanks- Hide quoted text - > > - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
I just did a test on one of my apps. Nearly all my data is in a single model. I have 163189 instances, and the total size (calculated by reading each instance and running to_xml() on it, and then adding up the results) is 281,145,536 bytes. Most of my properties have indexed=False. The dashboard reports using 890MB of data. I don't know whether the dashboard calculation is wrong, whether I should be using a different calculation to estimate my record size, or something else. If my indexes really are consuming 600MB, then I would work on redoing a chunk of the app to fix that problem. However, the only course at the moment appears to be to delete old data, and hope that the data consumption goes down. Not really very satisfactory. Philip On May 12, 1:38 am, Andy Freeman wrote: > Since index space can be significant, can we get some additional > information? > > For example, does an indexed db.ListProperty(db.Key) with three > elements take significantly more or less space than an indexed > db.StringListProperty with three elements whose value is str() of the > same keys? (The pickle of keys seems to be significantly larger than > the pickle of the equivalent strings.) > > On May 11, 5:04 pm, "Jason (Google)" wrote: > > > Hi Anthony. I'm very sorry for the late reply, and thank you for bearing > > with me. I've discussed this with thedatastoreteam and it's evident that > > the CSV file's size is not a great indicator of how much storage your > > entities will consume. On top of the size of the raw data, each entity has > > associated metadata, as you've already mentioned, but I'd bet that the > > indexes are consuming the greatest space. If you don't ever query on one or > > more of these 15 string properties, you may consider changing their property > > types to Text or declaring indexed=false in your model. If you can do this > > with one of your properties and re-build your indexes, I'd be interested in > > seeing how much your storageusagedecreases since you'll need one less > > index. > > > (Note that single-property indexes are present but not listed in the Admin > > Console.) > > > - Jason > > > On Sat, May 9, 2009 at 4:34 PM, Kugutsumen wrote: > > > > Two weeks ago, I've sent my applications ID to both you and Nick and I > > > haven't heard from you since then. > > > > Thanks- Hide quoted text - > > > - Show quoted text - > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
Hi Andy. In this case, the list of Key objects will be smaller than the list of key strings. Even though the picked db.Key object is larger, it is a binary-encoded protocol buffer form that gets stored, which is smaller than the pickled string. That said, I doubt it would make a tremendous difference unless you have a lot of these entities or these lists have a lot of values. - Jason On Mon, May 11, 2009 at 10:38 PM, Andy Freeman wrote: > > Since index space can be significant, can we get some additional > information? > > For example, does an indexed db.ListProperty(db.Key) with three > elements take significantly more or less space than an indexed > db.StringListProperty with three elements whose value is str() of the > same keys? (The pickle of keys seems to be significantly larger than > the pickle of the equivalent strings.) > > On May 11, 5:04 pm, "Jason (Google)" wrote: > > Hi Anthony. I'm very sorry for the late reply, and thank you for bearing > > with me. I've discussed this with the datastore team and it's evident > that > > the CSV file's size is not a great indicator of how much storage your > > entities will consume. On top of the size of the raw data, each entity > has > > associated metadata, as you've already mentioned, but I'd bet that the > > indexes are consuming the greatest space. If you don't ever query on one > or > > more of these 15 string properties, you may consider changing their > property > > types to Text or declaring indexed=false in your model. If you can do > this > > with one of your properties and re-build your indexes, I'd be interested > in > > seeing how much your storage usage decreases since you'll need one less > > index. > > > > (Note that single-property indexes are present but not listed in the > Admin > > Console.) > > > > - Jason > > > > > > > > On Sat, May 9, 2009 at 4:34 PM, Kugutsumen wrote: > > > > > Two weeks ago, I've sent my applications ID to both you and Nick and I > > > haven't heard from you since then. > > > > > Thanks- Hide quoted text - > > > > - Show quoted text - > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
Hi Philip. Calling to_xml() is not a great indicator of the size of your entity as stored in BigTable. Unfortunately, there is currently no straightforward way to estimate how large your entities are, although we're working on possible solutions to this problem. Without knowing your data model or index definitions, it's certainly not impossible to rule out the size of your indexes, particularly if your application is querying across mutliple multi-valued properties, although this isn't the only scenario that could lead to huge indexes. If you have a property that you're never querying against, I recommend you try removing this single property index and see if that makes a noticeable impact or see if you can eliminate any of your custom indexes which you don't use too often. - Jason On Tue, May 12, 2009 at 8:32 PM, WeatherPhilip < philip-goo...@gladstonefamily.net> wrote: > > I just did a test on one of my apps. Nearly all my data is in a single > model. > > I have 163189 instances, and the total size (calculated by reading > each instance and running to_xml() on it, and then adding up the > results) is 281,145,536 bytes. Most of my properties have > indexed=False. The dashboard reports using 890MB of data. I don't know > whether the dashboard calculation is wrong, whether I should be using > a different calculation to estimate my record size, or something else. > If my indexes really are consuming 600MB, then I would work on redoing > a chunk of the app to fix that problem. > > However, the only course at the moment appears to be to delete old > data, and hope that the data consumption goes down. Not really very > satisfactory. > > Philip > > On May 12, 1:38 am, Andy Freeman wrote: > > Since index space can be significant, can we get some additional > > information? > > > > For example, does an indexed db.ListProperty(db.Key) with three > > elements take significantly more or less space than an indexed > > db.StringListProperty with three elements whose value is str() of the > > same keys? (The pickle of keys seems to be significantly larger than > > the pickle of the equivalent strings.) > > > > On May 11, 5:04 pm, "Jason (Google)" wrote: > > > > > Hi Anthony. I'm very sorry for the late reply, and thank you for > bearing > > > with me. I've discussed this with thedatastoreteam and it's evident > that > > > the CSV file's size is not a great indicator of how much storage your > > > entities will consume. On top of the size of the raw data, each entity > has > > > associated metadata, as you've already mentioned, but I'd bet that the > > > indexes are consuming the greatest space. If you don't ever query on > one or > > > more of these 15 string properties, you may consider changing their > property > > > types to Text or declaring indexed=false in your model. If you can do > this > > > with one of your properties and re-build your indexes, I'd be > interested in > > > seeing how much your storageusagedecreases since you'll need one less > > > index. > > > > > (Note that single-property indexes are present but not listed in the > Admin > > > Console.) > > > > > - Jason > > > > > On Sat, May 9, 2009 at 4:34 PM, Kugutsumen > wrote: > > > > > > Two weeks ago, I've sent my applications ID to both you and Nick and > I > > > > haven't heard from you since then. > > > > > > Thanks- Hide quoted text - > > > > > - Show quoted text - > > > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
Jason I removed a bunch of single property indexes (by setting indexed=False and then loaded and stored each item. This didn't save much (a few percent). Also, the fact that I can't see the single property indexes makes it more tricky to figure out if they have really gone or not! I'm now deleting 10% of the records, but I've only reclaimed 2-3% of the space (0.92GB down to 0.90GB). I don't have any significant use of multi-value fields (there is one field, but only rarely does it have more than one (2) values). Philip On May 13, 2:51 pm, "Jason (Google)" wrote: > Hi Philip. Calling to_xml() is not a great indicator of the size of your > entity as stored in BigTable. Unfortunately, there is currently no > straightforward way to estimate how large your entities are, although we're > working on possible solutions to this problem. > > Without knowing your data model or index definitions, it's certainly not > impossible to rule out the size of your indexes, particularly if your > application is querying across mutliple multi-valued properties, although > this isn't the only scenario that could lead to huge indexes. If you have a > property that you're never querying against, I recommend you try removing > this single property index and see if that makes a noticeable impact or see > if you can eliminate any of your custom indexes which you don't use too > often. > > - Jason > > On Tue, May 12, 2009 at 8:32 PM, WeatherPhilip < > > philip-goo...@gladstonefamily.net> wrote: > > > I just did a test on one of my apps. Nearly all my data is in a single > > model. > > > I have 163189 instances, and the total size (calculated by reading > > each instance and running to_xml() on it, and then adding up the > > results) is 281,145,536 bytes. Most of my properties have > > indexed=False. The dashboard reports using 890MB of data. I don't know > > whether the dashboard calculation is wrong, whether I should be using > > a different calculation to estimate my record size, or something else. > > If my indexes really are consuming 600MB, then I would work on redoing > > a chunk of the app to fix that problem. > > > However, the only course at the moment appears to be to delete old > > data, and hope that the data consumption goes down. Not really very > > satisfactory. > > > Philip > > > On May 12, 1:38 am, Andy Freeman wrote: > > > Since index space can be significant, can we get some additional > > > information? > > > > For example, does an indexed db.ListProperty(db.Key) with three > > > elements take significantly more or less space than an indexed > > > db.StringListProperty with three elements whose value is str() of the > > > same keys? (The pickle of keys seems to be significantly larger than > > > the pickle of the equivalent strings.) > > > > On May 11, 5:04 pm, "Jason (Google)" wrote: > > > > > Hi Anthony. I'm very sorry for the late reply, and thank you for > > bearing > > > > with me. I've discussed this with thedatastoreteam and it's evident > > that > > > > the CSV file's size is not a great indicator of how much storage your > > > > entities will consume. On top of the size of the raw data, each entity > > has > > > > associated metadata, as you've already mentioned, but I'd bet that the > > > > indexes are consuming the greatest space. If you don't ever query on > > one or > > > > more of these 15 string properties, you may consider changing their > > property > > > > types to Text or declaring indexed=false in your model. If you can do > > this > > > > with one of your properties and re-build your indexes, I'd be > > interested in > > > > seeing how much your storageusagedecreases since you'll need one less > > > > index. > > > > > (Note that single-property indexes are present but not listed in the > > Admin > > > > Console.) > > > > > - Jason > > > > > On Sat, May 9, 2009 at 4:34 PM, Kugutsumen > > wrote: > > > > > > Two weeks ago, I've sent my applications ID to both you and Nick and > > I > > > > > haven't heard from you since then. > > > > > > Thanks- Hide quoted text - > > > > > - Show quoted text - > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
Argh! This means that one form (db.Key) is smaller than the other (comparable string) for the datastore while the reverse is true for memcache. How about defining a __getstate__ and __setstate__ for db.Key that is smaller than the string equivalent? This will help for memcaching any db.Model instance whose .key() is defined. On May 13, 11:41 am, "Jason (Google)" wrote: > Hi Andy. In this case, the list of Key objects will be smaller than the list > of key strings. Even though the picked db.Key object is larger, it is a > binary-encoded protocol buffer form that gets stored, which is smaller than > the pickled string. That said, I doubt it would make a tremendous difference > unless you have a lot of these entities or these lists have a lot of values. > > - Jason > > > > On Mon, May 11, 2009 at 10:38 PM, Andy Freeman wrote: > > > Since index space can be significant, can we get some additional > > information? > > > For example, does an indexed db.ListProperty(db.Key) with three > > elements take significantly more or less space than an indexed > > db.StringListProperty with three elements whose value is str() of the > > same keys? (The pickle of keys seems to be significantly larger than > > the pickle of the equivalent strings.) > > > On May 11, 5:04 pm, "Jason (Google)" wrote: > > > Hi Anthony. I'm very sorry for the late reply, and thank you for bearing > > > with me. I've discussed this with the datastore team and it's evident > > that > > > the CSV file's size is not a great indicator of how much storage your > > > entities will consume. On top of the size of the raw data, each entity > > has > > > associated metadata, as you've already mentioned, but I'd bet that the > > > indexes are consuming the greatest space. If you don't ever query on one > > or > > > more of these 15 string properties, you may consider changing their > > property > > > types to Text or declaring indexed=false in your model. If you can do > > this > > > with one of your properties and re-build your indexes, I'd be interested > > in > > > seeing how much your storage usage decreases since you'll need one less > > > index. > > > > (Note that single-property indexes are present but not listed in the > > Admin > > > Console.) > > > > - Jason > > > > On Sat, May 9, 2009 at 4:34 PM, Kugutsumen wrote: > > > > > Two weeks ago, I've sent my applications ID to both you and Nick and I > > > > haven't heard from you since then. > > > > > Thanks- Hide quoted text - > > > > - Show quoted text -- Hide quoted text - > > - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
Argh! This means that one form (db.Key) is smaller than the other (comparable string) for the datastore while the reverse is true for memcache. I've created am issue ( http://code.google.com/p/googleappengine/issues/detail?id=1538 )requesting a __getstate__ and __setstate__ for db.Key that is smaller than the string equivalent. In addition to eliminating the inconsistency betwen the datastore and memcache sizes, it will reduce the size of every memcache'd db.Model instance whose .key() is defined. On May 13, 11:41 am, "Jason (Google)" wrote: > Hi Andy. In this case, the list of Key objects will be smaller than the list > of key strings. Even though the picked db.Key object is larger, it is a > binary-encoded protocol buffer form that gets stored, which is smaller than > the pickled string. That said, I doubt it would make a tremendous difference > unless you have a lot of these entities or these lists have a lot of values. > > - Jason > > > > On Mon, May 11, 2009 at 10:38 PM, Andy Freeman wrote: > > > Since index space can be significant, can we get some additional > > information? > > > For example, does an indexed db.ListProperty(db.Key) with three > > elements take significantly more or less space than an indexed > > db.StringListProperty with three elements whose value is str() of the > > same keys? (The pickle of keys seems to be significantly larger than > > the pickle of the equivalent strings.) > > > On May 11, 5:04 pm, "Jason (Google)" wrote: > > > Hi Anthony. I'm very sorry for the late reply, and thank you for bearing > > > with me. I've discussed this with the datastore team and it's evident > > that > > > the CSV file's size is not a great indicator of how much storage your > > > entities will consume. On top of the size of the raw data, each entity > > has > > > associated metadata, as you've already mentioned, but I'd bet that the > > > indexes are consuming the greatest space. If you don't ever query on one > > or > > > more of these 15 string properties, you may consider changing their > > property > > > types to Text or declaring indexed=false in your model. If you can do > > this > > > with one of your properties and re-build your indexes, I'd be interested > > in > > > seeing how much your storage usage decreases since you'll need one less > > > index. > > > > (Note that single-property indexes are present but not listed in the > > Admin > > > Console.) > > > > - Jason > > > > On Sat, May 9, 2009 at 4:34 PM, Kugutsumen wrote: > > > > > Two weeks ago, I've sent my applications ID to both you and Nick and I > > > > haven't heard from you since then. > > > > > Thanks- Hide quoted text - > > > > - Show quoted text -- Hide quoted text - > > - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
Hi, The whole thing about datastore size is ***really*** frustating. I am using 30.94 GB for my app (twitterautofollow) and 1) I don't know where it is being consumed and 2) I don't trust the figures, I delete data and the size of the datastore never goes down, so effectively I feel like am paying and I don't know what it is that I am paying for and 3) I honestly don't know how I could be using that much storage. A case in point, I had another App where I spent 2 months deleting data never to see the size decrease, I removed all the indexes from the system then two days later it was empty. I just feel frustrated that I can't account for anything, and unfortunatly it is too late for me to design my app to have my own accounting in place. Paul. 2009/5/14 Andy Freeman > > Argh! > > This means that one form (db.Key) is smaller than the other > (comparable string) for the datastore while the reverse is true for > memcache. > > I've created am issue ( > http://code.google.com/p/googleappengine/issues/detail?id=1538 > )requesting a __getstate__ and __setstate__ for db.Key that is smaller > than the string equivalent. In addition to eliminating the > inconsistency betwen the datastore and memcache sizes, it will reduce > the size of every memcache'd db.Model instance whose .key() is > defined. > > On May 13, 11:41 am, "Jason (Google)" wrote: > > Hi Andy. In this case, the list of Key objects will be smaller than the > list > > of key strings. Even though the picked db.Key object is larger, it is a > > binary-encoded protocol buffer form that gets stored, which is smaller > than > > the pickled string. That said, I doubt it would make a tremendous > difference > > unless you have a lot of these entities or these lists have a lot of > values. > > > > - Jason > > > > > > > > On Mon, May 11, 2009 at 10:38 PM, Andy Freeman > wrote: > > > > > Since index space can be significant, can we get some additional > > > information? > > > > > For example, does an indexed db.ListProperty(db.Key) with three > > > elements take significantly more or less space than an indexed > > > db.StringListProperty with three elements whose value is str() of the > > > same keys? (The pickle of keys seems to be significantly larger than > > > the pickle of the equivalent strings.) > > > > > On May 11, 5:04 pm, "Jason (Google)" wrote: > > > > Hi Anthony. I'm very sorry for the late reply, and thank you for > bearing > > > > with me. I've discussed this with the datastore team and it's evident > > > that > > > > the CSV file's size is not a great indicator of how much storage your > > > > entities will consume. On top of the size of the raw data, each > entity > > > has > > > > associated metadata, as you've already mentioned, but I'd bet that > the > > > > indexes are consuming the greatest space. If you don't ever query on > one > > > or > > > > more of these 15 string properties, you may consider changing their > > > property > > > > types to Text or declaring indexed=false in your model. If you can do > > > this > > > > with one of your properties and re-build your indexes, I'd be > interested > > > in > > > > seeing how much your storage usage decreases since you'll need one > less > > > > index. > > > > > > (Note that single-property indexes are present but not listed in the > > > Admin > > > > Console.) > > > > > > - Jason > > > > > > On Sat, May 9, 2009 at 4:34 PM, Kugutsumen > wrote: > > > > > > > Two weeks ago, I've sent my applications ID to both you and Nick > and I > > > > > haven't heard from you since then. > > > > > > > Thanks- Hide quoted text - > > > > > > - Show quoted text -- Hide quoted text - > > > > - Show quoted text - > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
Howdy I agree with you paul. I just deleted the contents of my datastore (which took about 2 days - as if that amount of time is not wierd in itself, let alone 2 months), and at the end it was showing 130 meg (or 13% usage). What the? Sorry but what was the originaly argument against a "clear-all" switch on the data store again? cheers Sri On May 14, 7:41 pm, Paul Kinlan wrote: > Hi, > > The whole thing about datastore size is ***really*** frustating. I am using > 30.94 GB for my app (twitterautofollow) and 1) I don't know where it is > being consumed and 2) I don't trust the figures, I delete data and the size > of the datastore never goes down, so effectively I feel like am paying and I > don't know what it is that I am paying for and 3) I honestly don't know how > I could be using that much storage. > > A case in point, I had another App where I spent 2 months deleting data > never to see the size decrease, I removed all the indexes from the system > then two days later it was empty. > > I just feel frustrated that I can't account for anything, and unfortunatly > it is too late for me to design my app to have my own accounting in place. > > Paul. > > 2009/5/14 Andy Freeman > > > > > Argh! > > > This means that one form (db.Key) is smaller than the other > > (comparable string) for the datastore while the reverse is true for > > memcache. > > > I've created am issue ( > >http://code.google.com/p/googleappengine/issues/detail?id=1538 > > )requesting a __getstate__ and __setstate__ for db.Key that is smaller > > than the string equivalent. In addition to eliminating the > > inconsistency betwen the datastore and memcache sizes, it will reduce > > the size of every memcache'd db.Model instance whose .key() is > > defined. > > > On May 13, 11:41 am, "Jason (Google)" wrote: > > > Hi Andy. In this case, the list of Key objects will be smaller than the > > list > > > of key strings. Even though the picked db.Key object is larger, it is a > > > binary-encoded protocol buffer form that gets stored, which is smaller > > than > > > the pickled string. That said, I doubt it would make a tremendous > > difference > > > unless you have a lot of these entities or these lists have a lot of > > values. > > > > - Jason > > > > On Mon, May 11, 2009 at 10:38 PM, Andy Freeman > > wrote: > > > > > Since index space can be significant, can we get some additional > > > > information? > > > > > For example, does an indexed db.ListProperty(db.Key) with three > > > > elements take significantly more or less space than an indexed > > > > db.StringListProperty with three elements whose value is str() of the > > > > same keys? (The pickle of keys seems to be significantly larger than > > > > the pickle of the equivalent strings.) > > > > > On May 11, 5:04 pm, "Jason (Google)" wrote: > > > > > Hi Anthony. I'm very sorry for the late reply, and thank you for > > bearing > > > > > with me. I've discussed this with the datastore team and it's evident > > > > that > > > > > the CSV file's size is not a great indicator of how much storage your > > > > > entities will consume. On top of the size of the raw data, each > > entity > > > > has > > > > > associated metadata, as you've already mentioned, but I'd bet that > > the > > > > > indexes are consuming the greatest space. If you don't ever query on > > one > > > > or > > > > > more of these 15 string properties, you may consider changing their > > > > property > > > > > types to Text or declaring indexed=false in your model. If you can do > > > > this > > > > > with one of your properties and re-build your indexes, I'd be > > interested > > > > in > > > > > seeing how much your storage usage decreases since you'll need one > > less > > > > > index. > > > > > > (Note that single-property indexes are present but not listed in the > > > > Admin > > > > > Console.) > > > > > > - Jason > > > > > > On Sat, May 9, 2009 at 4:34 PM, Kugutsumen > > wrote: > > > > > > > Two weeks ago, I've sent my applications ID to both you and Nick > > and I > > > > > > haven't heard from you since then. > > > > > > > Thanks- Hide quoted text - > > > > > > - Show quoted text -- Hide quoted text - > > > > - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
Just to be fair, when I recently checked all the data had returned to 0% usage. But that doesnt explain the 3 entities i had uploaded 12 hours ago On May 14, 10:43 pm, Sri wrote: > Howdy > > I agree with you paul. I just deleted the contents of my > datastore (which took about 2 days - as if that amount of time is not > wierd in itself, let alone 2 months), and at the end it was showing > 130 meg (or 13% usage). What the? > > Sorry but what was the originaly argument against a "clear-all" switch > on the data store again? > > cheers > Sri > > On May 14, 7:41 pm, Paul Kinlan wrote: > > > Hi, > > > The whole thing about datastore size is ***really*** frustating. I am using > > 30.94 GB for my app (twitterautofollow) and 1) I don't know where it is > > being consumed and 2) I don't trust the figures, I delete data and the size > > of the datastore never goes down, so effectively I feel like am paying and I > > don't know what it is that I am paying for and 3) I honestly don't know how > > I could be using that much storage. > > > A case in point, I had another App where I spent 2 months deleting data > > never to see the size decrease, I removed all the indexes from the system > > then two days later it was empty. > > > I just feel frustrated that I can't account for anything, and unfortunatly > > it is too late for me to design my app to have my own accounting in place. > > > Paul. > > > 2009/5/14 Andy Freeman > > > > Argh! > > > > This means that one form (db.Key) is smaller than the other > > > (comparable string) for the datastore while the reverse is true for > > > memcache. > > > > I've created am issue ( > > >http://code.google.com/p/googleappengine/issues/detail?id=1538 > > > )requesting a __getstate__ and __setstate__ for db.Key that is smaller > > > than the string equivalent. In addition to eliminating the > > > inconsistency betwen the datastore and memcache sizes, it will reduce > > > the size of every memcache'd db.Model instance whose .key() is > > > defined. > > > > On May 13, 11:41 am, "Jason (Google)" wrote: > > > > Hi Andy. In this case, the list of Key objects will be smaller than the > > > list > > > > of key strings. Even though the picked db.Key object is larger, it is a > > > > binary-encoded protocol buffer form that gets stored, which is smaller > > > than > > > > the pickled string. That said, I doubt it would make a tremendous > > > difference > > > > unless you have a lot of these entities or these lists have a lot of > > > values. > > > > > - Jason > > > > > On Mon, May 11, 2009 at 10:38 PM, Andy Freeman > > > wrote: > > > > > > Since index space can be significant, can we get some additional > > > > > information? > > > > > > For example, does an indexed db.ListProperty(db.Key) with three > > > > > elements take significantly more or less space than an indexed > > > > > db.StringListProperty with three elements whose value is str() of the > > > > > same keys? (The pickle of keys seems to be significantly larger than > > > > > the pickle of the equivalent strings.) > > > > > > On May 11, 5:04 pm, "Jason (Google)" wrote: > > > > > > Hi Anthony. I'm very sorry for the late reply, and thank you for > > > bearing > > > > > > with me. I've discussed this with the datastore team and it's > > > > > > evident > > > > > that > > > > > > the CSV file's size is not a great indicator of how much storage > > > > > > your > > > > > > entities will consume. On top of the size of the raw data, each > > > entity > > > > > has > > > > > > associated metadata, as you've already mentioned, but I'd bet that > > > the > > > > > > indexes are consuming the greatest space. If you don't ever query on > > > one > > > > > or > > > > > > more of these 15 string properties, you may consider changing their > > > > > property > > > > > > types to Text or declaring indexed=false in your model. If you can > > > > > > do > > > > > this > > > > > > with one of your properties and re-build your indexes, I'd be > > > interested > > > > > in > > > > > > seeing how much your storage usage decreases since you'll need one > > > less > > > > > > index. > > > > > > > (Note that single-property indexes are present but not listed in the > > > > > Admin > > > > > > Console.) > > > > > > > - Jason > > > > > > > On Sat, May 9, 2009 at 4:34 PM, Kugutsumen > > > wrote: > > > > > > > > Two weeks ago, I've sent my applications ID to both you and Nick > > > and I > > > > > > > haven't heard from you since then. > > > > > > > > Thanks- Hide quoted text - > > > > > > > - Show quoted text -- Hide quoted text - > > > > > - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://group
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
My main issue is that I can't account for the data, and I don't know how to trust the value that I am getting billed for. Paul 2009/5/14 Sri > > Just to be fair, when I recently checked all the data had returned to > 0% usage. But that doesnt explain the 3 entities i had uploaded > 12 hours ago > > On May 14, 10:43 pm, Sri wrote: > > Howdy > > > > I agree with you paul. I just deleted the contents of my > > datastore (which took about 2 days - as if that amount of time is not > > wierd in itself, let alone 2 months), and at the end it was showing > > 130 meg (or 13% usage). What the? > > > > Sorry but what was the originaly argument against a "clear-all" switch > > on the data store again? > > > > cheers > > Sri > > > > On May 14, 7:41 pm, Paul Kinlan wrote: > > > > > Hi, > > > > > The whole thing about datastore size is ***really*** frustating. I am > using > > > 30.94 GB for my app (twitterautofollow) and 1) I don't know where it is > > > being consumed and 2) I don't trust the figures, I delete data and the > size > > > of the datastore never goes down, so effectively I feel like am paying > and I > > > don't know what it is that I am paying for and 3) I honestly don't know > how > > > I could be using that much storage. > > > > > A case in point, I had another App where I spent 2 months deleting data > > > never to see the size decrease, I removed all the indexes from the > system > > > then two days later it was empty. > > > > > I just feel frustrated that I can't account for anything, and > unfortunatly > > > it is too late for me to design my app to have my own accounting in > place. > > > > > Paul. > > > > > 2009/5/14 Andy Freeman > > > > > > Argh! > > > > > > This means that one form (db.Key) is smaller than the other > > > > (comparable string) for the datastore while the reverse is true for > > > > memcache. > > > > > > I've created am issue ( > > > >http://code.google.com/p/googleappengine/issues/detail?id=1538 > > > > )requesting a __getstate__ and __setstate__ for db.Key that is > smaller > > > > than the string equivalent. In addition to eliminating the > > > > inconsistency betwen the datastore and memcache sizes, it will reduce > > > > the size of every memcache'd db.Model instance whose .key() is > > > > defined. > > > > > > On May 13, 11:41 am, "Jason (Google)" wrote: > > > > > Hi Andy. In this case, the list of Key objects will be smaller than > the > > > > list > > > > > of key strings. Even though the picked db.Key object is larger, it > is a > > > > > binary-encoded protocol buffer form that gets stored, which is > smaller > > > > than > > > > > the pickled string. That said, I doubt it would make a tremendous > > > > difference > > > > > unless you have a lot of these entities or these lists have a lot > of > > > > values. > > > > > > > - Jason > > > > > > > On Mon, May 11, 2009 at 10:38 PM, Andy Freeman < > ana...@earthlink.net> > > > > wrote: > > > > > > > > Since index space can be significant, can we get some additional > > > > > > information? > > > > > > > > For example, does an indexed db.ListProperty(db.Key) with three > > > > > > elements take significantly more or less space than an indexed > > > > > > db.StringListProperty with three elements whose value is str() of > the > > > > > > same keys? (The pickle of keys seems to be significantly larger > than > > > > > > the pickle of the equivalent strings.) > > > > > > > > On May 11, 5:04 pm, "Jason (Google)" > wrote: > > > > > > > Hi Anthony. I'm very sorry for the late reply, and thank you > for > > > > bearing > > > > > > > with me. I've discussed this with the datastore team and it's > evident > > > > > > that > > > > > > > the CSV file's size is not a great indicator of how much > storage your > > > > > > > entities will consume. On top of the size of the raw data, each > > > > entity > > > > > > has > > > > > > > associated metadata, as you've already mentioned, but I'd bet > that > > > > the > > > > > > > indexes are consuming the greatest space. If you don't ever > query on > > > > one > > > > > > or > > > > > > > more of these 15 string properties, you may consider changing > their > > > > > > property > > > > > > > types to Text or declaring indexed=false in your model. If you > can do > > > > > > this > > > > > > > with one of your properties and re-build your indexes, I'd be > > > > interested > > > > > > in > > > > > > > seeing how much your storage usage decreases since you'll need > one > > > > less > > > > > > > index. > > > > > > > > > (Note that single-property indexes are present but not listed > in the > > > > > > Admin > > > > > > > Console.) > > > > > > > > > - Jason > > > > > > > > > On Sat, May 9, 2009 at 4:34 PM, Kugutsumen < > kugutsu...@gmail.com> > > > > wrote: > > > > > > > > > > Two weeks ago, I've sent my applications ID to both you and > Nick > > > > and I > > > > > > > > haven't heard from you since then. > > > > > > > > > > Thanks- Hide quoted text - > > > > > > > >
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
Yeah -- I just checked this evening, and my database size has now dropped by 10% -- roughly in line with the number of entities that I had deleted. Maybe there is some cleanup process that only runs occasionally However, it is *really* frustrating not to know what aspect of your application is consuming space Philip On May 14, 8:46 am, Sri wrote: > Just to be fair, when I recently checked all the data had returned to > 0% usage. But that doesnt explain the 3 entities i had uploaded > 12 hours ago > > On May 14, 10:43 pm, Sri wrote: > > > Howdy > > > I agree with you paul. I just deleted the contents of my > > datastore (which took about 2 days - as if that amount of time is not > > wierd in itself, let alone 2 months), and at the end it was showing > > 130 meg (or 13% usage). What the? > > > Sorry but what was the originaly argument against a "clear-all" switch > > on the data store again? > > > cheers > > Sri > > > On May 14, 7:41 pm, Paul Kinlan wrote: > > > > Hi, > > > > The whole thing about datastore size is ***really*** frustating. I am > > > using > > > 30.94 GB for my app (twitterautofollow) and 1) I don't know where it is > > > being consumed and 2) I don't trust the figures, I delete data and the > > > size > > > of the datastore never goes down, so effectively I feel like am paying > > > and I > > > don't know what it is that I am paying for and 3) I honestly don't know > > > how > > > I could be using that much storage. > > > > A case in point, I had another App where I spent 2 months deleting data > > > never to see the size decrease, I removed all the indexes from the system > > > then two days later it was empty. > > > > I just feel frustrated that I can't account for anything, and unfortunatly > > > it is too late for me to design my app to have my own accounting in place. > > > > Paul. > > > > 2009/5/14 Andy Freeman > > > > > Argh! > > > > > This means that one form (db.Key) is smaller than the other > > > > (comparable string) for the datastore while the reverse is true for > > > > memcache. > > > > > I've created am issue ( > > > >http://code.google.com/p/googleappengine/issues/detail?id=1538 > > > > )requesting a __getstate__ and __setstate__ for db.Key that is smaller > > > > than the string equivalent. In addition to eliminating the > > > > inconsistency betwen the datastore and memcache sizes, it will reduce > > > > the size of every memcache'd db.Model instance whose .key() is > > > > defined. > > > > > On May 13, 11:41 am, "Jason (Google)" wrote: > > > > > Hi Andy. In this case, the list of Key objects will be smaller than > > > > > the > > > > list > > > > > of key strings. Even though the picked db.Key object is larger, it is > > > > > a > > > > > binary-encoded protocol buffer form that gets stored, which is smaller > > > > than > > > > > the pickled string. That said, I doubt it would make a tremendous > > > > difference > > > > > unless you have a lot of these entities or these lists have a lot of > > > > values. > > > > > > - Jason > > > > > > On Mon, May 11, 2009 at 10:38 PM, Andy Freeman > > > > wrote: > > > > > > > Since index space can be significant, can we get some additional > > > > > > information? > > > > > > > For example, does an indexed db.ListProperty(db.Key) with three > > > > > > elements take significantly more or less space than an indexed > > > > > > db.StringListProperty with three elements whose value is str() of > > > > > > the > > > > > > same keys? (The pickle of keys seems to be significantly larger > > > > > > than > > > > > > the pickle of the equivalent strings.) > > > > > > > On May 11, 5:04 pm, "Jason (Google)" wrote: > > > > > > > Hi Anthony. I'm very sorry for the late reply, and thank you for > > > > bearing > > > > > > > with me. I've discussed this with the datastore team and it's > > > > > > > evident > > > > > > that > > > > > > > the CSV file's size is not a great indicator of how much storage > > > > > > > your > > > > > > > entities will consume. On top of the size of the raw data, each > > > > entity > > > > > > has > > > > > > > associated metadata, as you've already mentioned, but I'd bet that > > > > the > > > > > > > indexes are consuming the greatest space. If you don't ever query > > > > > > > on > > > > one > > > > > > or > > > > > > > more of these 15 string properties, you may consider changing > > > > > > > their > > > > > > property > > > > > > > types to Text or declaring indexed=false in your model. If you > > > > > > > can do > > > > > > this > > > > > > > with one of your properties and re-build your indexes, I'd be > > > > interested > > > > > > in > > > > > > > seeing how much your storage usage decreases since you'll need one > > > > less > > > > > > > index. > > > > > > > > (Note that single-property indexes are present but not listed in > > > > > > > the > > > > > > Admin > > > > > > > Console.) > > > > > > > > - Jason > > > > > > > > On Sat, May 9, 2009 at 4:34 PM,
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
fair enough mate. I still have to applaud your patience. hats off mate, if you can sit around for 2 months while your data is being deleted! On May 14, 10:55 pm, Paul Kinlan wrote: > My main issue is that I can't account for the data, and I don't know how to > trust the value that I am getting billed for. > > Paul > > 2009/5/14 Sri > > > > > Just to be fair, when I recently checked all the data had returned to > > 0% usage. But that doesnt explain the 3 entities i had uploaded > > 12 hours ago > > > On May 14, 10:43 pm, Sri wrote: > > > Howdy > > > > I agree with you paul. I just deleted the contents of my > > > datastore (which took about 2 days - as if that amount of time is not > > > wierd in itself, let alone 2 months), and at the end it was showing > > > 130 meg (or 13% usage). What the? > > > > Sorry but what was the originaly argument against a "clear-all" switch > > > on the data store again? > > > > cheers > > > Sri > > > > On May 14, 7:41 pm, Paul Kinlan wrote: > > > > > Hi, > > > > > The whole thing about datastore size is ***really*** frustating. I am > > using > > > > 30.94 GB for my app (twitterautofollow) and 1) I don't know where it is > > > > being consumed and 2) I don't trust the figures, I delete data and the > > size > > > > of the datastore never goes down, so effectively I feel like am paying > > and I > > > > don't know what it is that I am paying for and 3) I honestly don't know > > how > > > > I could be using that much storage. > > > > > A case in point, I had another App where I spent 2 months deleting data > > > > never to see the size decrease, I removed all the indexes from the > > system > > > > then two days later it was empty. > > > > > I just feel frustrated that I can't account for anything, and > > unfortunatly > > > > it is too late for me to design my app to have my own accounting in > > place. > > > > > Paul. > > > > > 2009/5/14 Andy Freeman > > > > > > Argh! > > > > > > This means that one form (db.Key) is smaller than the other > > > > > (comparable string) for the datastore while the reverse is true for > > > > > memcache. > > > > > > I've created am issue ( > > > > >http://code.google.com/p/googleappengine/issues/detail?id=1538 > > > > > )requesting a __getstate__ and __setstate__ for db.Key that is > > smaller > > > > > than the string equivalent. In addition to eliminating the > > > > > inconsistency betwen the datastore and memcache sizes, it will reduce > > > > > the size of every memcache'd db.Model instance whose .key() is > > > > > defined. > > > > > > On May 13, 11:41 am, "Jason (Google)" wrote: > > > > > > Hi Andy. In this case, the list of Key objects will be smaller than > > the > > > > > list > > > > > > of key strings. Even though the picked db.Key object is larger, it > > is a > > > > > > binary-encoded protocol buffer form that gets stored, which is > > smaller > > > > > than > > > > > > the pickled string. That said, I doubt it would make a tremendous > > > > > difference > > > > > > unless you have a lot of these entities or these lists have a lot > > of > > > > > values. > > > > > > > - Jason > > > > > > > On Mon, May 11, 2009 at 10:38 PM, Andy Freeman < > > ana...@earthlink.net> > > > > > wrote: > > > > > > > > Since index space can be significant, can we get some additional > > > > > > > information? > > > > > > > > For example, does an indexed db.ListProperty(db.Key) with three > > > > > > > elements take significantly more or less space than an indexed > > > > > > > db.StringListProperty with three elements whose value is str() of > > the > > > > > > > same keys? (The pickle of keys seems to be significantly larger > > than > > > > > > > the pickle of the equivalent strings.) > > > > > > > > On May 11, 5:04 pm, "Jason (Google)" > > wrote: > > > > > > > > Hi Anthony. I'm very sorry for the late reply, and thank you > > for > > > > > bearing > > > > > > > > with me. I've discussed this with the datastore team and it's > > evident > > > > > > > that > > > > > > > > the CSV file's size is not a great indicator of how much > > storage your > > > > > > > > entities will consume. On top of the size of the raw data, each > > > > > entity > > > > > > > has > > > > > > > > associated metadata, as you've already mentioned, but I'd bet > > that > > > > > the > > > > > > > > indexes are consuming the greatest space. If you don't ever > > query on > > > > > one > > > > > > > or > > > > > > > > more of these 15 string properties, you may consider changing > > their > > > > > > > property > > > > > > > > types to Text or declaring indexed=false in your model. If you > > can do > > > > > > > this > > > > > > > > with one of your properties and re-build your indexes, I'd be > > > > > interested > > > > > > > in > > > > > > > > seeing how much your storage usage decreases since you'll need > > one > > > > > less > > > > > > > > index. > > > > > > > > > (Note that single-property indexes are present but not listed > > in the > > > >
[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)
To be fair it was a process I kicked off to come in line with billing, I forgot about it, checked it and it was still running. Paul 2009/5/15 Sri > > fair enough mate. I still have to applaud your patience. hats off > mate, if you can sit around for 2 months while your data is being > deleted! > > On May 14, 10:55 pm, Paul Kinlan wrote: > > My main issue is that I can't account for the data, and I don't know how > to > > trust the value that I am getting billed for. > > > > Paul > > > > 2009/5/14 Sri > > > > > > > > > Just to be fair, when I recently checked all the data had returned to > > > 0% usage. But that doesnt explain the 3 entities i had uploaded > > > 12 hours ago > > > > > On May 14, 10:43 pm, Sri wrote: > > > > Howdy > > > > > > I agree with you paul. I just deleted the contents of my > > > > datastore (which took about 2 days - as if that amount of time is not > > > > wierd in itself, let alone 2 months), and at the end it was showing > > > > 130 meg (or 13% usage). What the? > > > > > > Sorry but what was the originaly argument against a "clear-all" > switch > > > > on the data store again? > > > > > > cheers > > > > Sri > > > > > > On May 14, 7:41 pm, Paul Kinlan wrote: > > > > > > > Hi, > > > > > > > The whole thing about datastore size is ***really*** frustating. I > am > > > using > > > > > 30.94 GB for my app (twitterautofollow) and 1) I don't know where > it is > > > > > being consumed and 2) I don't trust the figures, I delete data and > the > > > size > > > > > of the datastore never goes down, so effectively I feel like am > paying > > > and I > > > > > don't know what it is that I am paying for and 3) I honestly don't > know > > > how > > > > > I could be using that much storage. > > > > > > > A case in point, I had another App where I spent 2 months deleting > data > > > > > never to see the size decrease, I removed all the indexes from the > > > system > > > > > then two days later it was empty. > > > > > > > I just feel frustrated that I can't account for anything, and > > > unfortunatly > > > > > it is too late for me to design my app to have my own accounting in > > > place. > > > > > > > Paul. > > > > > > > 2009/5/14 Andy Freeman > > > > > > > > Argh! > > > > > > > > This means that one form (db.Key) is smaller than the other > > > > > > (comparable string) for the datastore while the reverse is true > for > > > > > > memcache. > > > > > > > > I've created am issue ( > > > > > >http://code.google.com/p/googleappengine/issues/detail?id=1538 > > > > > > )requesting a __getstate__ and __setstate__ for db.Key that is > > > smaller > > > > > > than the string equivalent. In addition to eliminating the > > > > > > inconsistency betwen the datastore and memcache sizes, it will > reduce > > > > > > the size of every memcache'd db.Model instance whose .key() is > > > > > > defined. > > > > > > > > On May 13, 11:41 am, "Jason (Google)" > wrote: > > > > > > > Hi Andy. In this case, the list of Key objects will be smaller > than > > > the > > > > > > list > > > > > > > of key strings. Even though the picked db.Key object is larger, > it > > > is a > > > > > > > binary-encoded protocol buffer form that gets stored, which is > > > smaller > > > > > > than > > > > > > > the pickled string. That said, I doubt it would make a > tremendous > > > > > > difference > > > > > > > unless you have a lot of these entities or these lists have a > lot > > > of > > > > > > values. > > > > > > > > > - Jason > > > > > > > > > On Mon, May 11, 2009 at 10:38 PM, Andy Freeman < > > > ana...@earthlink.net> > > > > > > wrote: > > > > > > > > > > Since index space can be significant, can we get some > additional > > > > > > > > information? > > > > > > > > > > For example, does an indexed db.ListProperty(db.Key) with > three > > > > > > > > elements take significantly more or less space than an > indexed > > > > > > > > db.StringListProperty with three elements whose value is > str() of > > > the > > > > > > > > same keys? (The pickle of keys seems to be significantly > larger > > > than > > > > > > > > the pickle of the equivalent strings.) > > > > > > > > > > On May 11, 5:04 pm, "Jason (Google)" > > > wrote: > > > > > > > > > Hi Anthony. I'm very sorry for the late reply, and thank > you > > > for > > > > > > bearing > > > > > > > > > with me. I've discussed this with the datastore team and > it's > > > evident > > > > > > > > that > > > > > > > > > the CSV file's size is not a great indicator of how much > > > storage your > > > > > > > > > entities will consume. On top of the size of the raw data, > each > > > > > > entity > > > > > > > > has > > > > > > > > > associated metadata, as you've already mentioned, but I'd > bet > > > that > > > > > > the > > > > > > > > > indexes are consuming the greatest space. If you don't ever > > > query on > > > > > > one > > > > > > > > or > > > > > > > > > more of these 15 string properties, you may consider > changing > > > their > > > > > >
