[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-15 Thread Sri
fair enough mate. I still have to applaud your patience. hats off mate, if you can sit around for 2 months while your data is being deleted! On May 14, 10:55 pm, Paul Kinlan paul.kin...@gmail.com wrote: My main issue is that I can't account for the data, and I don't know how to trust the

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-15 Thread Paul Kinlan
To be fair it was a process I kicked off to come in line with billing, I forgot about it, checked it and it was still running. Paul 2009/5/15 Sri sri.pan...@gmail.com fair enough mate. I still have to applaud your patience. hats off mate, if you can sit around for 2 months while your data

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-14 Thread Paul Kinlan
Hi, The whole thing about datastore size is ***really*** frustating. I am using 30.94 GB for my app (twitterautofollow) and 1) I don't know where it is being consumed and 2) I don't trust the figures, I delete data and the size of the datastore never goes down, so effectively I feel like am

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-14 Thread Sri
Howdy I agree with you paul. I just deleted the contents of my datastore (which took about 2 days - as if that amount of time is not wierd in itself, let alone 2 months), and at the end it was showing 130 meg (or 13% usage). What the? Sorry but what was the originaly argument against a

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-14 Thread Sri
Just to be fair, when I recently checked all the data had returned to 0% usage. But that doesnt explain the 3 entities i had uploaded 12 hours ago On May 14, 10:43 pm, Sri sri.pan...@gmail.com wrote: Howdy     I agree with you paul.  I just deleted the contents of my datastore

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-14 Thread Paul Kinlan
My main issue is that I can't account for the data, and I don't know how to trust the value that I am getting billed for. Paul 2009/5/14 Sri sri.pan...@gmail.com Just to be fair, when I recently checked all the data had returned to 0% usage. But that doesnt explain the 3 entities i had

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-14 Thread WeatherPhilip
Yeah -- I just checked this evening, and my database size has now dropped by 10% -- roughly in line with the number of entities that I had deleted. Maybe there is some cleanup process that only runs occasionally However, it is *really* frustrating not to know what aspect of your application

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-13 Thread Jason (Google)
Hi Andy. In this case, the list of Key objects will be smaller than the list of key strings. Even though the picked db.Key object is larger, it is a binary-encoded protocol buffer form that gets stored, which is smaller than the pickled string. That said, I doubt it would make a tremendous

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-13 Thread Jason (Google)
Hi Philip. Calling to_xml() is not a great indicator of the size of your entity as stored in BigTable. Unfortunately, there is currently no straightforward way to estimate how large your entities are, although we're working on possible solutions to this problem. Without knowing your data model or

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-13 Thread WeatherPhilip
Jason I removed a bunch of single property indexes (by setting indexed=False and then loaded and stored each item. This didn't save much (a few percent). Also, the fact that I can't see the single property indexes makes it more tricky to figure out if they have really gone or not! I'm now

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-13 Thread Andy Freeman
Argh! This means that one form (db.Key) is smaller than the other (comparable string) for the datastore while the reverse is true for memcache. How about defining a __getstate__ and __setstate__ for db.Key that is smaller than the string equivalent? This will help for memcaching any db.Model

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-13 Thread Andy Freeman
Argh! This means that one form (db.Key) is smaller than the other (comparable string) for the datastore while the reverse is true for memcache. I've created am issue ( http://code.google.com/p/googleappengine/issues/detail?id=1538 )requesting a __getstate__ and __setstate__ for db.Key that is

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-12 Thread WeatherPhilip
I just did a test on one of my apps. Nearly all my data is in a single model. I have 163189 instances, and the total size (calculated by reading each instance and running to_xml() on it, and then adding up the results) is 281,145,536 bytes. Most of my properties have indexed=False. The dashboard

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-11 Thread Jason (Google)
Hi Anthony. I'm very sorry for the late reply, and thank you for bearing with me. I've discussed this with the datastore team and it's evident that the CSV file's size is not a great indicator of how much storage your entities will consume. On top of the size of the raw data, each entity has

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-11 Thread Andy Freeman
Since index space can be significant, can we get some additional information? For example, does an indexed db.ListProperty(db.Key) with three elements take significantly more or less space than an indexed db.StringListProperty with three elements whose value is str() of the same keys? (The

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-05-09 Thread Kugutsumen
Two weeks ago, I've sent my applications ID to both you and Nick and I haven't heard from you since then. Thanks --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-04-30 Thread Kugutsumen
I've sent you my ID. Thanks for looking into this. On Apr 29, 4:06 am, Jason (Google) apija...@google.com wrote: Can you both provide your application IDs so I can investigate a bit? Thanks, - Jason On Sat, Apr 25, 2009 at 12:33 PM, Kugutsumen kugutsu...@gmail.com wrote: On Apr 23,

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-04-25 Thread Kugutsumen
On Apr 23, 4:47 am, Panos pa...@acm.org wrote: I have also been puzzled at times on where the space is going. I filed this request today: More granular accounting of how datastore space is usedhttp://code.google.com/p/googleappengine/issues/detail?id=1396 Please browse to the issue and

[google-appengine] Re: Datastore usage ~ 80 times more than expected (Add your vote to a datastore usage accounting feature)

2009-04-22 Thread Panos
I have also been puzzled at times on where the space is going. I filed this request today: More granular accounting of how datastore space is used http://code.google.com/p/googleappengine/issues/detail?id=1396 Please browse to the issue and add your vote/star if you want to see this feature