Hi Amir,
The amount of storage used depends hugely on what indexes you define.
In particular, there is the problem of 'exploding indexes'. Suppose
you have an entity with the following properties:
foo.alist = [1, 2, 3, 4, 5]
foo.anotherlist = ["foo", "bar", "baz", "quux"]
foo.anumber = 42
And s
On Apr 21, 2:10 pm, Nick Johnson wrote:
> Hi Amir,
>
> The amount of storage used depends hugely on what indexes you define.
> In particular, there is the problem of 'exploding indexes'. Suppose
> you have an entity with the following properties:
My models are of this form:
class A(db.Model):
Hi Amir,
If these are the only kinds and indexes you have then this is indeed
anomalous. Send me your app's ID (via email, if you wish) and I'll
have someone look into it.
-Nick Johnson
--~--~-~--~~~---~--~~
You received this message because you are subscribed to
On Wed, Apr 22, 2009 at 1:19 PM, Nick Johnson wrote:
>
> Hi Amir,
>
> If these are the only kinds and indexes you have then this is indeed
> anomalous. Send me your app's ID (via email, if you wish) and I'll
> have someone look into it.
>
> -Nick Johnson
>
> >
>
--~--~-~--~~--
How are you estimating the size? For example, do you think that
strings are stored using one byte per character or two? (I don't
know, but I do know that they're interpreted as unicode.)
I've asked for mechanisms to help estimate size - see
http://code.google.com/p/googleappengine/issues/detail
> A rough estimate shows the app engine is using 80 times more storage
> than one might expect given the data stored there.
Is your storage volume analysis based on 100 records or 10,000?
--~--~-~--~~~---~--~~
You received this message because you are subscribed to
Hi,
This turned out to be a temporary error. Usage is now (apparently)
reported correctly.
Amir
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to googl
I have also been puzzled at times on where the space is going. I filed
this request today:
"More granular accounting of how datastore space is used"
http://code.google.com/p/googleappengine/issues/detail?id=1396
Please browse to the issue and add your vote/star if you want to see
this feature im
On Apr 23, 4:47 am, Panos wrote:
> I have also been puzzled at times on where the space is going. I filed
> this request today:
>
> "More granular accounting of how datastore space is
> used"http://code.google.com/p/googleappengine/issues/detail?id=1396
>
> Please browse to the issue and add y
Can you both provide your application IDs so I can investigate a bit?
Thanks,
- Jason
On Sat, Apr 25, 2009 at 12:33 PM, Kugutsumen wrote:
>
>
>
> On Apr 23, 4:47 am, Panos wrote:
> > I have also been puzzled at times on where the space is going. I filed
> > this request today:
> >
> > "More gr
I've sent you my ID.
Thanks for looking into this.
On Apr 29, 4:06 am, "Jason (Google)" wrote:
> Can you both provide your application IDs so I can investigate a bit?
>
> Thanks,
> - Jason
>
> On Sat, Apr 25, 2009 at 12:33 PM, Kugutsumen wrote:
>
> > On Apr 23, 4:47 am, Panos wrote:
> > > I h
I've created the following issue:
http://code.google.com/p/googleappengine/issues/detail?id=1436
On Apr 29, 4:06 am, "Jason (Google)" wrote:
> Can you both provide your application IDs so I can investigate a bit?
>
> Thanks,
> - Jason
>
> On Sat, Apr 25, 2009 at 12:33 PM, Kugutsumen wrote:
>
>
Two weeks ago, I've sent my applications ID to both you and Nick and I
haven't heard from you since then.
Thanks
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send
Hi Anthony. I'm very sorry for the late reply, and thank you for bearing
with me. I've discussed this with the datastore team and it's evident that
the CSV file's size is not a great indicator of how much storage your
entities will consume. On top of the size of the raw data, each entity has
associ
Since index space can be significant, can we get some additional
information?
For example, does an indexed db.ListProperty(db.Key) with three
elements take significantly more or less space than an indexed
db.StringListProperty with three elements whose value is str() of the
same keys? (The pickl
I just did a test on one of my apps. Nearly all my data is in a single
model.
I have 163189 instances, and the total size (calculated by reading
each instance and running to_xml() on it, and then adding up the
results) is 281,145,536 bytes. Most of my properties have
indexed=False. The dashboard
Hi Andy. In this case, the list of Key objects will be smaller than the list
of key strings. Even though the picked db.Key object is larger, it is a
binary-encoded protocol buffer form that gets stored, which is smaller than
the pickled string. That said, I doubt it would make a tremendous differen
Hi Philip. Calling to_xml() is not a great indicator of the size of your
entity as stored in BigTable. Unfortunately, there is currently no
straightforward way to estimate how large your entities are, although we're
working on possible solutions to this problem.
Without knowing your data model or
Jason
I removed a bunch of single property indexes (by setting indexed=False
and then loaded and stored each item. This didn't save much (a few
percent). Also, the fact that I can't see the single property indexes
makes it more tricky to figure out if they have really gone or not!
I'm now deleti
Argh!
This means that one form (db.Key) is smaller than the other
(comparable string) for the datastore while the reverse is true for
memcache.
How about defining a __getstate__ and __setstate__ for db.Key that is
smaller than the string equivalent? This will help for memcaching any
db.Model in
Argh!
This means that one form (db.Key) is smaller than the other
(comparable string) for the datastore while the reverse is true for
memcache.
I've created am issue (
http://code.google.com/p/googleappengine/issues/detail?id=1538
)requesting a __getstate__ and __setstate__ for db.Key that is s
Hi,
The whole thing about datastore size is ***really*** frustating. I am using
30.94 GB for my app (twitterautofollow) and 1) I don't know where it is
being consumed and 2) I don't trust the figures, I delete data and the size
of the datastore never goes down, so effectively I feel like am payin
Howdy
I agree with you paul. I just deleted the contents of my
datastore (which took about 2 days - as if that amount of time is not
wierd in itself, let alone 2 months), and at the end it was showing
130 meg (or 13% usage). What the?
Sorry but what was the originaly argument against a "cl
Just to be fair, when I recently checked all the data had returned to
0% usage. But that doesnt explain the 3 entities i had uploaded
12 hours ago
On May 14, 10:43 pm, Sri wrote:
> Howdy
>
> I agree with you paul. I just deleted the contents of my
> datastore (which took about 2 da
My main issue is that I can't account for the data, and I don't know how to
trust the value that I am getting billed for.
Paul
2009/5/14 Sri
>
> Just to be fair, when I recently checked all the data had returned to
> 0% usage. But that doesnt explain the 3 entities i had uploaded
> 12 hour
Yeah -- I just checked this evening, and my database size has now
dropped by 10% -- roughly in line with the number of entities that I
had deleted. Maybe there is some cleanup process that only runs
occasionally
However, it is *really* frustrating not to know what aspect of your
application i
fair enough mate. I still have to applaud your patience. hats off
mate, if you can sit around for 2 months while your data is being
deleted!
On May 14, 10:55 pm, Paul Kinlan wrote:
> My main issue is that I can't account for the data, and I don't know how to
> trust the value that I am getting
To be fair it was a process I kicked off to come in line with billing, I
forgot about it, checked it and it was still running.
Paul
2009/5/15 Sri
>
> fair enough mate. I still have to applaud your patience. hats off
> mate, if you can sit around for 2 months while your data is being
> deleted
28 matches
Mail list logo
