Thanks Robert. I agree that the difference between 705MB and 2.31 GB is most likely due to indexes.
However, I do not think that the property names is the issue for the metadata. I now believe that the metadata is also such a big percentage because of exploding indexes. I expected that for indexes but not for metadata. In particular, I was surprised to find out that "In addition to the EntitiesByCompositeProperty Bigtable, custom index row data is stored directly in the Entities table as well" (source: http://code.google.com/appengine/articles/storage_breakdown.html) Unfortunately, I need all these multivalue properties in this entity in my app... On the upside, storage is inexpesive, on the downside any time I need to walk the whole table to make a change (e.g. data model change) it is very expensive with regards to CPU time. PK On Mar 5, 9:05 pm, Robert Kluin <robert.kl...@gmail.com> wrote: > Every entity stores the full property name for every property defined > on that model. So if you have long property names it takes a lot of > space. On models with only numbers, booleans, and short strings we > often see metadata make up 50%+ of a models, unless we use really > short property names (2 char). > > The difference between your 705MB and 2.31GB is probably due to > indexes. Unfortunately indexes are not accounted for on the Datastore > Stats page. In one of our apps we have a lot of composite indexes, > our 'total storage' is approximately 10x the 'total size of entities' > reported on the stats page. > > There are a lot of threads discussing these topics if you want more > info. There is also an article that explains how the entity storage > works:http://code.google.com/appengine/articles/storage_breakdown.html > > Robert > > > > On Fri, Mar 5, 2010 at 9:12 PM, PK <p...@gae123.com> wrote: > > The "datastore statistics" is a great concept. However, I consistently > > see that the majority of my space goes to metadata. From other > > postings I see that I am not the only one. Here is an example from my > > application that contains roughly 15,000 entities: > > > Breakdown by Property Type Property Type Size > > Blob 11 MBytes > > String 5 MBytes > > Integer 2 MBytes > > Key 2 MBytes > > NULL 726 KBytes > > Date/Time 705 KBytes > > Text 461 KBytes > > Boolean 13 KBytes > > GeoPt 5 KBytes > > Metadata 683 MBytes > > > The breakdown is really not useful when it breaks down the 3% of the > > space while the 97% of the space is consumed by metadata that I cannot > > drill into!!!!! > > > Based on previous postings on what metadata is, I am wondering, are > > the protocol buffers so inefficient? Can this be related to exploding > > indexes and if yes how is this possible, do metadata account for > > composite indexes as well? Any advice on how to troubleshoot this > > issue? > > > One more question, the size of my entities is reported as 705MB, > > however the dashboard shows that I use 2.31GB. Is the difference > > between these two figures the size of the composite indexes? > > > Thanks > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en.- Hide quoted text - > > - Show quoted text - -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
