Re: [appengine-java] Re: Datastore Statistics vs. Quota
On Tue, Dec 8, 2009 at 1:26 AM, Toby wrote: > Reading the doc carefully it tells me that indexes are created based > on the queries I make. Hence if I never query an entity on a certain > property there should be no index. That's not exactly what it says. Only the development servers "creates" indexes automatically by inserting a few lines in your datastore-indexes-auto.xml. (It doesn't actually create indexes as the development server doesn't need indexes to access data, I believe.) The point of that mechanism is to make it easy for you to know which indexes to request on the production appengine: just look in the datastore-indexes-auto.xml, and copy the lines that you want from it. If so configured, appcfg.sh will copy the entire content of datastore-indexes-auto.xml to the production server. > Also in the app-engine administration there is a menu entry "Indexes" > which only lists two indexes that I am actually using. > > Now if I understand right per default many more indexes are created if > I need them or not. This should be mentioned more clearly because it > could help to save a lot of space. That's it. Most of the indexes space is likely to be coming from implicit indexes. > Is there an JDO annotation to disable indexes on properties or do I > need to disable auto-index? There is: @Persistent @Extension(vendorName = "datanucleus", key = "gae.unindexed", value="true") private String unindexedString; It's in the doc: http://code.google.com/intl/en/appengine/docs/java/datastore/queriesandindexes.html -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Datastore Statistics vs. Quota
Toby, There's another thing to be aware of: sessions are backed with the datastore. If you have sessions enabled, you may want to write a scheduled task that clears session data. On Mon, Dec 7, 2009 at 9:00 AM, Eric Rannaud wrote: > On Mon, Dec 7, 2009 at 2:20 AM, Toby wrote: > > thank you for your update. In fact I was suspecting the index or other > > management data. But it is hard to believe that it leads to such a big > > overhead. I mean it is enormous to have an index that is 10 times more > > than the actual data, don't you think so? > > If you look at the article, it doesn't seem that out of place. > Remember that by default, two indexes are built for every property, > EntitiesByProperty ASC and EntitiesByProperty DESC. If you look at the > number of fields in the corresponding tables (see article), and if > your entity has 5-10 fields, I would not be surprised by such an > overhead. > > Marking properties as non-indexable, if you don't need the systematic > indexing that Google does, will help save a lot of space. > > > Furthermore in the datastore statistics they already list so called > > "Metadata". It is consuming about 44% of the space. I think that this > > is the index, is it not? > > I don't think it is, no. I believe it refers to other kinds of > metadata (see article linked earlier in the thread). Index disk space > usage is, I believe, nowhere explicit. > > Eric. > > -- > > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > > -- Ikai Lan Developer Programs Engineer, Google App Engine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Datastore Statistics vs. Quota
On Mon, Dec 7, 2009 at 2:20 AM, Toby wrote: > thank you for your update. In fact I was suspecting the index or other > management data. But it is hard to believe that it leads to such a big > overhead. I mean it is enormous to have an index that is 10 times more > than the actual data, don't you think so? If you look at the article, it doesn't seem that out of place. Remember that by default, two indexes are built for every property, EntitiesByProperty ASC and EntitiesByProperty DESC. If you look at the number of fields in the corresponding tables (see article), and if your entity has 5-10 fields, I would not be surprised by such an overhead. Marking properties as non-indexable, if you don't need the systematic indexing that Google does, will help save a lot of space. > Furthermore in the datastore statistics they already list so called > "Metadata". It is consuming about 44% of the space. I think that this > is the index, is it not? I don't think it is, no. I believe it refers to other kinds of metadata (see article linked earlier in the thread). Index disk space usage is, I believe, nowhere explicit. Eric. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Datastore Statistics vs. Quota
On Sat, Dec 5, 2009 at 10:05 AM, Eric Rannaud wrote: > They also talked about an article they will publish soon that gives > enough details on how indexes are built that you can at least predict > the size of your indexes. There it is: http://code.google.com/appengine/articles/storage_breakdown.html -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: Datastore Statistics vs. Quota
On Sat, Dec 5, 2009 at 9:29 AM, Toby wrote: > I know that they are updated at least once a day. In my case the data > volume has not changed for quite some time. And the discrepancy is > really huge. I mean 50MB to 500MB. So maybe my datastore contains > stuff that does not apear in the statistics. Or the statistics are > wrong. I wonder if other people find the same ratio. Is there any > other way to get details on the data consumption of the datastore. Do > e.g. log-files count into it or other data? The difference is likely coming from the indexes, both implicit indexes that Google builds systematically, and indexes that you have requested explicitly. When asked about the lack of visibility into indexes disk usage, during the IRC office hours, the Google people said they were working on improving that, and the admin console will likely give you more details about where that extra space is going. It will be very useful, as you can mark properties to be non-indexable, and thus save space. They also talked about an article they will publish soon that gives enough details on how indexes are built that you can at least predict the size of your indexes. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
