[appengine-java] Re: problem with Datastore quotas
When is Google intending to fix this major problem? I am in the red with the amount of stored data caused by the _ah_session object, which should be cleaned up by the servlet container. As you seem to know, the SessionCleanupServlet fails when the dataset is too big, which is my case. When can developers expect a fix for this problem? On Feb 2, 9:15 am, Dombrowski, Todd t...@pzlbox.com wrote: Nevermind -- looks like a wide spread problem, being discussed on the appengine list already ... On Tue, Feb 2, 2010 at 9:07 AM, Dombrowski, Todd t...@pzlbox.com wrote: Most likely indexes -- app engine maintains an index for every query the app engine will perform (from Programming Google App Engine) If you have a variety of queries that use multiple properties and sort orders, the datastore will need an index for every unique combination -- and this is in addition to the indexes already created on every key and property for a kind. Sort order is significant too -- you may need an index in ASC and DESC order for a property. Another thing to look out for: using more than one multi-valued property in an entity. Your index on those properties basically becomes the cross product of the values stored, which can make it huge. If you don't have it, the book Programming Google App Engine has the best treatment I've seen of the datastore. Thanks, Todd Dombrowski pzlbox On Tue, Feb 2, 2010 at 5:50 AM, Jorge athenas...@gmail.com wrote: Hi Dimitar, This is an idea. Take a closer look to your _ah_SESSION kind. It is possible you are storing huge session data and it remains there after the sessions expire. If that is the case, you probably want to clean your expired sessions periodically. Jorge Gonzalez On Feb 1, 1:05 pm, dmakariev ani...@gmail.com wrote: Hi again, I've read the documentation. and cannot find anything related to my datastore growth :(, I'm not using indexes. It is quite simple application, demonstrating the usage of JSF 2. I'm using session, and I've included the SessionCleanupServlet in my web.xml. It is started every hour. So the number of entities in _ah_SESSION is round 40 . My other suspect was the log file. I couldn't find any description about where the log files are stored. But I've updated the version number of the application.. deleted the old one with the big log files, and still the total used datastore quota didn't decrease. my current status is 35% of Total Stored Data and at the same time : Size of all entities 414 KBytes I'll be really happy to know where and how to manage properly the datastore space. Best Regards: Dimitar Makariev On Feb 1, 1:38 am, John Patterson jdpatter...@gmail.com wrote: On the page that shows those details is a link to a doc that explains where the extra space is used. Probably indexes could consume a lot. On 1 Feb 2010, at 16:29, dmakariev wrote: Hi all, I have the following problem : my quotas show the following : Total Stored Data 26% 0.26 of 1.00 GBytes Okay at the same time Datastore Statistics is showing : Size of all entities: 5 MBytes Breakdown by Property Type Property Type Size Blob 5 MBytes String 14 KBytes Integer 12 KBytes Date/Time 4 KBytes User 51 Bytes NULL 24 Bytes Metadata 71 KBytes And I have no idea where is the datastore space disappearing.. my app id is sandcode Does anybody has idea ? I'll appreciate your advices.. Best Regards: Dimitar Makariev -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com . To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2B unsubscr...@googlegroups.com . For more options, visit this group athttp:// groups.google.com/group/google-appengine-java?hl=en . -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2B unsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at
[appengine-java] Re: problem with Datastore quotas
Hi Dimitar, This is an idea. Take a closer look to your _ah_SESSION kind. It is possible you are storing huge session data and it remains there after the sessions expire. If that is the case, you probably want to clean your expired sessions periodically. Jorge Gonzalez On Feb 1, 1:05 pm, dmakariev ani...@gmail.com wrote: Hi again, I've read the documentation. and cannot find anything related to my datastore growth :(, I'm not using indexes. It is quite simple application, demonstrating the usage of JSF 2. I'm using session, and I've included the SessionCleanupServlet in my web.xml. It is started every hour. So the number of entities in _ah_SESSION is round 40 . My other suspect was the log file. I couldn't find any description about where the log files are stored. But I've updated the version number of the application.. deleted the old one with the big log files, and still the total used datastore quota didn't decrease. my current status is 35% of Total Stored Data and at the same time : Size of all entities 414 KBytes I'll be really happy to know where and how to manage properly the datastore space. Best Regards: Dimitar Makariev On Feb 1, 1:38 am, John Patterson jdpatter...@gmail.com wrote: On the page that shows those details is a link to a doc that explains where the extra space is used. Probably indexes could consume a lot. On 1 Feb 2010, at 16:29, dmakariev wrote: Hi all, I have the following problem : my quotas show the following : Total Stored Data 26% 0.26 of 1.00 GBytes Okay at the same time Datastore Statistics is showing : Size of all entities: 5 MBytes Breakdown by Property Type Property Type Size Blob 5 MBytes String 14 KBytes Integer 12 KBytes Date/Time 4 KBytes User 51 Bytes NULL 24 Bytes Metadata 71 KBytes And I have no idea where is the datastore space disappearing.. my app id is sandcode Does anybody has idea ? I'll appreciate your advices.. Best Regards: Dimitar Makariev -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com . To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com . For more options, visit this group athttp://groups.google.com/group/google-appengine-java?hl=en . -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: problem with Datastore quotas
Most likely indexes -- app engine maintains an index for every query the app engine will perform (from Programming Google App Engine) If you have a variety of queries that use multiple properties and sort orders, the datastore will need an index for every unique combination -- and this is in addition to the indexes already created on every key and property for a kind. Sort order is significant too -- you may need an index in ASC and DESC order for a property. Another thing to look out for: using more than one multi-valued property in an entity. Your index on those properties basically becomes the cross product of the values stored, which can make it huge. If you don't have it, the book Programming Google App Engine has the best treatment I've seen of the datastore. Thanks, Todd Dombrowski pzlbox On Tue, Feb 2, 2010 at 5:50 AM, Jorge athenas...@gmail.com wrote: Hi Dimitar, This is an idea. Take a closer look to your _ah_SESSION kind. It is possible you are storing huge session data and it remains there after the sessions expire. If that is the case, you probably want to clean your expired sessions periodically. Jorge Gonzalez On Feb 1, 1:05 pm, dmakariev ani...@gmail.com wrote: Hi again, I've read the documentation. and cannot find anything related to my datastore growth :(, I'm not using indexes. It is quite simple application, demonstrating the usage of JSF 2. I'm using session, and I've included the SessionCleanupServlet in my web.xml. It is started every hour. So the number of entities in _ah_SESSION is round 40 . My other suspect was the log file. I couldn't find any description about where the log files are stored. But I've updated the version number of the application.. deleted the old one with the big log files, and still the total used datastore quota didn't decrease. my current status is 35% of Total Stored Data and at the same time : Size of all entities 414 KBytes I'll be really happy to know where and how to manage properly the datastore space. Best Regards: Dimitar Makariev On Feb 1, 1:38 am, John Patterson jdpatter...@gmail.com wrote: On the page that shows those details is a link to a doc that explains where the extra space is used. Probably indexes could consume a lot. On 1 Feb 2010, at 16:29, dmakariev wrote: Hi all, I have the following problem : my quotas show the following : Total Stored Data 26% 0.26 of 1.00 GBytes Okay at the same time Datastore Statistics is showing : Size of all entities: 5 MBytes Breakdown by Property Type Property Type Size Blob5 MBytes String 14 KBytes Integer 12 KBytes Date/Time 4 KBytes User51 Bytes NULL24 Bytes Metadata71 KBytes And I have no idea where is the datastore space disappearing.. my app id is sandcode Does anybody has idea ? I'll appreciate your advices.. Best Regards: Dimitar Makariev -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com . To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group athttp:// groups.google.com/group/google-appengine-java?hl=en . -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.comgoogle-appengine-java%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: problem with Datastore quotas
Hi again, I've read the documentation. and cannot find anything related to my datastore growth :(, I'm not using indexes. It is quite simple application, demonstrating the usage of JSF 2. I'm using session, and I've included the SessionCleanupServlet in my web.xml. It is started every hour. So the number of entities in _ah_SESSION is round 40 . My other suspect was the log file. I couldn't find any description about where the log files are stored. But I've updated the version number of the application.. deleted the old one with the big log files, and still the total used datastore quota didn't decrease. my current status is 35% of Total Stored Data and at the same time : Size of all entities 414 KBytes I'll be really happy to know where and how to manage properly the datastore space. Best Regards: Dimitar Makariev On Feb 1, 1:38 am, John Patterson jdpatter...@gmail.com wrote: On the page that shows those details is a link to a doc that explains where the extra space is used. Probably indexes could consume a lot. On 1 Feb 2010, at 16:29, dmakariev wrote: Hi all, I have the following problem : my quotas show the following : Total Stored Data 26% 0.26 of 1.00 GBytes Okay at the same time Datastore Statistics is showing : Size of all entities: 5 MBytes Breakdown by Property Type Property Type Size Blob 5 MBytes String 14 KBytes Integer 12 KBytes Date/Time 4 KBytes User 51 Bytes NULL 24 Bytes Metadata 71 KBytes And I have no idea where is the datastore space disappearing.. my app id is sandcode Does anybody has idea ? I'll appreciate your advices.. Best Regards: Dimitar Makariev -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com . To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com . For more options, visit this group athttp://groups.google.com/group/google-appengine-java?hl=en . -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
