Hi Erik, As far as I know the persistence manager would not create any extra entities.
It sounds like you might want to explicitly disable indexing on any fields you will not be querying on. See the 'Properties that Aren't Indexed' section on the 'Queries and Indexes' page. http://code.google.com/appengine/docs/java/datastore/queriesandindexes.html You might also want to use shorter kind and property names. Kind and property names are stored with every entity, so it can add up pretty fast. Robert On Tue, Nov 9, 2010 at 08:05, Erik <erik.e.wil...@gmail.com> wrote: > Hi Robert, > > Thanks for the response, I did wait several days before sending that > message, but shortly afterwards the quota cleared to zero. I am using > a persistence manager with jdo, would these create temporary > entities? > > I need to explicitly index my keys in descending order for mapreduce, > and thought I needed to index my collections but as I am not > performing any queries don't think that is necessary. My persistent > objects consist of a single property with type of long collection, > entities are fetched by id. > > My latest 30MB dataset has already grown to consume 710MB, that is an > enormous increase. Datastore statistics say only 131MB is being used > with 71% of that as metadata. > > Any other ideas? > > Thanks, > -Erik > > On Nov 8, 1:03 pm, Robert Kluin <robert.kl...@gmail.com> wrote: >> Hi Erik, >> Several common sources of datastore stats / quota number funkiness: >> 1) The numbers are not updated in real time. Sometimes it can >> take a day for the numbers to get updated. >> 2) Because of 1, if you are using a session library (or something >> similar) that create lots of temporary entities the numbers can >> 'appear' to be out-of-sync. >> >> I assume when you say switched to explicit indexes, you mean you >> explicitly disabled indexing of any properties that do not need >> indexes? >> >> Also, you might want to star issue 2740. >> http://code.google.com/p/googleappengine/issues/detail?id=2740 >> >> Robert >> >> On Sat, Nov 6, 2010 at 19:29, Erik <erik.e.wil...@gmail.com> wrote: >> >> > Hello all, >> >> > I have been uploading a dataset which is composed of 100MB of CSV >> > values. During the process of uploading with bulkloader, which never >> > completed, the datastore expanded to consume 2GB of usage. I decided >> > to explicitly index my data and clear the datastore before a re-import >> > using the blobstore/mapreduce method. However, after clearing the >> > datastore there is still a persistent .37GB of data usage remaining, >> > while datastore statistics say 53KB is being consumed by 174 objects, >> > and nothing shows in datastore viewer. >> >> > Any ideas on how I can recover the .37GB of quota which is >> > mysteriously being used in my empty datastore? >> >> > Many thanks, >> > -Erik >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to google-appeng...@googlegroups.com. >> > To unsubscribe from this group, send email to >> > google-appengine+unsubscr...@googlegroups.com. >> > For more options, visit this group >> > athttp://groups.google.com/group/google-appengine?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
