oh man.. well, he's going to be wiping out 7GB of junk... :) When I went through process of deleting something like 400MB of junk.. it was not fun....
First I started off deleting by __key__ in batches of 500, then I had to limit down to 200.. then down to 100.. then down to 50.. then down to 10.. then it stopped responding for hours (I could not even fetch(1) from the Model). There must be a sanctioned way to remove 100,000s of entities based on how the datastore is structured. For example, does it make sense to do something like this. Use a cursor to: 1. Select __key__ from Model Order By __key__ 2. append every 10th (or 100th) result to a list.. and delete that list for every 100 or 200 or 500 entities added. 3. Once at end of cursor, start over at the beginning. That way, you wouldn't be deleting everything on the same table at the same time? The datastore completely died on me when I tried to straight delete by __key__ using GqlQuery in a loop.. Just kept getting slower and slower. (I think maybe directly deleting by key_name might work better but I never had to do a bulk delete again.. so have not tested that theory). On Mon, Mar 22, 2010 at 5:19 PM, Patrick Twohig <patr...@namazustudios.com>wrote: > I'd use a cursor on the task queue. Do bulk deletes in blocks of 500 (I > think that's the most keys you can pass to delete on a single call) and it > shouldn't be that hard to wipe it out. > > Cheers! > > > On Mon, Mar 22, 2010 at 1:45 PM, homunq <jameson.qu...@gmail.com> wrote: > >> OK, after hashing it out on IRC, I see that I have to erase my data >> and start again. Since it took me 3 days of CPU quota to add the data, >> I want to know if I can erase it quickly. >> >> 1. Is the overhead for erasing data (and thus whittling down indexes) >> over half the overhead from adding it? Under 10%? Or what? (I don't >> need exact numbers, just approximates. >> >> 2. If it's more like half - is there some way to just nuke all my data >> and start over? >> >> Thanks, >> Jameson >> >> >> On 22 mar, 03:42, "Nick Johnson (Google)" <nick.john...@google.com> >> wrote: >> > Hi, >> > >> > The discrepancy between datastore stats volume and stored data is >> generally >> > due to indexing overhead, which is not included in the datastore stats. >> This >> > can be very high for entities with many properties, or with long entity >> and >> > property names or entity keys. Do you have reason to suppose that's not >> the >> > case in your situation? >> > >> > -Nick Johnson >> > >> > >> > >> > >> > >> > On Sun, Mar 21, 2010 at 3:39 AM, homunq <jameson.qu...@gmail.com> >> wrote: >> > > Something is wrong. My app is showing with 7.42GB of total stored >> > > data, but only 615 MB of datastore. There is only one version string >> > > uploaded, which is almost 150MB, and nothing in the blobstore. This >> > > discrepancy has been getting worse - several hours ago (longer than >> > > the period since datastore statistics were updated, if you're >> > > wondering), there were the same 615 MB in the datastore, and only >> > > 3.09GB of "total stored data". (at that time, my theory was that it >> > > was old uploads of tweaks to the same "version" - but the numbers have >> > > gone far, far beyond that explanation now.) It's not some exploding >> > > index; the only non-default index I have is on an entity type with >> > > just 33 entities. >> > >> > > Here's the line from my dashboard: >> > > Total Stored Data $0.005/GByte-day 82% 7.42 >> of >> > > 9.00 GBytes >> > > $0.04 / $0.04 >> > >> > > And here is the word from my datastore statistics: >> > > Last updated Total number of entities Size of all entities >> > > 1:32:13 ago 232,867 615 MBytes >> > > (metadata 11%, if that matters) >> > >> > > Please, can someone help me figure out this issue? I'd be happy to >> > > share any info or code which would help track this down. My app id is >> > > vulahealth. >> > >> > > -- >> > > You received this message because you are subscribed to the Google >> Groups >> > > "Google App Engine" group. >> > > To post to this group, send email to >> google-appeng...@googlegroups.com. >> > > To unsubscribe from this group, send email to >> > > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com><google-appengine%2Bunsubscrib >> e...@googlegroups.com> >> > > . >> > > For more options, visit this group at >> > >http://groups.google.com/group/google-appengine?hl=en. >> > >> > -- >> > Nick Johnson, Developer Programs Engineer, App Engine >> > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration >> Number: >> > 368047 >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appeng...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > > > -- > Patrick H. Twohig. > > Namazu Studios > P.O. Box 34161 > San Diego, CA 92163-4161 > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
