I think the biggest thing for everyone to remember is that each delete of an entity also requires updating all indexes for that entity. So for 1 to 2 million entities that's a lot of work especially with transactions, concurrency, etc. I'm not sure that marking the entity for deletion and then having some background task later come through actually saves anything because all that work still needs to be done. Also, would those entities still show up in queries. If they shouldn't show up then you'd have to go and mark all index entries as deleted which is just as much work as deleting them in the first place. I would say however that perhaps there should be the equivalent of a DROP TABLE so that you can get rid of an entire entity kind. That could definitely be done in the background using bulk delete methods and would immediately remove that entity kind from being visible and used and as such there wouldn't be the need to worry about concurrency and transaction isolation etc.
On Sun, Nov 21, 2010 at 12:10 AM, Derrick Schneider < derrick.schnei...@gmail.com> wrote: > Thanks for the reminder about the thread. I had read the initial couple of > posts but hadn't noticed the follow-ups. > > My procedure is pretty straightforward, and is similar to the "fast" one > described: Kick off a task that does a keys-only query for n items and then > deletes them (using the form where you just pass a list of keys), grabs the > cursor, and kicks off another task that pulls in the cursor and gets n more, > and so on. (I forget the value of n.) > > I don't know exactly how many items were being deleted, but I've got ~7.5 > million entries in total, spread from July to today. Probably our largest > number of entries actually comes from August, but I was just deleting July. > So I guess "few hundred thousand" isn't accurate, and it was probably closer > to 1-2 million. Hm. > > But reading that Bulk Delete thread, I agree it would be nice to mark an > item as deletable and have some cheaper, belly-of-Google task sweep through > them. > Derrick > > On Sat, Nov 20, 2010 at 8:55 PM, Stephen Johnson > <onepagewo...@gmail.com>wrote: > >> Read post Bulk Deletion Woe that was posted this past week. It discusses >> this. You should add what procedure you did to delete your entities, how >> many etc. >> >> On Sat, Nov 20, 2010 at 9:21 PM, Derrick Schneider < >> derrick.schnei...@gmail.com> wrote: >> >>> One thing I've noticed as I'm purging older items in the datastore is >>> that deletes are really CPU intensive. Granted, I was probably deleting a >>> few hundred thousand entries, but I went from well under my free CPU >>> threshold for the day to burning out the rest of our budgeted 14.5 hours >>> over the course of about 40 minutes. >>> >>> Has anyone else noticed this? >>> >>> Derrick >>> >>> -- >>> Writer. Programmer. Puzzle Designer. >>> http://www.obsessionwithfood.com >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to google-appeng...@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >>> . >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appeng...@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > > > -- > Writer. Programmer. Puzzle Designer. > http://www.obsessionwithfood.com > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com<google-appengine%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
