Hey, we've accumulated a large dataset on our appengine (almost a terrabyte now), and we're struggling to delete it (mostly old data that is no longer needed).
After years of trying this and that, I gave backends a shot, but the behavior is really strange. Essentially, I've got a little deleter script set up like this: backends: - name: deleter class: B2 start: backends/deleter.py instances: 1 In the backends/deleter.py I then do: import logging logging.info("Starting deleter...") import datetime import common now = datetime.datetime.now() old_game_date = now+common.MAXIMUM_GAME_AGE logging.info("Finding games before %s"%old_game_date) from models import Game from google.appengine.ext import db q = db.Query(Game.Game, keys_only=True) q.filter("date <", old_game_date) games = q.fetch(500) count = 0 while games and len(games)>0: db.delete(games) games = q.fetch(500) count += len(games) logging.info("Done, deleted %s games."%count) This produces the following log message: A serious problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may have a memory leak in your application. (Error code 201) However, if I stop the deletion after a few games, like if count > 1000: break It actually works: 2011-09-03 05:13:51.386 Starting deleter... I 2011-09-03 05:13:51.390 Finding games before 2011-08-04 12:13:51.390348 I 2011-09-03 05:14:01.034 Done, deleted 1500 games. Also, the logservice.flush() doesn't seem to work at all, I was getting the same error I'm getting above and _no_ output at all, even if i only had a simple script like: from google.appengine.api import logservice import logging logging.info("Starting deleter...") logservice.flush() So something seems to be up in the land of the backends. Please advise. Cheers, Volker PS: our appengine id is "towermadness", the backend is "deleter" -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.