Hey,
we've accumulated a large dataset on our appengine (almost a terrabyte
now), and we're struggling to delete it (mostly old data that is no
longer needed).
After years of trying this and that, I gave backends a shot, but the
behavior is really strange.
Essentially, I've got a little deleter script set up like this:
backends:
- name: deleter
class: B2
start: backends/deleter.py
instances: 1
In the backends/deleter.py I then do:
import logging
logging.info("Starting deleter...")
import datetime
import common
now = datetime.datetime.now()
old_game_date = now+common.MAXIMUM_GAME_AGE
logging.info("Finding games before %s"%old_game_date)
from models import Game
from google.appengine.ext import db
q = db.Query(Game.Game, keys_only=True)
q.filter("date <", old_game_date)
games = q.fetch(500)
count = 0
while games and len(games)>0:
db.delete(games)
games = q.fetch(500)
count += len(games)
logging.info("Done, deleted %s games."%count)
This produces the following log message:
A serious problem was encountered with the process that handled this
request, causing it to exit. This is likely to cause a new process to
be used for the next request to your application. If you see this
message frequently, you may have a memory leak in your application.
(Error code 201)
However, if I stop the deletion after a few games, like
if count > 1000:
break
It actually works:
2011-09-03 05:13:51.386
Starting deleter...
I 2011-09-03 05:13:51.390
Finding games before 2011-08-04 12:13:51.390348
I 2011-09-03 05:14:01.034
Done, deleted 1500 games.
Also, the logservice.flush() doesn't seem to work at all, I was
getting the same error I'm getting above and _no_ output at all, even
if i only had a simple script like:
from google.appengine.api import logservice
import logging
logging.info("Starting deleter...")
logservice.flush()
So something seems to be up in the land of the backends. Please
advise.
Cheers,
Volker
PS: our appengine id is "towermadness", the backend is "deleter"
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
