Here's all of the code from the presentation:
http://code.google.com/events/io/sessions/OfflineProcessingAppEngine.html
class Counter(db.Model):
count = db.IntegerProperty(indexed=False)
class CounterHandler(webapp.RequestHandler):
def post(self):
key = self.request.get('key')
if (memcache.incr(key) is None and
not memcache.add(key, 1)):
memcache.incr(key)
if memcache.add(key + '_dirty', 1):
taskqueue.add(url='/worker',
params={'key': key})
class PageHitWorker(webapp.RequestHandler):
def post(self):
key = self.request.get('key')
memcache.delete(key + '_dirty'):
value = memcache.get(key)
if value is None:
logging.error('Failure for %s', key)
return
Counter(key_name=key, count=value).put()
One thing is that is bothering me is the ':' after memcache.delete in
PageHitWorker. I wonder if an 'if' statement is missing to only update
the counter entity when in fact was dirty. Most likely it's just an
extra character because nothing is indented and the main issue is to
make sure you don't spin more than one task at a time. Moving on.
Brett talks about this solution when you're willing to sacrifice
"near" real-time resolution down to whatever the queue configuration
might be. But you're also are sacrificing the possibility of data loss
because this method assumes that you'll never lose that key from
memcache.
I've begun writing another example that increments the entity counter
instead of replacing it. This way, at most, I'll only lose the current
value of my memcache key/counter. For that, I now have to wrap the
worker in a transaction and also memcache.decr the counter by the
amount added to the entity counter. But I wanted to hear what others
are doing now that sharded counters has been deemed too expensive.
I'm trying to keep several counters per-entity so I'm thinking this
whole thing can get out of hand too quickly. I was then thinking of an
alternative approach using memcache. What if I keep a counter in
memcache as a log index (using memcache.incr) and insert my requests
as name_{index_from_memcache} then simply have the task queue read
from memcache in batches and do all of the processing and counting
within a single transaction/entity.
Any thoughts?
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~----------~----~----~----~------~----~------~--~---
