I've got several blogs scattered around and use none of them, so I
thought I'd post this here and maybe some of the ideas and methods I'm
using might help other developers using appengine.
Let me preface this with, I've come to the conclusion that appengine
is not the right tool for everyjob, and when you start looking at
needing to do the things I mention below on a regular basis, you might
wish to consider moving back to a traditional server/database
solution, or look for other alternatives to BigTable.
I had two problems, I attacked them two different ways to reach
acceptable solutions.
There was a constraint for both problems that some of you may not be
aware of yet, so I'll explain that first. While the datastore has 1000
limit on fetch(), if you're regularly fetching 1000 entities you are
going to get execution timeouts. Especially if you're doing something
on each of those 1000 requests, like say, deleting them. I've found,
when you're doing a write for every request, 75 is for the most part a
safe number.
Finally, one other piece of information you need is what I'm
developing with.
The latest version of appenginepatch and the django supplied with it,
1.0.2 I believe.
The latest trunk version of gaeutilities (I need to get a new update
out to you all who are using it).
Ok... problems.
Paging.
I needed a nice paging system that would also be lightweight and user
friendly. I read the very interesting article on paging by key, and it
didn't really fit in this instance. keys didn't make for friendly urls
and my paging items are static, I page by a score that can change on
each entity.
My solution was to use the paging library that comes with django. I
run my query, and then page on top of it. Since the scores don't
change every request, I also cache those queries in order to offer
even better performance. I fetch 100 entities for the paging, which
gives me 10 pages of 10, when run through the django pagination
library. To increase the amount of pages, I could fetch more entities.
Deleting more than 1000 records:
My first solution was to create a page that deleted 75 records at a
time, and refresh a whole lot. This was a bit frustrating when I have
of 20000 records. The next time I had to do it, I got a little bit
more creative.
First, the view in Django:
def deleteAllStories(request):
''' This method deletes all stories in the database.'''
from google.appengine.ext import db
from fanatasticweb.models import Story
query = db.GqlQuery('SELECT * FROM Story')
results = query.fetch(75)
if len(results) == 75:
more = True
db.delete(results)
if more:
return HttpResponse('More')
else:
return HttpResponse('done')
This view is actually called from ajax request in this view
def deleteAllStoriesV(request):
return render_to_response('deleteall.html', {})
deleteall.html is a simple javascript function that calls the story
deleteview and checks the content. If the content is not "done" it
refreshes. That way, it will keep running even over the occastional
datastore timeout error you'll encounter.
The main thing I've learned is that when you need to start managing
lots of records, it's possible. Javascript is more than likely going
to be the answer. Javascript will also be useful for avoiding
datastore timeouts in some instances as well. However, when you need
to start creating multiple http requests to manage data on a regular
basis, it may be time to move back to a server with your favorite
database backend in order to be able to process those large
transactions.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~----------~----~----~----~------~----~------~--~---
