I've got several blogs scattered around and use none of them, so I thought I'd post this here and maybe some of the ideas and methods I'm using might help other developers using appengine.
Let me preface this with, I've come to the conclusion that appengine is not the right tool for everyjob, and when you start looking at needing to do the things I mention below on a regular basis, you might wish to consider moving back to a traditional server/database solution, or look for other alternatives to BigTable. I had two problems, I attacked them two different ways to reach acceptable solutions. There was a constraint for both problems that some of you may not be aware of yet, so I'll explain that first. While the datastore has 1000 limit on fetch(), if you're regularly fetching 1000 entities you are going to get execution timeouts. Especially if you're doing something on each of those 1000 requests, like say, deleting them. I've found, when you're doing a write for every request, 75 is for the most part a safe number. Finally, one other piece of information you need is what I'm developing with. The latest version of appenginepatch and the django supplied with it, 1.0.2 I believe. The latest trunk version of gaeutilities (I need to get a new update out to you all who are using it). Ok... problems. Paging. I needed a nice paging system that would also be lightweight and user friendly. I read the very interesting article on paging by key, and it didn't really fit in this instance. keys didn't make for friendly urls and my paging items are static, I page by a score that can change on each entity. My solution was to use the paging library that comes with django. I run my query, and then page on top of it. Since the scores don't change every request, I also cache those queries in order to offer even better performance. I fetch 100 entities for the paging, which gives me 10 pages of 10, when run through the django pagination library. To increase the amount of pages, I could fetch more entities. Deleting more than 1000 records: My first solution was to create a page that deleted 75 records at a time, and refresh a whole lot. This was a bit frustrating when I have of 20000 records. The next time I had to do it, I got a little bit more creative. First, the view in Django: def deleteAllStories(request): ''' This method deletes all stories in the database.''' from google.appengine.ext import db from fanatasticweb.models import Story query = db.GqlQuery('SELECT * FROM Story') results = query.fetch(75) if len(results) == 75: more = True db.delete(results) if more: return HttpResponse('More') else: return HttpResponse('done') This view is actually called from ajax request in this view def deleteAllStoriesV(request): return render_to_response('deleteall.html', {}) deleteall.html is a simple javascript function that calls the story deleteview and checks the content. If the content is not "done" it refreshes. That way, it will keep running even over the occastional datastore timeout error you'll encounter. The main thing I've learned is that when you need to start managing lots of records, it's possible. Javascript is more than likely going to be the answer. Javascript will also be useful for avoiding datastore timeouts in some instances as well. However, when you need to start creating multiple http requests to manage data on a regular basis, it may be time to move back to a server with your favorite database backend in order to be able to process those large transactions. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---