Thanks for all the responses, everybody. Responses below -- On May 9, 5:45 am, Tim Hoffman <zutes...@gmail.com> wrote: > 1. Is the set of entities changing constantly - ie adding or deleting > entities, or are you just changing a fixed set of entities.
Yes, the set of entities is changing constantly, and it's mostly adding/changing entities. > 2. How up to date must your view be, It's OK if it's a few seconds out of date, but not much longer than that is acceptable. > 3. Is it feasible to refetch/populate your cache via a background > task (if you write 20-30 times per read) Yes, this is something I've considered doing, but given my answer to the previous question (reads have to be fresh within a few seconds), that suggests the background task would have to run after each write, or regularly every few seconds or something -- and that seems wasteful in terms of App Engine's CPU usage. > Also I assume you do a fetch on the query rather than iterating of the > result (fetch is much quicker). Oh, we could be onto something here. I'm iterating over the result, not using fetch, because I was under the impression fetch() was only for a limited number of records. Are you saying that I could pass a dummy huge number to fetch(), like fetch(1000000), and it would be faster than iterating? If so, that would be an easy win. > I think if you adding/removing keys you could actually keep a list in > of keys in memcache as a single entity > then fetch the list of keys then do a db.get(list_of_keys) assuming > you have less 1000 entities as you are still limited to 1000's items > retrieved. This is a good idea, too -- thanks. I'll give it a shot. On May 9, 8:16 am, djidjadji <djidja...@gmail.com> wrote: > Maybe you can use the client side of the process. > Write a piece of Javascript that uses Ajax calls to perform multiple > requests simultaneous (Ajax=asynchronous) that each retrieve part of > the objects (there should be some range on the 'the_order' field, chop > it in pieces). Return XML or Jason for the objects found. > Combine the results of these queries and construct the webpage client > side or by sending the whole bunch to the server in another Ajax call. I ended up doing this shortly after I sent my original post to the group -- sharding the query, essentially, and reassembling the results on the client side -- and it's made a huge improvement in responsiveness. This technique is going to be an important part of my final solution. On May 9, 9:15 am, djidjadji <djidja...@gmail.com> wrote: > I don't think retrieving 1000 or 2000 objects/rows from an SQL > database is considerably faster then the distributed approach of > Bigtable. In my earlier version of the app, before App Engine, I used SQLite, and it vastly outperformed BigTable for the same query. It was on the order of 1000 times faster. Of course, that was with only a couple thousand rows in the SQLite database, and I'm sure performance would've gotten worse as the database grew. That seems to be a benefit of BigTable -- the performance stays generally the same no matter how many records you stuff in there... (Right...? I'm basing this assumption on what I've read.) On May 9, 6:16 pm, johnP <j...@thinkwave.com> wrote: > Another trick is to > convert objects to protobufs when saving and retrieving. Nick's blog > has an article on this: > http://blog.notdot.net/2009/9/Efficient-model-memcaching Thanks for the heads-up on that article -- hadn't seen it. > Also, do you need the entire entities? Or just some data from the > entities? You might be able to distill the essential information into > a list before putting to memcache. This would decrease the serialize/ > deserialize cost that Nick writes about. I need the entire entities, unfortunately. Wow, as I describe all of my requirements to you folks, this really does sound like a nightmare app, doesn't it? :-) Thanks again, everybody, Adrian -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
