Re: [google-appengine] A difficult app engine optimisation problem - selecting distinct entities across a large table

2011-02-15 Thread Tom Gibara
Hi Edward, (I'm by no means expert, so feel free to discount the following) In my somewhat limited experience, I've identified only three ways of making queries faster: 1. denormalizing data (pushing more work into the writes) 2. deferring work (typically via task queues) 3. admitting

Re: [google-appengine] A difficult app engine optimisation problem - selecting distinct entities across a large table

2011-02-15 Thread Jay
I am going to post two replies as I have a small idea that might help a little bit, and then a further post on the whole problem in general. First, Ed, what you have said is consistent and the problem of identifying the most recent Journey within a constrained time window does appear to imply

Re: [google-appengine] A difficult app engine optimisation problem - selecting distinct entities across a large table

2011-02-15 Thread Jay
I also wanted to add this comment which is probably not helpful, but speaks more to the root of the problem. :) And that is this, you can't get there from here. The fundamental problem does not appear capable of being reduced beyond looking at a bunch of Journeys. I'm going to summarize a

[google-appengine] A difficult app engine optimisation problem - selecting distinct entities across a large table

2011-02-14 Thread Edward Hartwell Goose
Hi, I was wondering if anyone can help me with this problem. We have an idea we'd like to implement, and we're currently unable to do this efficiently. I've anonymised the data as best as possible, but the structure is the same. We have two entities, Car and CarJourney. Each Car has 0 to many