Thanks very much all. We've had discussions with our client and have opted to make a short term fix to this problem by changing the requirements (the easy option ;) ).
So, this means that we're not going to be trying to solve this problem over the next 1-2 weeks. But we will be looking into it after that. I'm also going to see if I can tell you who the client (and thus the real problem) is as well, as your help has been invaluable. Tom & Jay - thanks very much for your input too. I think there are some great ideas in there! I'll also ensure that when we do come to a solution - I'll post it for all to see! If anyone does have any more ideas, do continue posting, it's been really interesting to get everyone's perspective! Cheers, Ed On Feb 16, 4:15 am, Jay <jbaker.w...@gmail.com> wrote: > I also wanted to add this comment which is probably not helpful, but speaks > more to the root of the problem. :) > And that is this, "you can't get there from here". The fundamental problem > does not appear capable of being reduced beyond looking at a bunch of > Journeys. I'm going to summarize a couple of things here, but in the end it > just comes down to look at a bunch of data ... unless you change something > else. > > A quick summary of some points: > > 1. You can de-normalize Car fields to Journey as it relates to this search. > So the problem does reduce, slightly, to looking for Journeys. The "two" > lookup aspect of it should not be much of a problem. I know this was > mentioned above. > > 2. You can try to squash aspects of your search into the Key as Stephen > suggested. This is a good idea but it does not change the nature of the > problem - looking at a whole lot of entities, if only the keys of those > entities. > > These other possibilities are excluded from being viable because of the > problem description. > > 3. Make the processing more parallel by turning it into a counting problem > and using a map reduce approach or similar. This is the same as changing the > requirements from "query" to "report" in practical terms. There is also the > sticky issue of having a reduce implementation on app engine right now. > > 4. Changing the search criteria to be more restricted in some way that > introduces a further relationship between Journeys. This additional > information could probably be used to make the search perform better. In > fact, one would engineer it in such a way, there would probably be no need > to make a change in the way this search worked unless it resulted in this > benefit. > > 5. There is also the "break the fourth wall" kind of solution. Take this > part of the problem off of app engine. You would pay a lot more for this. > Both in terms of complexity and cost (and maybe some other things too). It > may not be realistic, but how realistic is the alternative? I hope you will > report back at some stage and tell us. > (this might allow for a truly parallel query for example ... come on > Datastore Plus!) > > What I am getting at is that the system probably needs a higher level change > to achieve the desired goal given the constraints that exist at this time. > That is the "you can't get there from here". I think this has been virtually > proven at this point though I would be happy to be proved wrong. It comes to > trade-offs like everything. You can often trade space for speed, etc. etc. > But it is often a trade. So you may need to look at trades between the > existing use cases / user expectation and speed. > > Thanks Ed for sharing with the group. This has been a good discussion and > interesting to think about. > > -- Jay -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
