Headed into Thursday's meetup, this paper by Jeff Dean provides a very good description of strategies for getting fast response times with variable quality infrastructure.
http://research.google.com/people/jeff/latency.html The key point here is that it is very important to have asynchronous queries with a cancel. Above that level, there needs to be a simple strategy for pushing second versions of queries out to the workers and canceling defunct or redundant queries.
