@Brandon : This is the case for one of my popular web-sites too. I believe if latency time increases, Google bot will automatically scale down its crawl rate and I also fear that such an increased latency will have a negative effect on the page ranking.
By the way, have you started using 304 (Not Modified) responses, for pages which have not modified ? This can reduce the resource usage by Google Bot. Worst case option is to resort to a check for every Nth request (10 < N < 100) from the bot (of course, using memcache) and send a 503 status.. *503 Service Unavailable * The server is currently unavailable (because it is overloaded or down for maintenance). Generally, this is a temporary state. ..and this should also give a hint to the Google bot to scale down its crawl rate. This will be useful when you want to retain better latency and want to hint Google bot alone. Again, this can have an effect on the page rank too. And I am not sure which is worse bad latency or a 503. Anyone else has experience in this space ? Thanks & Regards, Raj On Sun, Sep 4, 2011 at 2:52 AM, Darien Caldwell <darien.caldw...@gmail.com>wrote: > Nice, so basically Google is using their own service to tack on > additional charges to your bill. Doesn't sound ethical. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Build online database applications, over Google App Engine. iFreeTools Creator - http://creator.ifreetools.com -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
