And what would be wrong if google was to constrain their bots to visit GAE pages only once per week?
Or even better, what if GAE had a mechanism by which the apps could announce when they have something new to be crawled? On Sep 4, 12:07 am, "Brandon Wirtz" <drak...@digerat.com> wrote: > Returning a 503 is REALLY REALLY Bad for SEO. > > 304 Seems to be ignored by Google Bot on GAE, but also Google Bot will try > queries to which there are no links, and which no user has ever made. > > If your latency goes higher Google Bot will throttle back. but the only way > to slow down the page serving is to put a wait timer, which burns Instance > time. Rob Paul to Pay Peter. > > From: google-appengine@googlegroups.com > [mailto:google-appengine@googlegroups.com] On Behalf Of Rajkumar > Radhakrishnan > Sent: Saturday, September 03, 2011 6:41 PM > To: google-appengine@googlegroups.com > Subject: Re: [google-appengine] Re: Google Bot Is Your Enemy > > @Brandon : > > This is the case for one of my popular web-sites too. I believe if latency > time increases, Google bot will automatically scale down its crawl rate and > I also fear that such an increased latency will have a negative effect on > the page ranking. > > By the way, have you started using 304 (Not Modified) responses, for pages > which have not modified ? This can reduce the resource usage by Google Bot. > > Worst case option is to resort to a check for every Nth request (10 < N < > 100) from the bot (of course, using memcache) and send a 503 status.. > > 503 Service Unavailable > > The server is currently unavailable (because it is overloaded or down for > maintenance). Generally, this is a temporary state. > > ..and this should also give a hint to the Google bot to scale down its crawl > rate. This will be useful when you want to retain better latency and want to > hint Google bot alone. Again, this can have an effect on the page rank too. > And I am not sure which is worse bad latency or a 503. > > Anyone else has experience in this space ? > > Thanks & Regards, > > Raj > > On Sun, Sep 4, 2011 at 2:52 AM, Darien Caldwell <darien.caldw...@gmail.com> > wrote: > > Nice, so basically Google is using their own service to tack on > additional charges to your bill. Doesn't sound ethical. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com > <mailto:google-appengine%2bunsubscr...@googlegroups.com> . > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Build online database applications, over Google App Engine. > > iFreeTools Creator -http://creator.ifreetools.com > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
