I would settle for plays by the same rules as all other websites. Though I do kind of think that the bandwidth consumed by GOOG on GOOG infrastructure should be free.
-----Original Message----- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of zdravko Sent: Sunday, September 04, 2011 12:50 AM To: Google App Engine Subject: [google-appengine] Re: Google Bot Is Your Enemy And what would be wrong if google was to constrain their bots to visit GAE pages only once per week? Or even better, what if GAE had a mechanism by which the apps could announce when they have something new to be crawled? On Sep 4, 12:07 am, "Brandon Wirtz" <drak...@digerat.com> wrote: > Returning a 503 is REALLY REALLY Bad for SEO. > > 304 Seems to be ignored by Google Bot on GAE, but also Google Bot will > try queries to which there are no links, and which no user has ever made. > > If your latency goes higher Google Bot will throttle back. but the > only way to slow down the page serving is to put a wait timer, which > burns Instance time. Rob Paul to Pay Peter. > > From: google-appengine@googlegroups.com > [mailto:google-appengine@googlegroups.com] On Behalf Of Rajkumar > Radhakrishnan > Sent: Saturday, September 03, 2011 6:41 PM > To: google-appengine@googlegroups.com > Subject: Re: [google-appengine] Re: Google Bot Is Your Enemy > > @Brandon : > > This is the case for one of my popular web-sites too. I believe if > latency time increases, Google bot will automatically scale down its > crawl rate and I also fear that such an increased latency will have a > negative effect on the page ranking. > > By the way, have you started using 304 (Not Modified) responses, for > pages which have not modified ? This can reduce the resource usage by Google Bot. > > Worst case option is to resort to a check for every Nth request (10 < > N < > 100) from the bot (of course, using memcache) and send a 503 status.. > > 503 Service Unavailable > > The server is currently unavailable (because it is overloaded or down > for maintenance). Generally, this is a temporary state. > > ..and this should also give a hint to the Google bot to scale down its > crawl rate. This will be useful when you want to retain better latency > and want to hint Google bot alone. Again, this can have an effect on the page rank too. > And I am not sure which is worse bad latency or a 503. > > Anyone else has experience in this space ? > > Thanks & Regards, > > Raj > > On Sun, Sep 4, 2011 at 2:52 AM, Darien Caldwell > <darien.caldw...@gmail.com> > wrote: > > Nice, so basically Google is using their own service to tack on > additional charges to your bill. Doesn't sound ethical. > > -- > You received this message because you are subscribed to the Google > Groups "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com > <mailto:google-appengine%2bunsubscr...@googlegroups.com> . > For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. > > -- > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > ~~~~ > > Build online database applications, over Google App Engine. > > iFreeTools Creator -http://creator.ifreetools.com > > -- > You received this message because you are subscribed to the Google > Groups "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
