Hey Barbie, Are you aware that you can set up "crawl-delay" directives in your robots.txt to request that crawlers hit you less aggressively?
http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive As far as I know it's obeyed by Bing, but for Googlebot you need to sign up for Google Webmaster Tools and use their settings. https://support.google.com/webmasters/answer/48620?hl=en But to be honest in my experience it's Bing that tends to be over-aggressive to the point of harming server performance. - Alex On 14 January 2015 at 22:59, Barbie <bar...@missbarbell.co.uk> wrote: > Hi, > > Sorry I had to disable access to the site, as Google and the like were > hitting it quite badly and taking up resources that was slowing the > reindexing down. > > Thanks, > Barbie. > > -- > Birmingham.pm - http://birmingham.pm.org > CPAN Testers - http://cpantesters.org > YAPC Surveys - http://yapc-surveys.org > Perl Jam - http://perljam.info > > On Wed, Jan 14, 2015 at 7:50 AM, Slaven Rezic <sla...@rezic.de> wrote: >> >> Diab Jerius <djer...@cfa.harvard.edu> writes: >> >> > Hi, >> > >> > I've been getting a redirect loop from cpantesters.org for the last >> > few days. I haven't seen any traffic about it here, so I'm wondering >> > if it's just me. >> >> No, everybody was affected. But it works again since today. >> >> Regards, >> Slaven >> >> -- >> Slaven Rezic - slaven <at> rezic <dot> de >> >> Berlin Perl Mongers - http://berlin.pm.org > >
