Hi Alex, I am aware of 'crawl-delay', but are you aware they all abuse it too? It was why Google and Microsoft were banned from the sites up until last year. Google will only honour it for a short period and then revert back to their aggressive tactics, though Microsoft are far worse. You have to remember that both companies do not have one and only one crawler, they have a dozen which attack at equal rates, and don't talk to each other to see whether they are launching a denial of service.
I have no sympathy for either company with regards to getting themselves blocked. Thanks, Barbie. -- Birmingham.pm - http://birmingham.pm.org CPAN Testers - http://cpantesters.org YAPC Surveys - http://yapc-surveys.org Perl Jam - http://perljam.info On Thu, Jan 15, 2015 at 9:57 AM, Alex Balhatchet <kao...@gmail.com> wrote: > Hey Barbie, > > Are you aware that you can set up "crawl-delay" directives in your > robots.txt to request that crawlers hit you less aggressively? > > > http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive > > As far as I know it's obeyed by Bing, but for Googlebot you need to > sign up for Google Webmaster Tools and use their settings. > > https://support.google.com/webmasters/answer/48620?hl=en > > But to be honest in my experience it's Bing that tends to be > over-aggressive to the point of harming server performance. > > - Alex > > On 14 January 2015 at 22:59, Barbie <bar...@missbarbell.co.uk> wrote: > > Hi, > > > > Sorry I had to disable access to the site, as Google and the like were > > hitting it quite badly and taking up resources that was slowing the > > reindexing down. > > > > Thanks, > > Barbie. > > > > -- > > Birmingham.pm - http://birmingham.pm.org > > CPAN Testers - http://cpantesters.org > > YAPC Surveys - http://yapc-surveys.org > > Perl Jam - http://perljam.info > > > > On Wed, Jan 14, 2015 at 7:50 AM, Slaven Rezic <sla...@rezic.de> wrote: > >> > >> Diab Jerius <djer...@cfa.harvard.edu> writes: > >> > >> > Hi, > >> > > >> > I've been getting a redirect loop from cpantesters.org for the last > >> > few days. I haven't seen any traffic about it here, so I'm wondering > >> > if it's just me. > >> > >> No, everybody was affected. But it works again since today. > >> > >> Regards, > >> Slaven > >> > >> -- > >> Slaven Rezic - slaven <at> rezic <dot> de > >> > >> Berlin Perl Mongers - http://berlin.pm.org > > > > >
