Hey Barbie,

Are you aware that you can set up "crawl-delay" directives in your
robots.txt to request that crawlers hit you less aggressively?

http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive

As far as I know it's obeyed by Bing, but for Googlebot you need to
sign up for Google Webmaster Tools and use their settings.

https://support.google.com/webmasters/answer/48620?hl=en

But to be honest in my experience it's Bing that tends to be
over-aggressive to the point of harming server performance.

- Alex

On 14 January 2015 at 22:59, Barbie <bar...@missbarbell.co.uk> wrote:
> Hi,
>
> Sorry I had to disable access to the site, as Google and the like were
> hitting it quite badly and taking up resources that was slowing the
> reindexing down.
>
> Thanks,
> Barbie.
>
> --
> Birmingham.pm - http://birmingham.pm.org
> CPAN Testers - http://cpantesters.org
> YAPC Surveys - http://yapc-surveys.org
> Perl Jam - http://perljam.info
>
> On Wed, Jan 14, 2015 at 7:50 AM, Slaven Rezic <sla...@rezic.de> wrote:
>>
>> Diab Jerius <djer...@cfa.harvard.edu> writes:
>>
>> > Hi,
>> >
>> > I've been getting a redirect loop from cpantesters.org for the last
>> > few days. I haven't seen any traffic about it here, so I'm wondering
>> > if it's just me.
>>
>> No, everybody was affected. But it works again since today.
>>
>> Regards,
>>     Slaven
>>
>> --
>> Slaven Rezic - slaven <at> rezic <dot> de
>>
>>     Berlin Perl Mongers - http://berlin.pm.org
>
>

Reply via email to