H Francis,
On Mon, Oct 14, 2013 at 03:23:03PM +0100, Francis Daly wrote:
> In your map, let $is_spider be empty if is not a spider ("default",
> presumably), and be something else if it is a spider (possibly
> $binary_remote_addr if every client should be counted individually,
> or something els
On Mon, Oct 14, 2013 at 01:59:23PM +0200, Toni Mueller wrote:
Hi there,
This is untested, but follows the docs at http://nginx.org/r/limit_req_zone:
> I therefore constructed a map to
> identify spiders, which works well, and then tried to
>
> limit_req_zone $binary_remote_addr zone=slow:10m ..
Hello,
On Mon, Oct 14, 2013 at 09:25:24AM -0400, Sylvia wrote:
> Doesnt robots.txt "Crawl-Delay" directive satisfy your needs?
I have it already there, but I don't know how long it takes for such a
directive, or any changes to robots.txt for that matter, to take effect.
Observing the logs, I'd
Hello.
Doesnt robots.txt "Crawl-Delay" directive satisfy your needs?
Normal spiders should obey robots.txt, if they dont - they can be banned.
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,243670,243674#msg-243674
___
nginx mailing list
ng
Hi,
I would like to put a brake on spiders which are hammering a site with
dynamic content generation. They should still get to see the content,
but only not generate excessive load. I therefore constructed a map to
identify spiders, which works well, and then tried to
limit_req_zone $binary_rem
