On 21 Jun 2011, at 10:44, Martin Hepp wrote: > PS: I will not release the IP ranges from which the trouble originated, but > rest assured, there were top research institutions among them.
The right answer is: name and shame. That is the way to teach them. Like Karl said, we should collect information about abusive crawlers so that site operators can defend themselves. It won't be *that* hard to research and collect the IP ranges of offending universities. I started a list here: http://www.w3.org/wiki/Bad_Crawlers The list is currently empty. I hope it stays that way. Thank you all, Richard
