On Friday 19 April 2002 6:55 am, Bill Moseley wrote: > Hi, > > Wasn't there just a thread on throttling a few weeks ago? > > I had a machine hit hard yesterday with a spider that ignored robots.txt.
I thought the standard practice these days was to put some URL at an un-reachable place (by a human), for example using something like <a href="..."></a>. And then ban that via robots.txt. And then automatically update your routing tables for any IP addresses that try and visit that URL. Just a thought, there's probably more to it. Matt.
