Chris Spot on, and I am getting the feeling this is where the value to a geo-ip service comes to play that offers defined "eyeball networks" to allow.
On Wed, Jul 16, 2025 at 12:57 PM Chris Adams via NANOG <[email protected]> wrote: > > Once upon a time, Marco Moock <[email protected]> said: > > Place a link to a file that is hidden to normal people. Exclude the > > directory via robots.txt. > > > > Then use fail2ban to block all IP addresses that poll the file. > > The problem with a lot of the "AI" scrapers is that they're apparently > using botnets and will often only make a single request from a given IP > address, so reactive blocking doesn't work (and can cause other issues, > like trying to block 100,000 IPs, which fail2ban for example doesn't > really handle well). > -- > Chris Adams <[email protected]> > _______________________________________________ > NANOG mailing list > https://lists.nanog.org/archives/list/[email protected]/message/AFJF4UQJZW6ALTY6SA7OHBN2AZC72SZQ/ -- - Andrew "lathama" Latham - _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/[email protected]/message/DHUYTBIXFMWE2KWC5NKCR7AJIWPYUL4E/
