As Chris states, broad IP based blocking is unlikely to be very effective , and likely more problematic down the line anyway.
For the slightly more 'honorable' crawlers, they'll respect robots.txt, and you can block their UAs there. Fail2ban is a very good option right now. It will be even better if nepenthes eventually integrates with it. Then you can have some real fun. On Wed, Jul 16, 2025 at 3:39 PM Andrew Latham via NANOG < [email protected]> wrote: > Chris > > Spot on, and I am getting the feeling this is where the value to a > geo-ip service comes to play that offers defined "eyeball networks" to > allow. > > On Wed, Jul 16, 2025 at 12:57 PM Chris Adams via NANOG > <[email protected]> wrote: > > > > Once upon a time, Marco Moock <[email protected]> said: > > > Place a link to a file that is hidden to normal people. Exclude the > > > directory via robots.txt. > > > > > > Then use fail2ban to block all IP addresses that poll the file. > > > > The problem with a lot of the "AI" scrapers is that they're apparently > > using botnets and will often only make a single request from a given IP > > address, so reactive blocking doesn't work (and can cause other issues, > > like trying to block 100,000 IPs, which fail2ban for example doesn't > > really handle well). > > -- > > Chris Adams <[email protected]> > > _______________________________________________ > > NANOG mailing list > > > https://lists.nanog.org/archives/list/[email protected]/message/AFJF4UQJZW6ALTY6SA7OHBN2AZC72SZQ/ > > > > -- > - Andrew "lathama" Latham - > _______________________________________________ > NANOG mailing list > > https://lists.nanog.org/archives/list/[email protected]/message/DHUYTBIXFMWE2KWC5NKCR7AJIWPYUL4E/ _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/[email protected]/message/ECB77Z6SVDZZ6SZ5YGWP4YJ5HVX6KRQE/
