* Constantine A. Murenin [Wed 02 Jul 2025, 05:23 CEST]:
But the bots are not a problem if you're doing proper caching and throttling.
Have you been following the news at all lately? Website operators are
complaining left and right about the load from scrapers related to AI
companies. They're seeing 10x, 100x the normal visitor load, with not
just User-Agents but also source IP addresses masked to present as
regular visitors. Captchas is unfortunately one of the more visible
ways to address this, even if not perfect.
For example,
https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/
-- Niels.
_______________________________________________
NANOG mailing list
https://lists.nanog.org/archives/list/[email protected]/message/L5WNOGOAOZRJYR5BAEWSGOD7SPDKKX32/
