Am 04.07.25 um 12:00 schrieb Gerd Hoffmann:
Basically these AI scrapers do not care about any restrictions like
robots.txt or whatever. They try access all pages and do so with ridiculous
frequency.
I'd name that DoS.
"Do not talk with terrorists" .. block theire entire networks. It's what
we in our datacenter do.
Somethimes they use a Class C for the GET request and another Class C
for the POST request ( in our case a WP cluster )
We stopped blocking ip by ip, we use /24 blocks now.
Theire entire buisness modell is based on our data , so if we stop that
data flow, we hit them in the long term.
best regards,
Marius Schwarz
--
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
