Hi,
On 2026-01-10 18:36, Steffen Nurpmeso wrote:
Admins of the wonderful repo.or.cz, would it be possible to create
some local configuration so that downloads of packagers become
possible again?
And if "somehow mystified" on the frontpage, to be updated as
necessary?
How about a useragent "EinMaennleinStehtImWalde"?
I understand your issue, I really do. We didn't set up Anubis because we
hate users or because we want to make downloads impossible or anything.
We did it because we got swamped by crawlers with at least 5 figures
worth of IP addresses distributed across many networks in many
countries,
and so many requests that all of our HTTP workers were fully saturated
and it wasn't even possible to view our (aggressively cached) landing
page anymore. We tried multiple times to relax our constraints but each
time they adapted within days. In fact right now the strictness of the
checks is load-dependent, i.e. the restrictions ramp up whenever we're
getting targeted. So, whenever you're having issues, you know we're
actively being attacked by crawlers in that very moment.
At the time of writing I can use the protected endpoints without any
challenge via e.g. curl, indicating that right now the crawlers are
leaving us alone for the most part, but it keeps coming and going.
For testing purposes I've added a rule that bypasses the restrictions
whenever the User-Agent header contains the string
"I-am-definitely-not-a-crawler" but if the crawlers adapt again, we'll
have to revert that.
Sorry for all the trouble. Let's put the blame where it belongs, though:
the stupid LLM data gold rush and the people who are willing to break
all
the rules to get the tiniest edge.
-Jan (repo.or.cz admin team)
_______________________________________________
Tinycc-devel mailing list
[email protected]
https://lists.nongnu.org/mailman/listinfo/tinycc-devel
