Hi Hraban, On Mon, 2026-01-05 at 17:10 +0100, Henning Hraban Ramm wrote: > Other scraping solutions, e.g. via wget, are blocked ATM by Anubis. > (AI bots killing servers otherwise.) > Maybe we could allow the IPs of the mirror servers.
The default configuration of Anubis only blocks things with "Mozilla"* in their user-agent, so "wget" should work fine. Indeed, I tested it right now, and I was able to "wget" the main page of the Wiki without any issues. But it is fairly easy to configure Anubis to whitelist IPs, so that's always an option too. (Taco, let me know if you want some pointers on how to do this) Thanks, -- Max *: All browsers include "Mozilla" in their user-agent for weird historical reasons, so this affects (anything pretending to be) Chrome/IE/Edge/Safari too, not just Firefox. ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : [email protected] / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl webpage : https://www.pragma-ade.nl / https://context.aanhet.net (mirror) archive : https://github.com/contextgarden/context wiki : https://wiki.contextgarden.net ___________________________________________________________________________________
