Wayne Sherman via fpc-pascal <[email protected]> schrieb am Do., 16. Apr. 2026, 17:35:
> The FPC wiki website is also blocked from archive.org. Note the > latest capture for this page: > > https://web.archive.org/web/20250909050155/https://wiki.freepascal.org/Release_engineering That is a side effect of Anubis and could be adjusted by whitelisting the crawler from the Archive. Are these pages also blocked from Google search and other search > engines? I am unable to check that but someone with write access to > the web server can check and monitor the status of Google's web > crawler indexes. See here: > https://support.google.com/webmasters/answer/9012289?hl=en Ordinary search engine crawlers are already whitelisted by Anubis. Question: > "What will the long term consequences be of blocking any of the Free > Pascal (fpc) websites (i.e. the main website, the forum, and the wiki) > from AI / LLM tools, archive.org, and web search engines? How will > users and potential new users of Free Pascal be affected? Summarize in > one paragraph." > It's not relevant what a Stochastic Parrot thinks about this. What is important is the reality: if we're *not* blocking the Ai crawlers then the Wiki will be down for *everyone*. Before we added Anubis as a protection the wiki was often not accessible by anyone due to the high amount of traffic added by Ai crawlers. You can see this with the forum which regularly receives high amounts of traffic due to crawlers and then isn't reachable until one of the admins blocks a wide range of IPs used by those crawlers. And no, a CDN like Cloudflare is not a solution we are willing to use because that is it's own can of worms. Regards, Sven >
_______________________________________________ fpc-pascal maillist - [email protected] https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
