Hi! I think I would have tried to return a "please crawl elsewhere!" message to these crawlers instead. What if the crawlers use an account now?
Kind regards, Ulrich Windl > -----Original Message----- > From: Users <[email protected]> On Behalf Of Reid Wahl > Sent: Tuesday, November 18, 2025 1:51 AM > To: Cluster Labs - All topics related to open-source clustering welcomed > <[email protected]> > Subject: [EXT] [ClusterLabs] Login is required for now on > projects.clusterlabs.org > > > A while back, we made tasks, commits, and source files on > projects.clusterlabs.org publicly accessible by default. > > Lately we've been inundated with HTTP requests from AI crawler bots > (from many IP addresses) that ignore robots.txt. To mitigate the > effects, we've set the permissions to "All Users" for the apps that > access tasks, commits, and source files. This means that no matter > what the permissions are for any particular object, you are now > required to log in before you can view it. > > It's easy to create an account, and you don't need any additional > permissions to view most things once you're logged in. > > We will probably continue requiring a login to view commits and source > code. I can't think of any reason those need to be publicly visible on > projects.clusterlabs.org, since they're already visible on GitHub. > > We hope to make the tasks/bug reports visible without a login again in > the future. > > -- > Regards, > > Reid Wahl (He/Him) > Senior Software Engineer, Red Hat > RHEL High Availability - Pacemaker > > _______________________________________________ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ _______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
