Thérèse Godefroy <godef...@free.fr> writes:
> Hello, > > I was searching for an article with DuckDuckGo, and guess what appeared > on top of the results... > https://lists.gnu.org/archive/html/www-commits/2023-05/msg00082.html > and > https://lists.gnu.org/archive/html/www-commits/2023-05/msg00062.html !! > > I understand it wouldn't be practical to make this list private, but it > could at least be off-limits for crawlers. > > Thanks! > > Thérèse robots.txt isn't about just not making things public, it is perfectly valid to use for SEO and helping google turn up the best results for your site and avoiding non-canonical duplicate pages. I've seen www-commits list archive pages come up in searches before and they were not what I wanted. In that light, this request sounds like a good idea to me. I will implement it. We can change it back if webmasters agree otherwise. -- Ian Kelling | Senior Systems Administrator, Free Software Foundation GPG Key: B125 F60B 7B28 7FF6 A2B7 DF8F 170A F0E2 9542 95DF https://fsf.org | https://gnu.org