The blogs.gnome.org uses a strange robots.txt <http://blogs.gnome.org/robots.txt> permitting only Googlebot to crawl it. This prevents using archive.org on the subdomain and cripples other search engines. It should simply use a wildcard for the first User-agent field with the second case discarded.
Thanks!
signature.asc
Description: OpenPGP digital signature
_______________________________________________ gnome-web-list mailing list [email protected] https://mail.gnome.org/mailman/listinfo/gnome-web-list
