A change was just now made to the shared proxy system for Toolforge which makes the proxy respond with default content for /favicon.ico and /robots.txt when a tool's webservice returns a 404 Not Found response for these files.
The default /favicon.ico is the same as <https://tools-static.wmflabs.org/toolforge/favicons/favicon.ico>. The default robots.txt denies access to all compliant web crawlers. We decided that this "fail closed" approach would be safer than a "fail open" telling all crawlers to crawl all tools. Any tool that does wish to be indexed by search engines and other crawlers can serve their own /robots.txt content. Please see <https://www.robotstxt.org/> for more information on /robots.txt in general. These changes fix a regression [0] in functionality caused by the toolforge.org migration and the introduction of the 2020 Kubernetes ingress layer. Previously the /robots.txt and /favicon.ico from the "admin" tool were served for all tools due to the use of a shared hostname. [0]: https://phabricator.wikimedia.org/T251628 Bryan, on behalf of the Toolforge admin team -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808 _______________________________________________ Wikimedia Cloud Services announce mailing list cloud-annou...@lists.wikimedia.org (formerly labs-annou...@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
