Author: ivan
Date: Mon Nov 2 10:05:07 2026
New Revision: 1939942
Log:
In site/publish:
Merge the outstanding changes from staging (Add robots-noindex.txt for staging
branches).
Added:
subversion/site/publish/robots-noindex.txt
- copied unchanged from r1939941,
subversion/site/staging/robots-noindex.txt
Modified:
subversion/site/publish/ (props changed)
subversion/site/publish/.htaccess
Modified: subversion/site/publish/.htaccess
==============================================================================
--- subversion/site/publish/.htaccess Mon Nov 2 09:40:12 2026
(r1939941)
+++ subversion/site/publish/.htaccess Mon Nov 2 10:05:07 2026
(r1939942)
@@ -19,3 +19,8 @@ RewriteRule ^/?download(\.html)?/?$ /dow
<If "%{SERVER_NAME} != 'subversion.apache.org'">
Header set X-Robots-Tag "noindex, nofollow"
</If>
+
+# Use robots-noindex.txt for /robots.txt for all VirtualHost except
+# the primary one to exclude them from indexing.
+RewriteCond expr "%{SERVER_NAME} != 'subversion.apache.org'"
+RewriteRule ^robots.txt(\?.*)?$ robots-noindex.txt$1 [PT]
Copied: subversion/site/publish/robots-noindex.txt (from r1939941,
subversion/site/staging/robots-noindex.txt)
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ subversion/site/publish/robots-noindex.txt Mon Nov 2 10:05:07 2026
(r1939942, copy of r1939941, subversion/site/staging/robots-noindex.txt)
@@ -0,0 +1,16 @@
+User-Agent: *
+Disallow: /
+
+# Explicitly block some Google crawlers that ignore '*' rule [1]
+# [1]:
https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers
+User-Agent: APIs-Google
+Disallow: /
+
+User-Agent: AdsBot-Google-Mobile
+Disallow: /
+
+User-Agent: AdsBot-Google
+Disallow: /
+
+User-Agent: Mediapartners-Google
+Disallow: /