Author: ivan Date: Sat Oct 31 15:08:21 2026 New Revision: 1939806 Log: In site/staging: Follow-up to r1939750: Fix list of Google crawlers that ignore '*' rule.
Found by: dsahlberg * robots-noindex.txt: Add AdsBot-Google and Mediapartners-Google User-Agent rules. Remove duplicate address for AdsBot-Google-Mobile. Modified: subversion/site/staging/robots-noindex.txt Modified: subversion/site/staging/robots-noindex.txt ============================================================================== --- subversion/site/staging/robots-noindex.txt Sat Oct 31 14:43:30 2026 (r1939805) +++ subversion/site/staging/robots-noindex.txt Sat Oct 31 15:08:21 2026 (r1939806) @@ -9,17 +9,8 @@ Disallow: / User-Agent: AdsBot-Google-Mobile Disallow: / -User-Agent: AdsBot-Google-Mobile -Disallow: / - -User-Agent: AdsBot-Google-Mobile +User-Agent: AdsBot-Google Disallow: / -User-Agent: AdsBot-Google-Mobile -Disallow: / - -User-Agent: AdsBot-Google-Mobile -Disallow: / - -User-Agent: AdsBot-Google-Mobile +User-Agent: Mediapartners-Google Disallow: /
