Hi, Is there a reason to repeat AdsBot-Google-Mobile?
If I understand the linked page correctly, we should block AdsBot-Google and Mediapartners-Google as well. Kind regards, Daniel Den fre 30 okt. 2026 kl 17:10 skrev <[email protected]>: > Author: ivan > Date: Fri Oct 30 16:08:09 2026 > New Revision: 1939750 > > Log: > In site/staging: > > Explicitly block some Google crawlers that ignore '*' rule [1] > > [1]: > https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers > > * robots-noindex.txt: Add explicit User-Agent sections. > > Modified: > subversion/site/staging/robots-noindex.txt > > Modified: subversion/site/staging/robots-noindex.txt > > ============================================================================== > --- subversion/site/staging/robots-noindex.txt Fri Oct 30 16:04:11 2026 > (r1939749) > +++ subversion/site/staging/robots-noindex.txt Fri Oct 30 16:08:09 2026 > (r1939750) > @@ -1,2 +1,25 @@ > User-Agent: * > Disallow: / > + > +# Explicitly block some Google crawlers that ignore '*' rule [1] > +# [1]: > https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers > +User-Agent > <https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers+User-Agent>: > APIs-Google > +Disallow: / > + > +User-Agent: AdsBot-Google-Mobile > +Disallow: / > + > +User-Agent: AdsBot-Google-Mobile > +Disallow: / > + > +User-Agent: AdsBot-Google-Mobile > +Disallow: / > + > +User-Agent: AdsBot-Google-Mobile > +Disallow: / > + > +User-Agent: AdsBot-Google-Mobile > +Disallow: / > + > +User-Agent: AdsBot-Google-Mobile > +Disallow: / > >
