Hi,
The next steps I'd like to do is to redirect anyone not in that RequireAll
statement to be redirected to the production site. Is this possible? Perhaps a
RewriteCond that depends upon certain IPs, then otherwise redirects to the
production site?
I don't think relying on the IPs is a good idea, since those will
change, and the proper process to validate them requires 2 DNS
lookups, if I'm not mistaken. Just use a rewriteCond + rewriteRule to
generously check the User-Agent and perform the redirect. You may have
to set an environment variable in the rewrite rule and check that in
your RequireAll statement to permit the 301 response to be sent. You
may want to verify that the Vary:User-Agent response header gets sent
to the client to prevent cache pollution.
I'm back to trying to work on this, and hoped you could assist further.
Is this along the lines of what I should be doing?
SetEnvIf user-agent "(?i:Googlebot)" stayout=1
RewriteCond %{HTTP_USER_AGENT} Googlebot
RewriteRule (.*) https://linuxsecurity.com$1 [E=stayout:1]
I'm also not sure about the Vary:User-Agent - we are using cloudflare,
but that appears related to triggering googlebot to also scan as another
user agent, such as its mobile bot?
dave
