We've had some instances where crawlers have stumbled onto a cgi script which refers to itself and start pounding the server with requests to that cgi.
There are so many CGI scripts on this server that I don't want to maintain a huge robots.txt file. Any suggestions on other techniques to keep crawlers away from cgi scripts? Check the browser with BrowserMatch and then do something creative with "deny from env="? --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: [EMAIL PROTECTED] " from the digest: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
