Hello Mathias, Mathias Walter wrote:
I don't know it exactly. The problem is that the sites are linked from anywhere. I'm not sure, if a crawler that follows the link http://mydomain:port/servlet/page.jsp, looks for the robots.txt in the ROOT webapp.
Just last week we've installed a robots.txt where none was before and it took more than 24 hours until most bots (GoogleBot, MsnBot, Yahoo Slurp) re-read it again. Yes, Bots are supposed to read the file /robots.txt when accessing /some/thing/here.jsp . If it's a malicious bot, you're out of luck anyway.
HTH, - Markus --------------------------------------------------------------------- To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
