did you place robots.txt in the root of the web site hierarchy ? Meaning the url for the robots.txt must be something like http://mywonderfulsite.somewhere.org/robots.txt, and not ttp:// mywonderfulsite.somewhere.org/JSPWiki/robots.txt Did you look at you access.log to see if robots.txt is asked for ?
/Harry 2009/12/5 Robert FORBES <[email protected]> > I don't want Google to index my pages, but I cannot find any means to block > it. As a last ditch effort, I have put the JSPWiki installation behind a > firewall, but this is defeating the purpose for which it is intended. > > I am using Tomcat right now as the app server, and I have a robots.txt file > in the WebApps folder, the folder below that (where Wiki.jsp is), the > WEB_INF folder, and anywhere else I can think of. The robots file looks > like this: > > User-agent: * > Disallow: / > > But it is not working. Am I missing something obvious ? (likely) > > Robert > >
