On Mon, Oct 18, 2004 at 06:33:50PM +0100, carl plugtwo wrote: > I think /0 means research the subject of robots.txt > > There's a good FAQ on Robot Exclusion and related topics here: > http://www.robotstxt.org/wc/faq.html
The following article ('No Bots Allowed!'): http://www.eweek.com/article2/0,1759,1248105,00.asp might be of interest as well; it also states the point I was trying to get across: However, the standard relies entirely on the courtesy of the visiting robot. It's completely optional. Nothing prevents robots from simply ignoring the directives in a robots.txt file and many robots do just that. In that sense, a robots.txt file is less like a locked door than a "no entry" sign hanging in an open doorway. Cheers, Michel