On Mon, Oct 18, 2004 at 06:33:50PM +0100, carl plugtwo wrote:
> I think /0 means research the subject of robots.txt
>
> There's a good FAQ on Robot Exclusion and related topics here:
> http://www.robotstxt.org/wc/faq.html
The following article ('No Bots Allowed!'):
http://www.eweek.com/article2/0,1759,1248105,00.asp
might be of interest as well; it also states the point I was trying to
get across:
However, the standard relies entirely on the courtesy of the visiting
robot. It's completely optional. Nothing prevents robots from simply
ignoring the directives in a robots.txt file and many robots do just
that. In that sense, a robots.txt file is less like a locked door than a
"no entry" sign hanging in an open doorway.
Cheers,
Michel
