<quote who="Russell Coker"> > > Why don't you just update your robots.txt to explicitly specify which > > files you don't or do, allow spiders access to. If it's a rule-obiding > > spider, that will be the end of it. > > I wasn't aware that there was any format to robots.txt, I thought that the > mere presense of such a file would prevent robots from visiting.
http://www.searchtools.com/robots/robots-txt.html - Jeff -- "Funny, I have no trouble distinguishing my mobile phone from the others because it's in my _own fucking pocket_!" - Mobile Rage -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
