[EMAIL PROTECTED] wrote: > On Thu, 23 Mar 2006 15:15:00 GMT, Dave Korn said: > > > difference? robots.txt is enforced (or ignored) by the client. If a > > server > > returns a 403 or doesn't, depending on what UserAgent you specified, > > then > > how could making the client ignore robots.txt somehow magically make the > > server not return a 403 when you try to fetch a page? > > It *can*, however, make the client *issue* a request it would otherwise > not have. > > If the client had snarfed a robots.txt, and it said "don't snarf anything > under /dontlook/here/", and a link pointed there. it wouldn't follow the > link. > >If you tell it 'robots=off', then it *would* follow the link.
Yes, these are all extremely obvious truisms, but I think now you need to go back and read the thread, because you haven't noticed that they're utterly irrelevant to the matter at hand. > Remember - robots.txt *isn't* for the pages that would 403. See, thing is, "pages that would 403" is /exactly/ what we were talking about. So saying "switch off robots.txt" is a completely irrelevant response. And the fact that doing so _would_ have /an/ effect in /other/ circumstances doesn't make it any less irrelevant, at least not according to any definition of the word "relevant" that I've ever seen! cheers, DaveK -- Can't think of a witty .sigline today.... _______________________________________________ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/