In article <[EMAIL PROTECTED]>, John Nagle <[EMAIL PROTECTED]> wrote:
> I asked over at Webmaster World, and over there, they recommend against > using redirects on robots.txt files, because they questioned whether all of > the major search engines understand that. Does a redirect for > "foo.com/robots.txt" mean that the robots.txt file applies to the domain > being redirected from, or the domain being redirected to? Good question. I'd guess the latter, but it's a little ambiguous. I agree that redirecting a request for robots.txt is probably not a good idea. Given that the robots.txt standard isn't as standard as it could be, I think it's a good idea in general to apply the KISS principle when dealing with things robots.txt-y. -- Philip http://NikitaTheSpider.com/ Whole-site HTML validation, link checking and more -- http://mail.python.org/mailman/listinfo/python-list
