On Wed, 31 Jan 2001, Faustus von Goethe wrote:
>Thanks for the info. I did misunderstand how "robots.txt" worked. I had
>confused them with metatags. Never claimed to be an expert . . . just told
>our web guy to "minimize our profile" until we have a product.
No problem. I only recently started incorporating robots.txt files into my
websites because I got sick of them coming up as missing files in web stat
logs.
Some good sites to check out in regards to how they work and how to write
them are as follows:
http://www.rietta.com/robogen/intro.shtml
http://www.searchtools.com/robots/robots-txt.html
http://www.pageresource.com/zine/robotstxt.htm
Thus, to exclude a specific page, you would create a file that loks like
this:
User-agent: *
Disallow: /pathname/filename.html
