There's apparently some confusion here about how robots.txt and spidering
really works.
>You're misunderstanding me. I said that I am not putting "D&D", "D20", or
>"OGL" in my robots.txt file (unlike many other D20 sites) or ANYWHERE on my
>commercial pages, AND
robots.txt is an exclusionary file. It's used to list directories and files
that you do not want spidered. It is not a place to put keywords. That's
where Meta tags come in. In short, robots.txt is completely irrelevant here.
>
>Further, I do not have the text "earth 1066" ANYWHERE on the FAQ page - so I
>am not getting ANY cross-links because of hosting the FAQ.
The FAQ page is located on http://www.earth1066.com/D20FAQ.htm
Since indexing services include the full URL path in spidering, ALL
cross-links associated with this FAQ will include mention of earth1066.com
>
>OC I'll be spidered ... but the FAQ won't be spidered for "EARTH 1066" and
>MY PAGES won't be spidered for "OGL", "D20", or "OPEN GAMING FOUNDATION".
In fact, your pages will be spidered for all those terms, since the latter
three are included in the body copy and the first is in the URL. Because
that's how spiders work.
________________________________________________________________________
[EMAIL PROTECTED] http://www.aeforge.com
