Thanks for the info. I did misunderstand how "robots.txt" worked. I had
confused them with metatags. Never claimed to be an expert . . . just told
our web guy to "minimize our profile" until we have a product.
Faust
>From: Aeon <[EMAIL PROTECTED]>
>
>There's apparently some confusion here about how robots.txt and spidering
>really works.
>
>
> >You're misunderstanding me. I said that I am not putting "D&D", "D20",
>or
> >"OGL" in my robots.txt file (unlike many other D20 sites) or ANYWHERE on
>my
> >commercial pages, AND
>
>robots.txt is an exclusionary file. It's used to list directories and files
>that you do not want spidered. It is not a place to put keywords. That's
>where Meta tags come in. In short, robots.txt is completely irrelevant
>here.
>
>
> >
> >Further, I do not have the text "earth 1066" ANYWHERE on the FAQ page -
>so I
> >am not getting ANY cross-links because of hosting the FAQ.
>
>The FAQ page is located on http://www.earth1066.com/D20FAQ.htm
>Since indexing services include the full URL path in spidering, ALL
>cross-links associated with this FAQ will include mention of earth1066.com
>
>
>
> >
> >OC I'll be spidered ... but the FAQ won't be spidered for "EARTH 1066"
>and
> >MY PAGES won't be spidered for "OGL", "D20", or "OPEN GAMING FOUNDATION".
>
>In fact, your pages will be spidered for all those terms, since the latter
>three are included in the body copy and the first is in the URL. Because
>that's how spiders work.
>
>
>
>
>________________________________________________________________________
> [EMAIL PROTECTED] http://www.aeforge.com
>
_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com
