On Wed, 5 Jan 2000, John Caldwell wrote:
> I'm working on a site with some dynamic content, and i want to be able to
> point htdig at an index of pages so that it can spider them, but I don't
> want the main index page to ever show up in the search results. Is there
> way to 'exclude' a file but still follow the links on it?
Sure:
<meta name="robots" content="noindex,follow">
This works with any robot that follows the META robot standard (e.g.
ht://Dig ;-)
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
