On Fri, 29 Oct 1999, Kari Suomela wrote:
> This happens only on *my* own site. The others work ok. What is causing
> it to be 'skipped'? There is no "robots.txt". What is the reason for
> looking for one?
There is a standard for search engines. You specify the pages that can or
cannot be indexed using a file at the root of the server called
'robots.txt' so this m must be retrieved from all servers.
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word unsubscribe in
the SUBJECT of the message.
