Friday October 29 1999 12:59, Geoff Hutchison wrote to Kari Suomela:
>> causing it to be 'skipped'? There is no "robots.txt". What is the
>> reason for looking for one?
GH> There is a standard for search engines. You specify the pages that
GH> can
GH> or cannot be indexed using a file at the root of the server
GH> called 'robots.txt' so this m must be retrieved from all servers.
I found some "info" on it, but no clear instructions as to the contents
of it.
"Disallow: /" obviously does what it says, but how do I allow htdig to
do its stuff?
KS
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word unsubscribe in
the SUBJECT of the message.
