Re: [htdig] robotstxt_name

[Geoff Hutchison]

At 9:25 AM -0600 1/7/01, Mike Paradis wrote:
>Is this directive so that htdig will know to look for robots.txt or a custom
>name file that does the same. Or, is this to make our digger look 
>like something else?

It sets the name htdig uses when parsing the robots.txt file and only 
that. So this will affect how htdig considers robot-specific 
directives in the robots.txt file. This is useful, for example, when 
you want to make directives specific to your copy of htdig, etc.

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/


To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
List archives:  
FAQ:

[htdig] robotstxt_name

[Mike Paradis]

In the directives list, this is explained...


robotstxt_name
type:  string
used by: htdig
default: htdig
description: Sets the name that htdig will look for when parsing robots.txt files. This
can be used to make htdig appear as a different spider than ht://Dig. Useful to
distinguish between a private and a global index.
example:  robotstxt_name: myhtdig


However, I am still not clear on this because of the mention os what htdig can be made 
to
look like. Is this directive so that htdig will know to look for robots.txt or a custom
name file that does the same. Or, is this to make our digger look like something else?

Mike






To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
List archives:  
FAQ: