On Tue, 15 Sep 1998 [EMAIL PROTECTED] wrote:
> On 15 Sep, Geoff Hutchison wrote:
> >
> >>Brings up an interesting point I've been sort of researching. What's
> >>the best way to write up a robots.txt file that allows the local htdig
> >>to index, but nobody else? I know using a user-agent of htdig should
> >>allow it to index, but seems to allow remote htdig's also. I guess I
> >>could disable robot.txt looking in my local copy, but even though I
> >>trust myself not to abuse that...
> >
> > Well you can set the robotstxt_name that htdig uses when it parses the
> > robots.txt file. So try something like:
> > robotstxt_name: plaidworks-htdig
> >
> > Then use "plaidworks-htdig" in your robots.txt file for your "local" htdig
> > and you're set.
> >
>
> Shold we then also expect to see 'plaidworks-htdig' as the user-agent in
> the log file?
>
I don't know the answer to that... I will let Geoff answer.
I don't truly understand why there is a user_agent and robotstxt_name.
I know that if you want to set a Robot Name on your machine you have to
define the
user_agent: RomanticRobot
Cheers,
Maren.
> Cheers
> -- David Robley
>
> WEBMASTER | Phone +61 8 8374 0970
> RESEARCH CENTRE FOR INJURY STUDIES | http://www.nisu.flinders.edu.au/
> AusEinet | http://auseinet.flinders.edu.au/
> Flinders University, ADELAIDE, SOUTH AUSTRALIA
> Visit the PHP mirror at http://au.php.net:81/
>
> ----------------------------------------------------------------------
> To unsubscribe from the htdig mailing list, send a message to
> [EMAIL PROTECTED] containing the single word "unsubscribe" in
> the body of the message.
>
----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the body of the message.
