On 14 Sep, Chuq Von Rospach wrote:
> At 5:56 PM -0700 9/14/98, Geoff Hutchison wrote:
>
>> Sorry, the robots exlusion standard people (and myself) are ahead of you.
>
>> <meta name="robots" content="noindex">
>
>
> Brings up an interesting point I've been sort of researching. What's
> the best way to write up a robots.txt file that allows the local htdig
> to index, but nobody else? I know using a user-agent of htdig should
> allow it to index, but seems to allow remote htdig's also. I guess I
> could disable robot.txt looking in my local copy, but even though I
> trust myself not to abuse that...
>
In theory, you can use a configuration file entry
robotstxt_name: yourhtdig
to make htdig advertise itself as that name. In practice (at least in
my experience) that doesn't seem to work. You could also modify the
default in common/defaults.cc.
Cheers
--
David Robley
WEBMASTER | Phone +61 8 8374 0970
RESEARCH CENTRE FOR INJURY STUDIES | http://www.nisu.flinders.edu.au/
AusEinet | http://auseinet.flinders.edu.au/
Flinders University, ADELAIDE, SOUTH AUSTRALIA
Visit the PHP mirror at http://au.php.net:81/
----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the body of the message.
