On 15 Sep, Geoff Hutchison wrote:
>
>>Brings up an interesting point I've been sort of researching. What's
>>the best way to write up a robots.txt file that allows the local htdig
>>to index, but nobody else? I know using a user-agent of htdig should
>>allow it to index, but seems to allow remote htdig's also. I guess I
>>could disable robot.txt looking in my local copy, but even though I
>>trust myself not to abuse that...
>
> Well you can set the robotstxt_name that htdig uses when it parses the
> robots.txt file. So try something like:
> robotstxt_name: plaidworks-htdig
>
> Then use "plaidworks-htdig" in your robots.txt file for your "local" htdig
> and you're set.
>
Shold we then also expect to see 'plaidworks-htdig' as the user-agent in
the log file?
Cheers
-- David Robley
WEBMASTER | Phone +61 8 8374 0970
RESEARCH CENTRE FOR INJURY STUDIES | http://www.nisu.flinders.edu.au/
AusEinet | http://auseinet.flinders.edu.au/
Flinders University, ADELAIDE, SOUTH AUSTRALIA
Visit the PHP mirror at http://au.php.net:81/
----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the body of the message.
