On 7 Sep, simon wrote:
> Greetings from downunder,
>
> Is there a way to turn the robot exclusions protocol off.
>
> We have no robots tags to stop external bots on some pages but want
> htdig to go in.
>
> Thanks for all the work that has gone into this project.
>
> Cheers,
> Simon
>
You'll find a description of how to do this at
http://info.webcrawler.com/mak/projects/robots/norobots.html
Essentially, you need a section in robots.txt that gives permissions to
your htdig robot which aren't available to others. Example:
#Keep out the nasties
User-agent: *
Disallow: whatever
# Let htdig at everything
User-agent: htdig (or whatever you have called it)
Disallow:
Cheers
--
David Robley
WEBMASTER | Phone +61 8 8374 0970
RESEARCH CENTRE FOR INJURY STUDIES | http://www.nisu.flinders.edu.au/
AusEinet | http://auseinet.flinders.edu.au/
Flinders University, ADELAIDE, SOUTH AUSTRALIA
Visit the PHP mirror at http://au.php.net:81/
----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the body of the message.
