Hello People !!!
How can I tell htdig to *ignore* the robots.txt-files, on the whole web or
on specified servers ?
That's my problem:
title: ReferateFundus
href: http://www.fundus.org/index1.htm ()
resolving 'http://www.fundus.org/index1.htm'
pushing http://www.fundus.org/index1.htm
+href: http://www.fundus.org/indexrechts.htm ()
resolving 'http://www.fundus.org/indexrechts.htm'
pushing http://www.fundus.org/indexrechts.htm
+A tag: pos = 2, position =
="http://www.fundus.org/cgi/ref_anz.cgi?Biographien">
href: http://www.fundus.org/cgi/ref_anz.cgi?Biographien (Biographien [290])
Rejected: Item in the exclude list: item # 2 length: 4
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^EXLCUDE LIST ?!?
How can i turn this exlcude list *OFF* ?!?
Thank you,
Gunther Stammwitz
url rejected: (level 1)http://www.fundus.org/cgi/ref_anz.cgi?Biographien
A tag: pos = 2, position =
="http://www.fundus.org/cgi/ref_anz.cgi?Biologie">
href: http://www.fundus.org/cgi/ref_anz.cgi?Biologie (Biologie [238])
Rejected: Item in the exclude list: item # 2 length: 4
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the SUBJECT of the message.
