Hello,
I tried to htdig a certain number of sites, and I had the message : "url
rejected (level 1)".
I know the reason of that : my start_url is
http://www.domain.com/datas/index.html, and this page links to documents whose
the URL is http://www.domain.com/documents/page*.html. So of course, it doesn't
match.
(i didn't exclude any urls, there wasn't any robots.txt)
I used a max hop of 1, and of course, the -i option.
It's not possible to use directly http://www.domain.com as a start_url, because
it would index too much pages.
Is there a solution ?
Regards,
--
Christophe BAEGERT [EMAIL PROTECTED]
>>>>>>> http://biographie.net <<<<<<<
The first biographical search engine
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word unsubscribe in
the SUBJECT of the message.
