There is a way to limit what pages are indexed in htdig, using meta
tags. I propose an additional flag that says "collect URLs from this
page, but do NOT index this page."
I'd use this for mailing list archives to avoid indexing the list
index page, but to gather all the contained URLs.
--Michael
----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the body of the message.
