Does anyone know if there is anyway to prevent ht://dig from adding HTML files to the Index, but still parse them for links? Basically, I want ht://dig to index .doc, .pdf and .xls files on a site, but not the .html (or actually, .php) files. But it still needs to parse these pages as the links to the .doc/.pdf/.xls files are linked from the .php pages.
The documentation says it obeys the standard rules for ROBOTS metadata:
http://www.robotstxt.org/wc/meta-user.html
and I have no cause to doubt that it does just that!
regards, Malcolm.
------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ ht://Dig general mailing list: <[EMAIL PROTECTED]> ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-general
