Oops. Perhaps I should've articulated a little more.
I don't want to use META robots since that will prevent Search Engines from indexing the site. I thought about creating a custom htdig-robots.txt file specifically for htdig. Regards, Rupert. -----Original Message----- From: Malcolm Austen [mailto:[EMAIL PROTECTED] Sent: 25 March 2004 15:28 To: Rupert Jones Cc: [EMAIL PROTECTED] Subject: Re: [htdig] Prevent indexing of HTML files? On 25/03/2004 14:58, Rupert Jones wrote: > Does anyone know if there is anyway to prevent ht://dig from adding HTML > files to the Index, but still parse them for links? > Basically, I want ht://dig to index .doc, .pdf and .xls files on a site, but > not the .html (or actually, .php) files. But it still needs to parse these > pages as the links to the .doc/.pdf/.xls files are linked from the .php > pages. The documentation says it obeys the standard rules for ROBOTS metadata: http://www.robotstxt.org/wc/meta-user.html and I have no cause to doubt that it does just that! regards, Malcolm. ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ ht://Dig general mailing list: <[EMAIL PROTECTED]> ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-general
