Duh! Thanks, Jim. The page wasn't being indexed because of a nofollow meta tag in the page that links to it. I should have tested more thoroughly. ht://Dig is working properly.
Regards, Garrel Renick -----Original Message----- From: Jim Cole Subject: Re: [htdig] Indexing problem with the Robots meta tag On Wednesday, November 20, 2002, at 10:38 AM, Renick, Garrel wrote: > Hello. I'm using ht://Dig 3.1.6. I seem to be having the same problem > reported in 1998 where ht:dig was not indexing pages with the robots > meta tag set to content="index,nofollow": > > http://www.htdig.org/mail/1998/09/0235.html > http://www.htdig.org/mail/1998/09/0303.html > > Are any of you experiencing this same problem with version 3.1.6? > Could there be some other configuration problem I'm overlooking? It seems to work OK for me. Is there a chance that some of the pages that are not being indexed are only reachable through a page with nofollow? If you explicitly add the pages not being indexed to your start_url, are the pages then indexed? Jim ------------------------------------------------------- This SF.net email is sponsored by: The Sourceforge Network Survey Take Our Survey and You Could Win a $500 Gift Certificate! http://ugamsolutions.com/psurvey/osdn/SourceForge/index_sourceforge.htm _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
