I'm using Solr to search the Nutch Lucene index (can't use the nutch searcher in our current app.) Using the latest Nutch nightly.
There are a lot of duplicate URLs in the Lucene index-- http:// url.com/ vs. http://url.com are two different Lucene documents, as are http://url.com/index.html#a and http://url.com/index.html The Nutch search jsp seems to have some intelligence to remove the duplicates -- with the "show all hits" toggle button at the end of the results. Is there a tool to remove duplicates directly from the Lucene index? I do call 'nutch dedup' in my crawl script but it doesn't seem to affect the results. ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
