NG-Marketing, M.Schneider wrote: >> DeleteDuplicates removes documents having the same digest or the same >> url. If you use the TextProfileSigniture instead of MD5Signiture, it >> will remove near similar documents. The MD5Signiture class set digest as >> the md5 of all the content, whereas textProfileSigniture sets digest as >> the md5 of significant terms. You should check the class for >> implementation details. also look at signitureFactory for how to change >> the configuration. >> > > DeleteDuplicates does NOT delete same URLs, it compares only the digest. See > Nutch 371 > http://www.mail-archive.com/[email protected]/msg04635.html >
Erhm, please see the followup here: http://issues.apache.org/jira/browse/NUTCH-371 . This issue is fixed now. -- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
