Shailesh Kochhar wrote:
I not very familiar with the Nutch API though I know there's a MD5 signature based deduping method in place and a Signature class to extend for offline duplicate detection. I was wondering if anyone had tried search time deduping and what would be good places to try and implement it.
Nutch already does search-time deduping. By default it limits things to two hits per host, but you can dedup by other fields and with other per-dup counts. This is available through NutchBean:
http://lucene.apache.org/nutch/apidocs/org/apache/nutch/searcher/NutchBean.html#search(org.apache.nutch.searcher.Query,%20int,%20int,%20java.lang.String) and though the OpenSearch servlet. Doug ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
