[ https://issues.apache.org/jira/browse/NUTCH-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sebastian Nagel updated NUTCH-1693: ----------------------------------- Attachment: NUTCH-1693-2x-v2.patch Hi [~markus17], +1 for your patch (a late review). Updated and completed patch for 2.x: need also to call signature calculation after text has been stored in WebPage. Opened NUTCH-1807 to address the problem with default locales/charsets. > TextMD5Signatue compute on textual content > ------------------------------------------ > > Key: NUTCH-1693 > URL: https://issues.apache.org/jira/browse/NUTCH-1693 > Project: Nutch > Issue Type: New Feature > Reporter: Tien Nguyen Manh > Assignee: Markus Jelsma > Priority: Minor > Fix For: 1.9 > > Attachments: NUTCH-1693-2x-v2.patch, NUTCH-1693-trunk.patch, > NUTCH-1693-trunk.patch, NUTCH-1693.patch > > > I create a new MD5Signature that based on textual content. In our case we use > boilerpipe to extract main text from content so this signature is more > effective to deduplicate. -- This message was sent by Atlassian JIRA (v6.2#6252)