[
https://issues.apache.org/jira/browse/NUTCH-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16980217#comment-16980217
]
Sebastian Nagel commented on NUTCH-2275:
----------------------------------------
The problem is that the "feed" plugin emits one document for each feed item
without the need to actually fetch the URL/page of the item. There should be no
signature at all for yet unfetched URLs in the CrawlDb. Everything done on the
signature - deduplication, not-modified recognition - does not make sense.
> MD5Signature by default doesn't take in account parse
> -----------------------------------------------------
>
> Key: NUTCH-2275
> URL: https://issues.apache.org/jira/browse/NUTCH-2275
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.11
> Reporter: Francesco Capponi
> Priority: Major
> Fix For: 1.17
>
>
> I'm testing Apache Nutch with the feed's plugin. I've noticed that for each
> page it generates the same digest/signature, therefore the dedup cleans
> everything up from the database.
> I'm wondering why the class MD5Signature is the default one instead of
> TextMD5Signature.
> Anyhow now I've modified a little bit the MD5Signature to let it work with
> the feed plugin
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
