[
https://issues.apache.org/jira/browse/NUTCH-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Markus Jelsma resolved NUTCH-2219.
----------------------------------
Resolution: Fixed
Committed to trunk in revision 1731651. Thanks Ron van der Vegt
> Criteria order to be configurable in DeduplicationJob
> -----------------------------------------------------
>
> Key: NUTCH-2219
> URL: https://issues.apache.org/jira/browse/NUTCH-2219
> Project: Nutch
> Issue Type: New Feature
> Components: crawldb
> Reporter: Ron van der Vegt
> Assignee: Markus Jelsma
> Attachments: NUTCH-2219.patch, NUTCH-2219.patch
>
>
> Current implementation:
> "This command takes a path to a crawldb as parameter and finds duplicates
> based on the signature. If several entries share the same signature, the one
> with the highest score is kept. If the scores are the same, then the fetch
> time is used to determine which one to keep with the most recent one being
> kept. If their fetch times are the same we keep the one with the shortest
> URL."
> The order in which the main document is selected is currently not changeable.
> Therefore I think this option would be nice:
> -compareOrder <score>,<fetchTime>,<urlLength>
> I have written a patch on trunk (rev 1730516). I'm looking forward for any
> peer review.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
