[ https://issues.apache.org/jira/browse/NUTCH-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-2219: --------------------------------- Attachment: NUTCH-2219.patch Thanks, looks fine! Slightly updated patch: * changed usage output * changed url to urlLength for clarity Will commit sometime soon unless objections > Dedup script, allow users to change the order in which main documents are > selected. > ----------------------------------------------------------------------------------- > > Key: NUTCH-2219 > URL: https://issues.apache.org/jira/browse/NUTCH-2219 > Project: Nutch > Issue Type: New Feature > Components: crawldb > Reporter: Ron van der Vegt > Assignee: Markus Jelsma > Attachments: NUTCH-2219.patch, NUTCH-2219.patch > > > Current implementation: > "This command takes a path to a crawldb as parameter and finds duplicates > based on the signature. If several entries share the same signature, the one > with the highest score is kept. If the scores are the same, then the fetch > time is used to determine which one to keep with the most recent one being > kept. If their fetch times are the same we keep the one with the shortest > URL." > The order in which the main document is selected is currently not changeable. > Therefore I think this option would be nice: > -compareOrder <score>,<fetchTime>,<url> > I have written a patch on trunk (rev 1730516). I'm looking forward for any > peer review. -- This message was sent by Atlassian JIRA (v6.3.4#6332)