[ https://issues.apache.org/jira/browse/LUCENE-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Harwood updated LUCENE-6747: --------------------------------- Attachment: fingerprintv1.patch Proposed implementation and test > FingerprintFilter - a TokenFilter for clustering/linking purposes > ----------------------------------------------------------------- > > Key: LUCENE-6747 > URL: https://issues.apache.org/jira/browse/LUCENE-6747 > Project: Lucene - Core > Issue Type: New Feature > Components: modules/analysis > Reporter: Mark Harwood > Priority: Minor > Attachments: fingerprintv1.patch > > > A TokenFilter that emits a single token which is a sorted, de-duplicated set > of the input tokens. > This approach to normalizing text is used in tools like OpenRefine[1] and > elsewhere [2] to help in clustering or linking texts. > The implementation proposed here has a an upper limit on the size of the > combined token which is output. > [1] https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-Depth > [2] > https://rajmak.wordpress.com/2013/04/27/clustering-text-map-reduce-in-python/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org