[
https://issues.apache.org/jira/browse/LUCENE-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716626#comment-14716626
]
ASF subversion and git services commented on LUCENE-6747:
---------------------------------------------------------
Commit 1698145 from mharw...@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1698145 ]
LUCENE-6747: FingerprintFilter is a new TokenFilter that outputs a single token
which is a concatenation of the sorted and de-duplicated set of input tokens.
> FingerprintFilter - a TokenFilter for clustering/linking purposes
> -----------------------------------------------------------------
>
> Key: LUCENE-6747
> URL: https://issues.apache.org/jira/browse/LUCENE-6747
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: Mark Harwood
> Priority: Minor
> Attachments: fingerprintv1.patch, fingerprintv2.patch,
> fingerprintv3.patch, fingerprintv4.patch
>
>
> A TokenFilter that emits a single token which is a sorted, de-duplicated set
> of the input tokens.
> This approach to normalizing text is used in tools like OpenRefine[1] and
> elsewhere [2] to help in clustering or linking texts.
> The implementation proposed here has a an upper limit on the size of the
> combined token which is output.
> [1] https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-Depth
> [2]
> https://rajmak.wordpress.com/2013/04/27/clustering-text-map-reduce-in-python/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
