[ https://issues.apache.org/jira/browse/LUCENE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Goldfarb updated LUCENE-4072: ----------------------------------- Attachment: LUCENE-4072.patch This patch dodges the use of hasBoundaryAfter, and the tests pass. Note in doTestMode there's a clause that checks if the normalized string has length zero. It seems the nfkc_cf-normalized output of some strings is empty. Examples I found: '\uDB40\uDCD9' '\uDB43\uDF86' '\uFE04' > CharFilter that Unicode-normalizes input > ---------------------------------------- > > Key: LUCENE-4072 > URL: https://issues.apache.org/jira/browse/LUCENE-4072 > Project: Lucene - Core > Issue Type: New Feature > Components: modules/analysis > Reporter: Ippei UKAI > Attachments: DebugCode.txt, LUCENE-4072.patch, LUCENE-4072.patch, > LUCENE-4072.patch, LUCENE-4072.patch, LUCENE-4072.patch, > ippeiukai-ICUNormalizer2CharFilter-4752cad.zip > > > I'd like to contribute a CharFilter that Unicode-normalizes input with ICU4J. > The benefit of having this process as CharFilter is that tokenizer can work > on normalised text while offset-correction ensuring fast vector highlighter > and other offset-dependent features do not break. > The implementation is available at following repository: > https://github.com/ippeiukai/ICUNormalizer2CharFilter > Unfortunately this is my unpaid side-project and cannot spend much time to > merge my work to Lucene to make appropriate patch. I'd appreciate it if > anyone could give it a go. I'm happy to relicense it to whatever that meets > your needs. -- This message was sent by Atlassian JIRA (v6.1.5#6160) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org