[ https://issues.apache.org/jira/browse/LUCENE-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283943#comment-13283943 ]
Ippei UKAI commented on LUCENE-4072: ------------------------------------ I've backported your additional test case of random string and reproduced the problem. However, I haven't managed to pin point the problem so far. Having no clue coming out of AssertionError, attacking LUCENE-3900 first might be more productive. This final offset thing looks like a common occurrence among CharFilters (re LUCENE-3820 LUCENE-3971) but I'm yet to find a common theme in them. > CharFilter that Unicode-normalizes input > ---------------------------------------- > > Key: LUCENE-4072 > URL: https://issues.apache.org/jira/browse/LUCENE-4072 > Project: Lucene - Java > Issue Type: New Feature > Components: modules/analysis > Reporter: Ippei UKAI > Attachments: LUCENE-4072.patch, > ippeiukai-ICUNormalizer2CharFilter-4752cad.zip > > > I'd like to contribute a CharFilter that Unicode-normalizes input with ICU4J. > The benefit of having this process as CharFilter is that tokenizer can work > on normalised text while offset-correction ensuring fast vector highlighter > and other offset-dependent features do not break. > The implementation is available at following repository: > https://github.com/ippeiukai/ICUNormalizer2CharFilter > Unfortunately this is my unpaid side-project and cannot spend much time to > merge my work to Lucene to make appropriate patch. I'd appreciate it if > anyone could give it a go. I'm happy to relicense it to whatever that meets > your needs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org