[
https://issues.apache.org/jira/browse/LUCENE-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir reopened LUCENE-3717:
---------------------------------
reopening since we have more work to do / more bugs.
I'll look at committing/backporting the current patch as a start but i think we
should check every tokenizer/filter/etc and just clean this up.
> Add fake charfilter to BaseTokenStreamTestCase to find offsets bugs
> -------------------------------------------------------------------
>
> Key: LUCENE-3717
> URL: https://issues.apache.org/jira/browse/LUCENE-3717
> Project: Lucene - Java
> Issue Type: Task
> Reporter: Robert Muir
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3717.patch, LUCENE-3717_more.patch
>
>
> Recently lots of issues have been fixed about broken offsets, but it would be
> nice to improve the
> test coverage and test that they work across the board (especially with
> charfilters).
> in BaseTokenStreamTestCase.checkRandomData, we can sometimes pass the
> analyzer a reader wrapped
> in a "MockCharFilter" (the one in the patch sometimes doubles characters). If
> the analyzer does
> not call correctOffsets or does incorrect "offset math" (LUCENE-3642, etc)
> then eventually
> this will create offsets and the test will fail.
> Other than tests bugs, this found 2 real bugs: ICUTokenizer did not call
> correctOffset() in its end(),
> and ThaiWordFilter did incorrect offset math.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
