[
https://issues.apache.org/jira/browse/LUCENE-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13863570#comment-13863570
]
Benson Margulies commented on LUCENE-5386:
------------------------------------------
Can you help me with how this relates to your previous remark about attributes
other than Offset? What other attributes would get manipulated and how?
> Make Tokenizers deliver their final offsets
> -------------------------------------------
>
> Key: LUCENE-5386
> URL: https://issues.apache.org/jira/browse/LUCENE-5386
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Benson Margulies
>
> Tokenizers _must_ have an implementation of #end() in which they set up the
> final offset. Currently, nothing enforces this. end() has a useful
> implementation in TokenStream, so just making it abstract is not attractive.
> Proposal: add
> abstract int finalOffset();
> to tokenizer, and then make
> void end() {
> super.end();
> int fo = finalOffset();
> offsetAttr.setOffsets(fo, fo);
> }
> or something to that effect.
> Other alternative to be considered depending on how this looks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
