[
https://issues.apache.org/jira/browse/LUCENE-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13761155#comment-13761155
]
Benson Margulies commented on LUCENE-5202:
------------------------------------------
I'm a bit confused here.
I have two cases. In the test case at hand, I'm trying very hard not to create
any new tokens, or so I thought. All I wanted to do was to use knowledge from
lookahead to influence the values in some attributes.
In a second case, I want to insert additional tokens into the stream. In fact,
in my 'real' code that this test case is cut down from, afterPosition calls
insertToken and does pretty nearly precisely what you coded here when, indeed,
I want to create a new token.
So I really don't want a new position-increment 0 token. I want to really just
change the values. Did you think that I wanted a new PI=0 token, or are you
telling me that I can't do what I want using this class as a base class.
> LookaheadTokenFilter consumes an extra token in nextToken
> ---------------------------------------------------------
>
> Key: LUCENE-5202
> URL: https://issues.apache.org/jira/browse/LUCENE-5202
> Project: Lucene - Core
> Issue Type: Bug
> Affects Versions: 4.3.1
> Reporter: Benson Margulies
> Attachments: LUCENE-5202.patch, LUCENE-5202.patch
>
>
> This is a bit hard to explain except by looking at the test case. I've coded
> a filter that uses LookaheadTokenFilter. The incrementToken method peeks some
> tokens. Then, it seems, nextToken in the Lookahead class calls peekToken
> itself, which seems to me to consume a token so that it's not seen when the
> derived class sets out to process the next set of tokens.
> In passing, this test case can be used to demonstrate that it does not work
> to try to use the afterPosition method to set up attributes of the token that
> we're 'after'. Probably that was never intended. However, I'm hoping for some
> feedback as to whether the rest of the structure here is as intended for
> subclasses of LookaheadTokenFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
