[
https://issues.apache.org/jira/browse/LUCENE-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12757224#action_12757224
]
Uwe Schindler commented on LUCENE-1919:
---------------------------------------
Token.clone() is not called by next() anymore (and even not in 2.4.1) - because
of this we need the Payload cloning (Token.clone() would do it). The full
private Token is done like this now (and this was the same in 1057):
next(Token) is simply called with a new Token instance. The fix for this issue
does it in the same way (like 2.4.1). The extension here is, that it now works
exactly the same with incrementToken(). After calling next(Token) with the
private token, the Payload is explicitely cloned.
As I told some comments above, I think more the problem is the following: If a
next(Token) or incrementToken() method sets a Payload, the payload data is not
copied, its only a reference to the byte array. E.g. the next(Token) methods
uses a pre-allocated array and sets this always as data (which is perfectly
legal in the reuse case), only modifying the data contents. If you call
next(Token) with a new allocated Token, this Token is private, but if
next(Token) sets again the preallocated byte array, it is not private anymore
(you will se the modifications in the previous token, too). You would have the
same bug like now (even in 2.4.1). I think because of this the payload is
cloned separately to be sure that it is private like the newly allocated Token.
> Analysis back compat break
> --------------------------
>
> Key: LUCENE-1919
> URL: https://issues.apache.org/jira/browse/LUCENE-1919
> Project: Lucene - Java
> Issue Type: Bug
> Reporter: Yonik Seeley
> Assignee: Uwe Schindler
> Fix For: 2.9
>
> Attachments: LUCENE-1919.patch, LUCENE-1919.patch, LUCENE-1919.patch,
> LUCENE-1919.patch, LUCENE-1919.patch, LUCENE-1919.patch
>
>
> Old and new style token streams don't mix well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
