[ https://issues.apache.org/jira/browse/LUCENE-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12757224#action_12757224 ]
Uwe Schindler commented on LUCENE-1919: --------------------------------------- Token.clone() is not called by next() anymore (and even not in 2.4.1) - because of this we need the Payload cloning (Token.clone() would do it). The full private Token is done like this now (and this was the same in 1057): next(Token) is simply called with a new Token instance. The fix for this issue does it in the same way (like 2.4.1). The extension here is, that it now works exactly the same with incrementToken(). After calling next(Token) with the private token, the Payload is explicitely cloned. As I told some comments above, I think more the problem is the following: If a next(Token) or incrementToken() method sets a Payload, the payload data is not copied, its only a reference to the byte array. E.g. the next(Token) methods uses a pre-allocated array and sets this always as data (which is perfectly legal in the reuse case), only modifying the data contents. If you call next(Token) with a new allocated Token, this Token is private, but if next(Token) sets again the preallocated byte array, it is not private anymore (you will se the modifications in the previous token, too). You would have the same bug like now (even in 2.4.1). I think because of this the payload is cloned separately to be sure that it is private like the newly allocated Token. > Analysis back compat break > -------------------------- > > Key: LUCENE-1919 > URL: https://issues.apache.org/jira/browse/LUCENE-1919 > Project: Lucene - Java > Issue Type: Bug > Reporter: Yonik Seeley > Assignee: Uwe Schindler > Fix For: 2.9 > > Attachments: LUCENE-1919.patch, LUCENE-1919.patch, LUCENE-1919.patch, > LUCENE-1919.patch, LUCENE-1919.patch, LUCENE-1919.patch > > > Old and new style token streams don't mix well. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org