msokolov opened a new issue, #15899:
URL: https://github.com/apache/lucene/issues/15899
### Description
We've been using TruncateTokenFilter to impose a max token length of 64 in
some fields and recently found that when the 64th character in the string is an
emoji (like 🎁 for example), the indexed token we end up with has a Unicode
replacement character (� == U+FFFD) as its final character. In itself, this
doesn't seem super terrible, but in our case it is leading to worse problems.
I've been attempting to enforce the uniqueness of terms with custom term
frequencies: each such term should occur no more than once in a document. Our
indexing code guarantees this. However, due to the broken truncation filter,
two strings like
1080p_4ct_antenna_antenna_digital_hdtv_indoor_long_range__hdtv_🎁
1080p_4ct_antenna_antenna_digital_hdtv_indoor_long_range__hdtv_😃
will still be unique after truncation, but their final "character" is now
the first half of a surrogate pair, and later, when Lucene indexes these terms,
it (apparently) converts those characters to \UFFFD causing the tokens to no
longer be unique.
Anyway, it seems as if TruncateTokenFilter ought to be checking if the final
character in its buffer is the first half of a surrogate pair, and if so,
truncating that too.
### Version and environment details
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
