twosom commented on code in PR #12047:
URL: https://github.com/apache/lucene/pull/12047#discussion_r1059413652
##########
lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseNumberFilter.java:
##########
@@ -39,9 +39,9 @@
* <p>Notice that this analyzer uses a token composition scheme and relies on
punctuation tokens
* being found in the token stream. Please make sure your {@link
JapaneseTokenizer} has {@code
* discardPunctuation} set to false. In case punctuation characters, such as ．
(U+FF0E FULLWIDTH
- * FULL STOP), is removed from the token stream, this filter would find input
tokens tokens ３ and ２千
Review Comment:
removed repeated words `tokens tokens`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
