[ https://issues.apache.org/jira/browse/LUCENE-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17515909#comment-17515909 ]
Tomoko Uchida commented on LUCENE-10493: ---------------------------------------- I looked through the `parse()` method of JapaneseTokenizer and KoreanTokenizer (I'm inclined to change the method name to `forward()`, this is more aligned with the terminology of the algorithm to me). I perhaps could be too optimistic, but they are not so diverged as I first thought - except for the "unknown" word handling: this is inevitably language-specific. The N-best path part is also language-agnostic, so I'm expecting that we can safely factor it out from JapaneseTokenizer and have the common utility for performing the minimum n-best cost path calculation. Maybe we will have a utility class for the Viterbi search, and possibly a base tokenizer. I'll try to make a draft patch. > Can we unify the viterbi search logic in the tokenizers of kuromoji and nori? > ----------------------------------------------------------------------------- > > Key: LUCENE-10493 > URL: https://issues.apache.org/jira/browse/LUCENE-10493 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/analysis > Reporter: Tomoko Uchida > Priority: Major > > We now have common dictionary interfaces for kuromoji and nori > ([LUCENE-10393]). A natural question would be: is it possible to unify the > Japanese/Korean tokenizers? > The core methods of the two tokenizers are `parse()` and `backtrace()` to > calculate the minimum cost path by Viterbi search. I'd set the goal of this > issue to factoring out them into a separate class (in analysis-common) that > is shared between JapaneseTokenizer and KoreanTokenizer. > The algorithm to solve the minimum cost path itself is of course > language-agnostic, so I think it should be theoretically possible; the most > difficult part here might be the N-best path calculation - which is supported > only by JapaneseTokenizer and not by KoreanTokenizer. -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org