[ https://issues.apache.org/jira/browse/LUCENE-8816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16861672#comment-16861672 ]
Tomoko Uchida commented on LUCENE-8816: --------------------------------------- I'd like to add some more information: the leftID (and rightID) is tied to the POS tags and in practice there are not so much pos tag variations. I think the current constraint {{leftId < 4096}} (or {{leftId < 8191}}, if it can be easily changed so) is perfectly okay if following conditions are met. 1. The dictionary learner/re-trainer program included in the mecab-ipadic devtool does not generate leftID (and rightID) values larger than 4196 (or 8191). 2. UniDic (I'd like to support this dictionary on this issue as I wrote in the issue description) has no leftID (and rightID) values greater than 4196 (or 8191). 3. A few well-known variants of mecab-ipadic or unidic does not have leftID (and rightID) values larger than 4196 (or 8191). Give me some time to examine if we need to re-consider the constraint. (It's just a guess but the original mecab itself is also a performance-savvy software, so it could have similar restrictions for its dictionary format.) At least about the point 3, I think I can talk with the dictionary developers about it before tackling with Lucene code, if it's needed. There is another possibility that users give large values to leftIDs (and rightIDs) in their customized dictionary by hand, however I don't think we should take care about that. I have no idea about Korian dictionaries. I agree with that it will be better to change the all assertions to some Exceptions so that users can figure out the problem with their customized dictionary. > Decouple Kuromoji's morphological analyser and its dictionary > ------------------------------------------------------------- > > Key: LUCENE-8816 > URL: https://issues.apache.org/jira/browse/LUCENE-8816 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/analysis > Reporter: Tomoko Uchida > Priority: Major > > I've inspired by this mail-list thread. > > [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201905.mbox/%3CCAGUSZHA3U_vWpRfxQb4jttT7sAOu%2BuaU8MfvXSYgNP9s9JNsXw%40mail.gmail.com%3E] > As many Japanese already know, default built-in dictionary bundled with > Kuromoji (MeCab IPADIC) is a bit old and no longer maintained for many years. > While it has been slowly obsoleted, well-maintained and/or extended > dictionaries risen up in recent years (e.g. > [mecab-ipadic-neologd|https://github.com/neologd/mecab-ipadic-neologd], > [UniDic|https://unidic.ninjal.ac.jp/]). To use them with Kuromoji, some > attempts/projects/efforts are made in Japan. > However current architecture - dictionary bundled jar - is essentially > incompatible with the idea "switch the system dictionary", and developers > have difficulties to do so. > Traditionally, the morphological analysis engine (viterbi logic) and the > encoded dictionary (language model) had been decoupled (like MeCab, the > origin of Kuromoji, or lucene-gosen). So actually decoupling them is a > natural idea, and I feel that it's good time to re-think the current > architecture. > Also this would be good for advanced users who have customized/re-trained > their own system dictionary. > Goals of this issue: > * Decouple JapaneseTokenizer itself and encoded system dictionary. > * Implement dynamic dictionary load mechanism. > * Provide developer-oriented dictionary build tool. > Non-goals: > * Provide learner or language model (it's up to users and should be outside > the scope). > I have not dove into the code yet, so have no idea about it's easy or > difficult at this moment. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org