[ https://issues.apache.org/jira/browse/LUCENE-3767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214148#comment-13214148 ]
Robert Muir commented on LUCENE-3767: ------------------------------------- {quote} Robert mentioned earlier that he believes IPADIC could have been annotated with compounds as the documentation mentions them, but they're not part of the IPADIC model we are using. If it is possible to get the decompounds from the training data (Kyoto Corpus), a better overall approach is then to do regular segmentation (normal mode) and then provide the decompounds directly from the token info for the compounds. We might need to retrain the model and preserving the decompounds in order for this to work, but I think it is worth investigating. {quote} The dictionary documentation for the original ipadic has the ability to hold compound data (not in mecab-ipadic though, so maybe it was never implemented?!), but I don't actually see it in any implementations. So yeah, we would need to find a corpus containing compound information (and of course extend the file format and add support to kuromoji) to support that. However, would this really solve the total issue? Wouldn't that really only help for known kanji compounds... whereas most katakana compounds (e.g. the software engineer example) are expected to be OOV anyway? So it seems like, even if we ensured the dictionary was annotated for long kanji such that we always used decompounded forms, we need a 'heuristical' decomposition like search-mode either way, at least for the unknown katakana case? And I tend to like Mike's improvements from a relevance perspective for these reasons: # keeping the original compound term for improved precision # preventing compound decomposition from having any unrelated negative impact on the rest of the tokenization So I think we should pursue this change, even if we want to separately train a dictionary in the future, because in that case, we would just disable the kanji decomposition heuristic but keep the heuristic (obviously re-tuned!) for katakana? > Explore streaming Viterbi search in Kuromoji > -------------------------------------------- > > Key: LUCENE-3767 > URL: https://issues.apache.org/jira/browse/LUCENE-3767 > Project: Lucene - Java > Issue Type: Improvement > Components: modules/analysis > Reporter: Michael McCandless > Assignee: Michael McCandless > Fix For: 3.6, 4.0 > > Attachments: LUCENE-3767.patch, LUCENE-3767.patch, LUCENE-3767.patch, > compound_diffs.txt > > > I've been playing with the idea of changing the Kuromoji viterbi > search to be 2 passes (intersect, backtrace) instead of 4 passes > (break into sentences, intersect, score, backtrace)... this is very > much a work in progress, so I'm just getting my current state up. > It's got tons of nocommits, doesn't properly handle the user dict nor > extended modes yet, etc. > One thing I'm playing with is to add a double backtrace for the long > compound tokens, ie, instead of penalizing these tokens so that > shorter tokens are picked, leave the scores unchanged but on backtrace > take that penalty and use it as a threshold for a 2nd best > segmentation... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org