Kristian Rickert created OPENNLP-1894:
-----------------------------------------
Summary: Add dictionary-based tokenization for Japanese, Korean,
and Chinese
Key: OPENNLP-1894
URL: https://issues.apache.org/jira/browse/OPENNLP-1894
Project: OpenNLP
Issue Type: Improvement
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Adds Viterbi lattice segmentation over user-supplied MeCab-format dictionaries,
covering Japanese (IPADIC) and Korean (mecab-ko-dic) dictionary layouts, plus a
unigram segmenter for Chinese.
Measured on the real IPADIC dictionary:
* about 5 million characters per second segmentation throughput, single-threaded
* 392k dictionary entries load in under one second
Dictionaries are always user-supplied and never bundled, keeping the
distribution free of third-party lexical data. The tokenizer manual gains a
section whose example is asserted by a test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
