[
https://issues.apache.org/jira/browse/OPENNLP-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Zowalla updated OPENNLP-1894:
-------------------------------------
Fix Version/s: 3.0.0
> Add dictionary-based tokenization for Japanese, Korean, and Chinese
> -------------------------------------------------------------------
>
> Key: OPENNLP-1894
> URL: https://issues.apache.org/jira/browse/OPENNLP-1894
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Adds Viterbi lattice segmentation over user-supplied MeCab-format
> dictionaries, covering Japanese (IPADIC) and Korean (mecab-ko-dic) dictionary
> layouts, plus a unigram segmenter for Chinese.
> Measured on the real IPADIC dictionary:
> * about 5 million characters per second segmentation throughput,
> single-threaded
> * 392k dictionary entries load in under one second
> Dictionaries are always user-supplied and never bundled, keeping the
> distribution free of third-party lexical data. The tokenizer manual gains a
> section whose example is asserted by a test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
