[
https://issues.apache.org/jira/browse/TIKA-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218751#comment-13218751
]
James Sullivan commented on TIKA-856:
-------------------------------------
Actually, calling what I used a corpus may be overly generous. All I did was
take the first 40,000 articles from the Japanese Wikipedia content and then
removed the XML and non-Japanese content (of which there was a lot given the
nature of Wikipedia). 40,000 articles was enough to get to the around 6,1
million that Jan H mentioned but it still was not working. I will attach the
generated ja.ngp.
> Support CJK (Chinese, Japanese and Korean) language detection
> -------------------------------------------------------------
>
> Key: TIKA-856
> URL: https://issues.apache.org/jira/browse/TIKA-856
> Project: Tika
> Issue Type: New Feature
> Components: languageidentifier
> Affects Versions: 1.0
> Environment: All
> Reporter: James Sullivan
> Labels: Chinese, Japanese
>
> Support language detection of CJK (Chinese, Japanese and Korean).
> Some estimates have Chinese users overtaking English users on the Internet
> so it is important that these languages used by large number of people be
> supported.
> See TIKA-855
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
