[ https://issues.apache.org/jira/browse/TIKA-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14646530#comment-14646530 ]
Paul Ramirez commented on TIKA-1696: ------------------------------------ The algorithm that is used is described here: https://en.wikipedia.org/wiki/Margin_Infused_Relaxed_Algorithm > Language Identification with Text Processing Toolkit from MITLL > --------------------------------------------------------------- > > Key: TIKA-1696 > URL: https://issues.apache.org/jira/browse/TIKA-1696 > Project: Tika > Issue Type: New Feature > Components: languageidentifier > Reporter: Paul Ramirez > Fix For: 1.10 > > > The aim here is to extend the methods for language identification within > text. MIT Lincoln Labs has an open source library [1] written in Julia. > Having spoken with the MITLL guys there is a possibility that there is a > scala version of this library which would make it easier to package in with > Tika. > At this point I'm not quite sure how many languages this library supports by > default but it can be extended when provided some training data. > [1] https://github.com/mit-nlp/Text.jl -- This message was sent by Atlassian JIRA (v6.3.4#6332)