[ https://issues.apache.org/jira/browse/TIKA-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14342533#comment-14342533 ]
Tyler Palsulich commented on TIKA-369: -------------------------------------- Thanks, Ken! In that case, I definitely agree. > Improve accuracy of language detection > -------------------------------------- > > Key: TIKA-369 > URL: https://issues.apache.org/jira/browse/TIKA-369 > Project: Tika > Issue Type: Improvement > Components: languageidentifier > Affects Versions: 0.6 > Reporter: Ken Krugler > Assignee: Ken Krugler > Attachments: Surprise and Coincidence.pdf, lingdet-mccs.pdf, > textcat.pdf > > > Currently the LanguageProfile code uses 3-grams to find the best language > profile using Pearson's chi-square test. This has three issues: > 1. The results aren't very good for short runs of text. Ted Dunning's paper > (attached) indicates that a log-likelihood ratio (LLR) test works much > better, which would then make language detection faster due to less text > needing to be processed. > 2. The current LanguageIdentifier.isReasonablyCertain() method uses an exact > value as a threshold for certainty. This is very sensitive to the amount of > text being processed, and thus gives false negative results for short runs of > text. > 3. Certainty should also be based on how much better the result is for > language X, compared to the next best language. If two languages both had > identical sum-of-squares values, and this value was below the threshold, then > the result is still not very certain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)