[
https://issues.apache.org/jira/browse/TIKA-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778137#action_12778137
]
Luke Nezda commented on TIKA-322:
---------------------------------
http://code.google.com/p/juniversalchardet/ has a pretty good, efficient
charset decoder which is a Java port of the Mozilla universalchardet
algorithms. It is licensed under Mozilla Public License Version 1.1. I am not
sure if MPL is ASF compatible; it appears to be, but ianal. afaik, it does not
provide detection confidence or language detection features ICU4J does and I
think it has code/data files for less encodings, but it is primarily
statistical so they could be added. I am also not sure what choices were made
with regard to multiple encodings. In theory, it should detect what Firefox
detects for a given URL/file.
> Improve encoding detection speed and accuracy
> ---------------------------------------------
>
> Key: TIKA-322
> URL: https://issues.apache.org/jira/browse/TIKA-322
> Project: Tika
> Issue Type: Improvement
> Components: mime
> Reporter: Jukka Zitting
> Priority: Minor
>
> The encoding detection code we took from ICU4J is not very efficient and
> sometimes produces odd results when more than one encoding matches the given
> input data. It would be good to refactor the code to be faster for
> easy-to-detect encodings and to have better heuristics in case multiple
> matches are found.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
