[
https://issues.apache.org/jira/browse/TIKA-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425790#comment-13425790
]
Ray Gauss II commented on TIKA-965:
-----------------------------------
I do have a test file and it's more than a few bytes long.
The problem arrises when there's no UTF-8 BOM and the parsers doesn't have
access to the file name as a hint.
I'm looking into a solution that will call some UTF-8 validation if the
TextDetector doesn't see enough ASCII. It appears the universalchardet library
is also looking for a BOM on initial investigation.
I'll look into this further.
> Text Detection Fails on Mostly Non-ASCII UTF-8 Files
> ----------------------------------------------------
>
> Key: TIKA-965
> URL: https://issues.apache.org/jira/browse/TIKA-965
> Project: Tika
> Issue Type: Bug
> Components: general
> Affects Versions: 1.2
> Reporter: Ray Gauss II
>
> If a file contains relatively few ASCII characters and more 8 bit UTF-8
> characters the TextDetector and TextStatistics classes fail to detect it as
> text.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
