[
https://issues.apache.org/jira/browse/TIKA-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372183#comment-14372183
]
Tyler Palsulich commented on TIKA-1406:
---------------------------------------
Can someone familiar with the UniversalEncodingDetector take a look at this
patch? Sorry no one ever got back, [~almson]! Do you have a couple test files
which demonstrate these issues?
> Problems in TXT encoding detection
> ----------------------------------
>
> Key: TIKA-1406
> URL: https://issues.apache.org/jira/browse/TIKA-1406
> Project: Tika
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.6, 1.7
> Reporter: Aleksandr Dubinsky
> Attachments: 0001-fix-TXT-encoding-detection.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The detection of TXT file encoding often makes mistakes. Two things can
> improve the detection in many cases:
> - Increase the lookahead from 16k bytes to a larger number, such as 128k or
> larger.
> - Improve on the brain-dead heuristic that if a file doesn't have a \r then
> it must be ISO 8859-1(5) instead of Windows-1252. (For one, it mis-detects
> files that don't have any newlines.)
> A remaining problem that doesn't have an immediate solution, is the frequent
> mis-detection of Windows-1252 as Shift-JIS. A flag to forbit Shift-JIS is
> desirable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
