Tim Allison created TIKA-3479:
---------------------------------
Summary: UniversalCharsetDetector in 2.x is misidentifying
windows-1252(?) as ISO-8859-1
Key: TIKA-3479
URL: https://issues.apache.org/jira/browse/TIKA-3479
Project: Tika
Issue Type: Task
Reporter: Tim Allison
We've lost quite a few "common words" for Czech and Slovak text files in 2.x
vs. 1.x. The key issue appears to be the following (which we do not have in
1.x).
{noformat}
/*
* hex value 0x81, 0x8d, 0x8f, 0x90 don't exist in charset windows-1252.
* If these value's count > 0, return true
* */
private Boolean hasNonexistentHexInCharsetWindows1252() {
return (statistics.count(0x81) > 0 || statistics.count(0x8d) > 0 ||
statistics.count(0x8f) > 0 || statistics.count(0x90) > 0 ||
statistics.count(0x9d) > 0);
}
{noformat}
I _think_ the files are actually https://en.wikipedia.org/wiki/Code_page_852,
and they do have these characters. windows-1252 is _generally_ a better batch
for cp852 than ISO-8859-1.
Not sure how best to handle this...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
