This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new a40d54f6a6 TIKA-5052: Big5 whatwg (#3595)
a40d54f6a6 is described below
commit a40d54f6a6d9cf8ce485f10bb07011cd32ef5925
Author: Tim Allison <[email protected]>
AuthorDate: Mon Nov 9 17:13:35 2026 -0500
TIKA-5052: Big5 whatwg (#3595)
* TIKA-5052: decode Big5 as browsers do: Big5-HKSCS with the WHATWG big5
patch
* TIKA-5052: generalize the Big5 browser patch; guard decode-as charset
names
* TIKA-5052: decode EUC-JP as browsers do: x-eucJP-Open with the WHATWG
patch
---
CHANGES.txt | 13 +
LICENSE.txt | 30 ++
.../pages/advanced/charset-detection-design.adoc | 6 +-
.../src/main/appended-resources/META-INF/LICENSE | 29 ++
.../src/main/appended-resources/META-INF/LICENSE | 29 ++
tika-core/pom.xml | 2 +
.../src/main/appended-resources/META-INF/LICENSE | 29 ++
.../org/apache/tika/detect/AutoDetectReader.java | 7 +-
.../org/apache/tika/detect/CharsetSupersets.java | 57 ++-
.../org/apache/tika/detect/PatchedCharset.java | 278 +++++++++++++
.../org/apache/tika/detect/x-Big5-HKSCS-WHATWG.tsv | 177 +++++++++
.../org/apache/tika/detect/x-eucJP-Open-WHATWG.tsv | 43 ++
.../apache/tika/detect/CharsetSupersetsTest.java | 20 +-
.../apache/tika/detect/DecodeAsNameUsageTest.java | 123 ++++++
.../org/apache/tika/detect/PatchedCharsetTest.java | 293 ++++++++++++++
.../apache/tika/detect/WhatwgPatchGenerator.java | 439 +++++++++++++++++++++
.../ml/junkdetect/JunkFilterEncodingDetector.java | 2 +-
.../org/apache/tika/parser/html/JSoupParser.java | 10 +-
.../apache/tika/parser/html/HtmlParserTest.java | 21 +
.../tika/parser/microsoft/OutlookExtractor.java | 4 +-
.../src/main/appended-resources/META-INF/LICENSE | 29 ++
21 files changed, 1630 insertions(+), 11 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 630012c002..fa3dd54880 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -445,6 +445,19 @@ Release 4.3.0 - ???
tika-eval:languagenessCoverage. Languageness is -99 (unknown) for text
with no letters
or with under 90% of its letters in scripts the model was trained on
(TIKA-5027).
+ * Big5 text decodes as browsers do: Big5-HKSCS with the 144 byte pairs
where the WHATWG
+ Encoding Standard differs patched to the WHATWG value. A3E1 is the euro
sign instead of
+ U+FFFD; A145, A1E3 and A1FE are the CP950 forms U+2027, U+FF5E and
U+FF0F; six pairs in
+ C6CF-C6DF are radicals and ditto marks, not kana. Detected-charset
metadata is unchanged;
+ tk:decoded-charset is x-Big5-HKSCS-WHATWG, which Charset.forName does not
resolve
+ (TIKA-5052).
+
+ * EUC-JP text decodes as browsers do: x-eucJP-Open with nine pairs patched
to the WHATWG
+ value, the Microsoft forms windows-31j already gives for Shift_JIS (A1C1
wave dash is
+ U+FF5E, A1DD minus U+FF0D, A1F1/A1F2/A2CC cent, pound and not signs
U+FFE0-FFE2, A1BD
+ U+2015, A1C2 U+2225, FCFB/FCFC U+FFE2/U+FFE4). tk:decoded-charset is
+ x-eucJP-Open-WHATWG (TIKA-5052).
+
* Charset detection finds non-ASCII text past the first 16 KB, as in an
email with a
long ASCII header block, a log or a CSV: those were reported as
windows-1252. When the
leading sample is not conclusive, detection reads on (up to 16 MB of a
file or byte
diff --git a/LICENSE.txt b/LICENSE.txt
index 17ea384d61..fd3653e2be 100644
--- a/LICENSE.txt
+++ b/LICENSE.txt
@@ -208,6 +208,36 @@ APACHE TIKA SUBCOMPONENTS
Apache Tika includes a number of subcomponents with separate copyright notices
and license terms. Your use of these subcomponents is subject to the terms and
conditions of the following licenses.
+-------------------------------
+Big5 and EUC-JP decode patch tables in tika-core
(org/apache/tika/detect/*-WHATWG.tsv),
+derived from index-big5.txt and index-jis0208.txt of the WHATWG Encoding
Standard
+(https://encoding.spec.whatwg.org/)
+
+ Copyright (c) WHATWG (Apple, Google, Mozilla, Microsoft).
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are met:
+ 1. Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following disclaimer.
+ 2. Redistributions in binary form must reproduce the above copyright
notice,
+ this list of conditions and the following disclaimer in the
documentation
+ and/or other materials provided with the distribution.
+ 3. Neither the name of the copyright holder nor the names of its
+ contributors may be used to endorse or promote products derived from
+ this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ POSSIBILITY OF SUCH DAMAGE.
+
-------------------------------
MIME type information from file-4.26.tar.gz (http://www.darwinsys.com/file/)
diff --git a/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
b/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
index f9ef4f079f..a083a9832f 100644
--- a/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
+++ b/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
@@ -277,7 +277,11 @@ Two corrections make the rate trustworthy:
* **Decode under the vendor superset, not the strict base** — real vendor
extensions (NEC/IBM for Shift_JIS/EUC-JP, HKSCS for Big5) would otherwise
count
- as failures and penalize genuine CJK.
+ as failures and penalize genuine CJK. Big5 and Big5-HKSCS decode as
+ `x-Big5-HKSCS-WHATWG`: Big5-HKSCS with the 144 byte pairs where the WHATWG
+ Encoding Standard differs (A3E1 euro sign, CP950 punctuation forms) patched
to
+ the browser value. It is not resolvable by `Charset.forName`; APIs that
take a
+ charset name use `CharsetSupersets.jdkDecodeAs`.
* **Discount embedded UTF-8** — mixed-encoding pages (legacy CJK body + UTF-8
widgets) would otherwise read as 2–9.5% failure. The validator *skips*
positions that begin a valid UTF-8 sequence rather than physically stripping
diff --git a/tika-app/src/main/appended-resources/META-INF/LICENSE
b/tika-app/src/main/appended-resources/META-INF/LICENSE
index 7365b20014..f4985b484c 100644
--- a/tika-app/src/main/appended-resources/META-INF/LICENSE
+++ b/tika-app/src/main/appended-resources/META-INF/LICENSE
@@ -186,6 +186,35 @@ ASM library (asm)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
THE POSSIBILITY OF SUCH DAMAGE.
+Big5 and EUC-JP decode patch tables in tika-core
(org/apache/tika/detect/*-WHATWG.tsv),
+derived from index-big5.txt and index-jis0208.txt of the WHATWG Encoding
Standard
+(https://encoding.spec.whatwg.org/)
+
+ Copyright (c) WHATWG (Apple, Google, Mozilla, Microsoft).
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are met:
+ 1. Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following disclaimer.
+ 2. Redistributions in binary form must reproduce the above copyright
notice,
+ this list of conditions and the following disclaimer in the
documentation
+ and/or other materials provided with the distribution.
+ 3. Neither the name of the copyright holder nor the names of its
+ contributors may be used to endorse or promote products derived from
+ this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ POSSIBILITY OF SUCH DAMAGE.
+
MIME type information from file-4.26.tar.gz (http://www.darwinsys.com/file/)
Copyright (c) Ian F. Darwin 1986, 1987, 1989, 1990, 1991, 1992, 1994, 1995.
diff --git
a/tika-bundles/tika-bundle-standard/src/main/appended-resources/META-INF/LICENSE
b/tika-bundles/tika-bundle-standard/src/main/appended-resources/META-INF/LICENSE
index c008bdaa63..8ffb8fda7a 100644
---
a/tika-bundles/tika-bundle-standard/src/main/appended-resources/META-INF/LICENSE
+++
b/tika-bundles/tika-bundle-standard/src/main/appended-resources/META-INF/LICENSE
@@ -186,6 +186,35 @@ ASM library (asm)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
THE POSSIBILITY OF SUCH DAMAGE.
+Big5 and EUC-JP decode patch tables in tika-core
(org/apache/tika/detect/*-WHATWG.tsv),
+derived from index-big5.txt and index-jis0208.txt of the WHATWG Encoding
Standard
+(https://encoding.spec.whatwg.org/)
+
+ Copyright (c) WHATWG (Apple, Google, Mozilla, Microsoft).
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are met:
+ 1. Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following disclaimer.
+ 2. Redistributions in binary form must reproduce the above copyright
notice,
+ this list of conditions and the following disclaimer in the
documentation
+ and/or other materials provided with the distribution.
+ 3. Neither the name of the copyright holder nor the names of its
+ contributors may be used to endorse or promote products derived from
+ this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ POSSIBILITY OF SUCH DAMAGE.
+
MIME type information from file-4.26.tar.gz (http://www.darwinsys.com/file/)
Copyright (c) Ian F. Darwin 1986, 1987, 1989, 1990, 1991, 1992, 1994, 1995.
diff --git a/tika-core/pom.xml b/tika-core/pom.xml
index ea6c22db6a..05524441b7 100644
--- a/tika-core/pom.xml
+++ b/tika-core/pom.xml
@@ -203,6 +203,8 @@
<inputExclude>src/test/resources/org/apache/tika/**</inputExclude>
<inputExclude>src/main/resources/org/apache/tika/language/*.ngp</inputExclude>
<inputExclude>src/main/resources/org/apache/tika/detect/*.nnmodel</inputExclude>
+ <!-- BSD-3 (WHATWG Encoding Standard); notice in LICENSE -->
+
<inputExclude>src/main/resources/org/apache/tika/detect/*-WHATWG.tsv</inputExclude>
<inputExclude>src/test/resources/test-documents/file-list.txt</inputExclude>
<inputExclude>src/test/resources/test-documents/ang20150420t182050_corr_v1e_img.hdr</inputExclude>
<inputExclude>src/test/resources/test-documents/*.pdf</inputExclude>
diff --git a/tika-core/src/main/appended-resources/META-INF/LICENSE
b/tika-core/src/main/appended-resources/META-INF/LICENSE
index b834fe542e..dd575e1711 100644
--- a/tika-core/src/main/appended-resources/META-INF/LICENSE
+++ b/tika-core/src/main/appended-resources/META-INF/LICENSE
@@ -4,6 +4,35 @@ Apache Tika includes a number of subcomponents with separate
copyright notices
and license terms. Your use of these subcomponents is subject to the terms and
conditions of the following licenses.
+Big5 and EUC-JP decode patch tables in tika-core
(org/apache/tika/detect/*-WHATWG.tsv),
+derived from index-big5.txt and index-jis0208.txt of the WHATWG Encoding
Standard
+(https://encoding.spec.whatwg.org/)
+
+ Copyright (c) WHATWG (Apple, Google, Mozilla, Microsoft).
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are met:
+ 1. Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following disclaimer.
+ 2. Redistributions in binary form must reproduce the above copyright
notice,
+ this list of conditions and the following disclaimer in the
documentation
+ and/or other materials provided with the distribution.
+ 3. Neither the name of the copyright holder nor the names of its
+ contributors may be used to endorse or promote products derived from
+ this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ POSSIBILITY OF SUCH DAMAGE.
+
MIME type information from file-4.26.tar.gz (http://www.darwinsys.com/file/)
Copyright (c) Ian F. Darwin 1986, 1987, 1989, 1990, 1991, 1992, 1994, 1995.
diff --git
a/tika-core/src/main/java/org/apache/tika/detect/AutoDetectReader.java
b/tika-core/src/main/java/org/apache/tika/detect/AutoDetectReader.java
index 9e63889ded..fc4a8cb4e1 100644
--- a/tika-core/src/main/java/org/apache/tika/detect/AutoDetectReader.java
+++ b/tika-core/src/main/java/org/apache/tika/detect/AutoDetectReader.java
@@ -58,7 +58,7 @@ public class AutoDetectReader extends BufferedReader {
private AutoDetectReader(InputStream stream, Charset charset) throws
IOException {
super(new InputStreamReader(stream, charset));
- this.charset = charset;
+ this.charset = CharsetSupersets.jdkCharset(charset);
// TIKA-240: Drop the BOM if present
mark(1);
@@ -152,6 +152,11 @@ public class AutoDetectReader extends BufferedReader {
}
+ /**
+ * The charset the content was decoded as, always one {@link
Charset#forName} resolves: for a
+ * {@link CharsetSupersets} browser patch, its JDK base (the patched
decoder's name is in
+ * {@link org.apache.tika.metadata.TikaCoreProperties#DECODED_CHARSET}).
+ */
public Charset getCharset() {
return charset;
}
diff --git
a/tika-core/src/main/java/org/apache/tika/detect/CharsetSupersets.java
b/tika-core/src/main/java/org/apache/tika/detect/CharsetSupersets.java
index f862e37595..1bdf569c82 100644
--- a/tika-core/src/main/java/org/apache/tika/detect/CharsetSupersets.java
+++ b/tika-core/src/main/java/org/apache/tika/detect/CharsetSupersets.java
@@ -17,6 +17,7 @@
package org.apache.tika.detect;
import java.nio.charset.Charset;
+import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
@@ -38,7 +39,8 @@ import java.util.Map;
* <ul>
* <li>EUC-KR → x-windows-949 (MS949 is a strict superset: all EUC-KR byte
sequences
* decode identically, extension chars in x-windows-949 would mojibake
under EUC-KR)</li>
- * <li>Big5 → Big5-HKSCS (HKSCS adds Hong Kong Supplementary Characters)</li>
+ * <li>Big5 → Big5-HKSCS (HKSCS adds Hong Kong Supplementary Characters),
with the pairs
+ * where the WHATWG big5 decoder differs patched to the WHATWG value
(see below)</li>
* <li>GB2312 → GB18030 (GB18030 is a strict superset of both GB2312 and
GBK)</li>
* <li>GBK → GB18030 (GB18030 is a strict superset; enables 4-byte extension
sequences)</li>
* <li>Shift_JIS → windows-31j (MS932 is a strict superset with NEC/IBM
extensions)</li>
@@ -48,12 +50,22 @@ import java.util.Map;
* windows-874 (what browsers do: 0x80-0x9F are punctuation, not C1
controls; the few
* bytes the windows code page leaves undefined decode as U+FFFD)</li>
* </ul>
+ *
+ * <h3>Browser patches</h3>
+ * <p>Big5-HKSCS and x-eucJP-Open (detected, or reached as a superset) decode
as
+ * {@code x-Big5-HKSCS-WHATWG} and {@code x-eucJP-Open-WHATWG}: the JDK
charset with the byte
+ * pairs where the WHATWG Encoding Standard differs decoded as browsers do
(Big5: A3E1 euro
+ * sign, CP950 punctuation forms; EUC-JP: the Microsoft forms of nine
punctuation and symbol
+ * pairs, e.g. A1C1 U+FF5E, as windows-31j gives for Shift_JIS). These
charsets are not
+ * resolvable by {@link Charset#forName}; APIs that take a charset name use
+ * {@link #jdkDecodeAs(Charset)}.</p>
*/
public final class CharsetSupersets {
/**
* Maps detected charset canonical names (case-sensitive, as returned by
- * {@link Charset#name()}) to their superset charset canonical name.
+ * {@link Charset#name()}) to their JDK superset charset canonical name,
before the
+ * browser patch.
*/
public static final Map<String, String> SUPERSET_MAP;
@@ -73,6 +85,19 @@ public final class CharsetSupersets {
SUPERSET_MAP = Collections.unmodifiableMap(m);
}
+ /** Browser patches, by the JDK charset they patch. */
+ private static final Map<String, Charset> PATCHED;
+
+ static {
+ Map<String, Charset> m = new HashMap<>();
+ for (String table : new String[]{"x-Big5-HKSCS-WHATWG.tsv",
+ "x-eucJP-Open-WHATWG.tsv"}) {
+ PatchedCharset cs = PatchedCharset.load(table);
+ m.put(cs.base().name(), cs);
+ }
+ PATCHED = Collections.unmodifiableMap(m);
+ }
+
private CharsetSupersets() {
}
@@ -84,6 +109,34 @@ public final class CharsetSupersets {
* @return superset charset, or {@code null} if none is defined
*/
public static Charset supersetOf(Charset detected) {
+ if (detected == null) {
+ return null;
+ }
+ Charset jdk = jdkSupersetOf(detected);
+ Charset patched = PATCHED.get(jdk != null ? jdk.name() :
detected.name());
+ return patched != null ? patched : jdk;
+ }
+
+ /**
+ * As {@link #decodeAs(Charset)}, but always a JDK charset that {@link
Charset#forName}
+ * resolves (the browser patch left off): for APIs that take a charset
name.
+ */
+ public static Charset jdkDecodeAs(Charset detected) {
+ Charset jdk = jdkSupersetOf(detected);
+ return jdk != null ? jdk : detected;
+ }
+
+ /** The browser-patched charsets. */
+ static Collection<Charset> patched() {
+ return PATCHED.values();
+ }
+
+ /** {@code decodeAs} itself, or the JDK base of a browser-patched charset.
*/
+ static Charset jdkCharset(Charset decodeAs) {
+ return decodeAs instanceof PatchedCharset ? ((PatchedCharset)
decodeAs).base() : decodeAs;
+ }
+
+ private static Charset jdkSupersetOf(Charset detected) {
if (detected == null) {
return null;
}
diff --git a/tika-core/src/main/java/org/apache/tika/detect/PatchedCharset.java
b/tika-core/src/main/java/org/apache/tika/detect/PatchedCharset.java
new file mode 100644
index 0000000000..35248924ac
--- /dev/null
+++ b/tika-core/src/main/java/org/apache/tika/detect/PatchedCharset.java
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.detect;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.nio.CharBuffer;
+import java.nio.charset.Charset;
+import java.nio.charset.CharsetDecoder;
+import java.nio.charset.CharsetEncoder;
+import java.nio.charset.CoderResult;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A JDK charset with some byte sequences decoded differently, e.g. as the
WHATWG Encoding
+ * Standard (browsers) decodes them. Everything not in the patch table,
including invalid and
+ * unmappable input, is the base decoder's to decode or report; encoding is
the base's.
+ *
+ * <p>The table is a resource written by {@code WhatwgPatchGenerator}
(tika-core tests):
+ * {@code name}, {@code base} and optional {@code lead} and {@code lead3}
header lines, then one
+ * {@code HEXBYTES<TAB>U+XXXX[ U+XXXX...]} line per patched sequence. {@code
lead} lists the
+ * bytes the base always reads as a unit with the next byte (e.g. {@code
80-FF}), {@code lead3}
+ * those it reads with the next two; every other byte is a unit by itself.
Patched sequences are
+ * one- or two-byte units.
+ *
+ * <p>Not registered with {@link Charset#forName}; obtain instances from
+ * {@link CharsetSupersets}.
+ */
+final class PatchedCharset extends Charset {
+
+ private final Charset base;
+
+ /** Bytes in the base's unit that starts with this byte: 1, 2 or 3. */
+ private final byte[] unitLength = new byte[256];
+
+ private final String[] single = new String[256];
+
+ /** 1 + index into {@link #pairValues}, by {@code lead << 8 | trail}; null
if no pairs. */
+ private final short[] pairSlot;
+
+ private final String[] pairValues;
+
+ private final int[] patched;
+
+ private PatchedCharset(String name, Charset base, byte[] unitLength,
List<int[]> keys,
+ List<String> values) {
+ super(name, null);
+ this.base = base;
+ System.arraycopy(unitLength, 0, this.unitLength, 0, 256);
+ List<String> pv = new ArrayList<>();
+ short[] slot = null;
+ patched = new int[keys.size()];
+ for (int i = 0; i < keys.size(); i++) {
+ int[] k = keys.get(i);
+ if (k.length == 1) {
+ patched[i] = k[0];
+ single[k[0]] = values.get(i);
+ } else {
+ patched[i] = k[0] << 8 | k[1];
+ if (slot == null) {
+ slot = new short[0x10000];
+ }
+ pv.add(values.get(i));
+ slot[patched[i]] = (short) pv.size();
+ }
+ }
+ pairSlot = slot;
+ pairValues = pv.toArray(new String[0]);
+ }
+
+ /** Loads a patch table resource next to this class. */
+ static PatchedCharset load(String resource) {
+ try (InputStream in =
PatchedCharset.class.getResourceAsStream(resource)) {
+ if (in == null) {
+ throw new IllegalStateException("missing charset patch " +
resource);
+ }
+ return parse(new BufferedReader(new InputStreamReader(in,
StandardCharsets.UTF_8)),
+ resource);
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);
+ }
+ }
+
+ private static PatchedCharset parse(BufferedReader reader, String resource)
+ throws IOException {
+ String name = null;
+ Charset base = null;
+ byte[] unitLength = new byte[256];
+ Arrays.fill(unitLength, (byte) 1);
+ List<int[]> keys = new ArrayList<>();
+ List<String> values = new ArrayList<>();
+ String line;
+ while ((line = reader.readLine()) != null) {
+ if (line.isEmpty() || line.startsWith("#")) {
+ continue;
+ }
+ String[] f = line.split("\t");
+ if (f.length != 2) {
+ throw new IllegalStateException(resource + ": bad line " +
line);
+ }
+ switch (f[0]) {
+ case "name":
+ name = f[1];
+ break;
+ case "base":
+ base = Charset.forName(f[1]);
+ break;
+ case "lead":
+ case "lead3":
+ for (String range : f[1].split(",")) {
+ String[] lh = range.split("-");
+ int lo = Integer.parseInt(lh[0], 16);
+ int hi = Integer.parseInt(lh[lh.length - 1], 16);
+ for (int b = lo; b <= hi; b++) {
+ unitLength[b] = (byte) (f[0].equals("lead") ? 2 :
3);
+ }
+ }
+ break;
+ default:
+ keys.add(key(f[0], unitLength, resource));
+ values.add(value(f[1]));
+ }
+ }
+ if (name == null || base == null) {
+ throw new IllegalStateException(resource + ": needs name and
base");
+ }
+ return new PatchedCharset(name, base, unitLength, keys, values);
+ }
+
+ private static int[] key(String hex, byte[] unitLength, String resource) {
+ int n = hex.length() / 2;
+ int[] k = new int[n];
+ for (int i = 0; i < n; i++) {
+ k[i] = Integer.parseInt(hex.substring(2 * i, 2 * i + 2), 16);
+ }
+ if (hex.length() % 2 != 0 || n > 2 || n != unitLength[k[0]]) {
+ throw new IllegalStateException(resource + ": " + hex + " is not a
1- or 2-byte unit");
+ }
+ return k;
+ }
+
+ private static String value(String codePoints) {
+ StringBuilder sb = new StringBuilder();
+ for (String cp : codePoints.split(" ")) {
+ sb.appendCodePoint(Integer.parseInt(cp.substring(2), 16));
+ }
+ return sb.toString();
+ }
+
+ /** The JDK charset this patches. */
+ Charset base() {
+ return base;
+ }
+
+ /** Bytes in the base's unit that starts with {@code b}. */
+ int unitLength(int b) {
+ return unitLength[b & 0xFF];
+ }
+
+ /** The patched sequences: a byte, or {@code lead << 8 | trail}. */
+ int[] patchedUnits() {
+ return patched.clone();
+ }
+
+ /** The patched decode of a one-byte unit, or {@code null}. */
+ String patch(int b) {
+ return single[b & 0xFF];
+ }
+
+ /** The patched decode of a two-byte unit, or {@code null}. */
+ String patch(int lead, int trail) {
+ if (pairSlot == null) {
+ return null;
+ }
+ int s = pairSlot[(lead & 0xFF) << 8 | (trail & 0xFF)];
+ return s == 0 ? null : pairValues[s - 1];
+ }
+
+ @Override
+ public boolean contains(Charset cs) {
+ return cs == this || base.contains(cs);
+ }
+
+ @Override
+ public CharsetDecoder newDecoder() {
+ return new Decoder(this);
+ }
+
+ @Override
+ public CharsetEncoder newEncoder() {
+ return base.newEncoder();
+ }
+
+ private static final class Decoder extends CharsetDecoder {
+
+ private final PatchedCharset cs;
+
+ private final CharsetDecoder base;
+
+ private Decoder(PatchedCharset cs) {
+ super(cs, cs.base.newDecoder().averageCharsPerByte(),
+ cs.base.newDecoder().maxCharsPerByte());
+ this.cs = cs;
+ this.base = cs.base.newDecoder();
+ }
+
+ @Override
+ protected CoderResult decodeLoop(ByteBuffer in, CharBuffer out) {
+ while (true) {
+ int start = in.position();
+ int limit = in.limit();
+ // Walk the base's units so a patched sequence is only matched
at a unit boundary.
+ int i = start;
+ int unit = 0;
+ String value = null;
+ while (i < limit) {
+ int b = in.get(i) & 0xFF;
+ int n = cs.unitLength[b];
+ if (n == 1) {
+ value = cs.single[b];
+ } else if (i + n > limit) {
+ break;
+ } else if (n == 2) {
+ value = cs.patch(b, in.get(i + 1));
+ }
+ if (value != null) {
+ unit = n;
+ break;
+ }
+ i += n;
+ }
+ if (i > start) {
+ ByteBuffer run = in.duplicate();
+ run.limit(i);
+ CoderResult cr = base.decode(run, out, false);
+ in.position(run.position());
+ if (cr.isError() || cr.isOverflow() || run.hasRemaining())
{
+ return cr;
+ }
+ }
+ if (value == null) {
+ return CoderResult.UNDERFLOW;
+ }
+ if (out.remaining() < value.length()) {
+ return CoderResult.OVERFLOW;
+ }
+ out.put(value);
+ in.position(i + unit);
+ }
+ }
+
+ @Override
+ protected void implReset() {
+ base.reset();
+ }
+ }
+}
diff --git
a/tika-core/src/main/resources/org/apache/tika/detect/x-Big5-HKSCS-WHATWG.tsv
b/tika-core/src/main/resources/org/apache/tika/detect/x-Big5-HKSCS-WHATWG.tsv
new file mode 100644
index 0000000000..32d860e187
--- /dev/null
+++
b/tika-core/src/main/resources/org/apache/tika/detect/x-Big5-HKSCS-WHATWG.tsv
@@ -0,0 +1,177 @@
+# Big5-HKSCS decoded as the WHATWG Encoding Standard big5 decoder, where they
differ.
+# Generated by WhatwgPatchGenerator from
https://encoding.spec.whatwg.org/index-big5.txt
+# Identifier: 8dfc771062e7be0810919082c2c06baa2236147909e0ecc235b1cb9ad782ac82
+# Date: 2024-09-18
+#
+# Copyright (c) WHATWG (Apple, Google, Mozilla, Microsoft).
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+# 1. Redistributions of source code must retain the above copyright notice,
+# this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+# this list of conditions and the following disclaimer in the documentation
+# and/or other materials provided with the distribution.
+# 3. Neither the name of the copyright holder nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+#
+name x-Big5-HKSCS-WHATWG
+base Big5-HKSCS
+lead 80-FF
+8862 U+00CA U+0304
+8864 U+00CA U+030C
+88A3 U+00EA U+0304
+88A5 U+00EA U+030C
+8E69 U+7BB8
+8E6F U+7C06
+8E7E U+7CCE
+8EAB U+7DD2
+8EB4 U+7E1D
+8ECD U+8005
+8ED0 U+8028
+8F57 U+83C1
+8F69 U+84A8
+8F6E U+840F
+8FCB U+89A6
+8FCC U+89A9
+8FFE U+8D77
+906D U+90FD
+907A U+92B9
+90DC U+975C
+90F1 U+97FF
+91BF U+9F16
+9244 U+8503
+92AF U+5159
+92B0 U+515B
+92B1 U+515D
+92B2 U+515E
+92C8 U+936E
+92D1 U+7479
+9447 U+6D67
+94CA U+799B
+95D9 U+9097
+9644 U+975D
+96ED U+701E
+96FC U+5B28
+9B76 U+7201
+9B78 U+77D7
+9B7B U+7E87
+9BC6 U+99D6
+9BDE U+91D4
+9BEC U+60DE
+9BF6 U+6FB6
+9C42 U+8F36
+9C53 U+4FBB
+9C62 U+71DF
+9C68 U+9104
+9C6B U+9DF0
+9C77 U+83CF
+9CBC U+5C10
+9CBD U+79E3
+9CD0 U+5A67
+9D57 U+8F0B
+9D5A U+7B51
+9DC4 U+62D0
+9EA9 U+6062
+9EEF U+75F9
+9EFD U+6C4A
+9F60 U+9B2E
+9F66 U+9F17
+9FCB U+50ED
+9FD8 U+5F0C
+A063 U+880F
+A077 U+62CE
+A0D5 U+7468
+A0DF U+7162
+A0E4 U+7250
+A145 U+2027
+A14E U+FE51
+A15A U+2574
+A1C2 U+00AF
+A1C3 U+FFE3
+A1C5 U+02CD
+A1E3 U+FF5E
+A1F2 U+2295
+A1F3 U+2299
+A1FE U+FF0F
+A240 U+FF3C
+A241 U+2215
+A242 U+FE68
+A244 U+FFE5
+A246 U+FFE0
+A247 U+FFE1
+A3C0 U+2400
+A3C1 U+2401
+A3C2 U+2402
+A3C3 U+2403
+A3C4 U+2404
+A3C5 U+2405
+A3C6 U+2406
+A3C7 U+2407
+A3C8 U+2408
+A3C9 U+2409
+A3CA U+240A
+A3CB U+240B
+A3CC U+240C
+A3CD U+240D
+A3CE U+240E
+A3CF U+240F
+A3D0 U+2410
+A3D1 U+2411
+A3D2 U+2412
+A3D3 U+2413
+A3D4 U+2414
+A3D5 U+2415
+A3D6 U+2416
+A3D7 U+2417
+A3D8 U+2418
+A3D9 U+2419
+A3DA U+241A
+A3DB U+241B
+A3DC U+241C
+A3DD U+241D
+A3DE U+241E
+A3DF U+241F
+A3E0 U+2421
+A3E1 U+20AC
+C6CF U+5EF4
+C6D3 U+65E0
+C6D5 U+7676
+C6D7 U+96B6
+C6DE U+3003
+C6DF U+4EDD
+FA5F U+5029
+FA66 U+507D
+FABD U+5305
+FAC5 U+5344
+FAD5 U+537F
+FB48 U+5605
+FBB8 U+5A77
+FBF3 U+5E75
+FBF9 U+5ED0
+FC4F U+5F58
+FC6C U+60A4
+FCB9 U+6490
+FCE2 U+6674
+FCF1 U+675E
+FDB7 U+6C9C
+FDB8 U+6E1D
+FDBB U+6E2F
+FDF1 U+716E
+FE52 U+732A
+FE6F U+745C
+FEAA U+74E9
+FEDD U+7809
diff --git
a/tika-core/src/main/resources/org/apache/tika/detect/x-eucJP-Open-WHATWG.tsv
b/tika-core/src/main/resources/org/apache/tika/detect/x-eucJP-Open-WHATWG.tsv
new file mode 100644
index 0000000000..74cde9f55f
--- /dev/null
+++
b/tika-core/src/main/resources/org/apache/tika/detect/x-eucJP-Open-WHATWG.tsv
@@ -0,0 +1,43 @@
+# x-eucJP-Open decoded as the WHATWG Encoding Standard euc-jp decoder, where
they differ.
+# Generated by WhatwgPatchGenerator from
https://encoding.spec.whatwg.org/index-jis0208.txt
+# Identifier: cbaa91f3deb7d0841faf5c33041fc15a285da0e87e64ab802c4bf04b7c4da861
+# Date: 2024-09-18
+#
+# Copyright (c) WHATWG (Apple, Google, Mozilla, Microsoft).
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+# 1. Redistributions of source code must retain the above copyright notice,
+# this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright notice,
+# this list of conditions and the following disclaimer in the documentation
+# and/or other materials provided with the distribution.
+# 3. Neither the name of the copyright holder nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+#
+name x-eucJP-Open-WHATWG
+base x-eucJP-Open
+lead 80-8E,90-FF
+lead3 8F-8F
+A1BD U+2015
+A1C1 U+FF5E
+A1C2 U+2225
+A1DD U+FF0D
+A1F1 U+FFE0
+A1F2 U+FFE1
+A2CC U+FFE2
+FCFB U+FFE2
+FCFC U+FFE4
diff --git
a/tika-core/src/test/java/org/apache/tika/detect/CharsetSupersetsTest.java
b/tika-core/src/test/java/org/apache/tika/detect/CharsetSupersetsTest.java
index d8cfd6d9e5..e4074ba739 100644
--- a/tika-core/src/test/java/org/apache/tika/detect/CharsetSupersetsTest.java
+++ b/tika-core/src/test/java/org/apache/tika/detect/CharsetSupersetsTest.java
@@ -34,11 +34,13 @@ public class CharsetSupersetsTest {
@Test
public void mapsLegacyCjkToVendorSupersets() {
assertEquals("x-windows-949", name("EUC-KR"));
- assertEquals("Big5-HKSCS", name("Big5"));
+ assertEquals("x-Big5-HKSCS-WHATWG", name("Big5"));
+ assertEquals("x-Big5-HKSCS-WHATWG", name("Big5-HKSCS"));
assertEquals("GB18030", name("GB2312"));
assertEquals("GB18030", name("GBK"));
assertEquals("windows-31j", name("Shift_JIS"));
- assertEquals("x-eucJP-Open", name("EUC-JP"));
+ assertEquals("x-eucJP-Open-WHATWG", name("EUC-JP"));
+ assertEquals("x-eucJP-Open-WHATWG", name("x-eucJP-Open"));
assertEquals("x-windows-50221", name("ISO-2022-JP"));
}
@@ -49,6 +51,17 @@ public class CharsetSupersetsTest {
assertNull(CharsetSupersets.supersetOf(Charset.forName("windows-1252")));
// Superset targets have no further superset.
assertNull(CharsetSupersets.supersetOf(Charset.forName("GB18030")));
+
assertNull(CharsetSupersets.supersetOf(CharsetSupersets.decodeAs(Charset.forName("Big5"))));
+ }
+
+ @Test
+ public void jdkDecodeAsLeavesBrowserPatchOff() {
+ assertEquals("Big5-HKSCS",
CharsetSupersets.jdkDecodeAs(Charset.forName("Big5")).name());
+ assertEquals("Big5-HKSCS",
+
CharsetSupersets.jdkDecodeAs(Charset.forName("Big5-HKSCS")).name());
+ assertEquals("windows-31j",
+
CharsetSupersets.jdkDecodeAs(Charset.forName("Shift_JIS")).name());
+ assertEquals(StandardCharsets.UTF_8,
CharsetSupersets.jdkDecodeAs(StandardCharsets.UTF_8));
}
/** The point of the map: vendor-extension bytes the strict base drops to
@@ -63,7 +76,8 @@ public class CharsetSupersetsTest {
byte[] eucjp = {(byte) 0xAD, (byte) 0xA1};
assertEquals('\uFFFD', new String(eucjp,
Charset.forName("EUC-JP")).charAt(0));
- assertEquals("\u2460", new String(eucjp,
Charset.forName(name("EUC-JP"))));
+ assertEquals("\u2460", new String(eucjp,
+ CharsetSupersets.supersetOf(Charset.forName("EUC-JP"))));
byte[] jis =
"\u65e5\u2460".getBytes(Charset.forName("x-windows-50221"));
assertEquals('\uFFFD', new String(jis,
Charset.forName("ISO-2022-JP")).charAt(1));
diff --git
a/tika-core/src/test/java/org/apache/tika/detect/DecodeAsNameUsageTest.java
b/tika-core/src/test/java/org/apache/tika/detect/DecodeAsNameUsageTest.java
new file mode 100644
index 0000000000..a22c66541c
--- /dev/null
+++ b/tika-core/src/test/java/org/apache/tika/detect/DecodeAsNameUsageTest.java
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.detect;
+
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.junit.jupiter.api.Test;
+
+/**
+ * A decode-as charset may be a {@link PatchedCharset}, whose name {@link
java.nio.charset.Charset#forName}
+ * and name-taking library APIs cannot resolve. Fails on main sources anywhere
in the repo that
+ * take the name straight off a decode-as call; use {@link
CharsetSupersets#jdkDecodeAs} for
+ * those APIs. forbidden-apis bans signatures, not call chains, so this scans
source.
+ */
+public class DecodeAsNameUsageTest {
+
+ // getDecodeAs() / decodeAs(...) / supersetOf(...), then .name() /
.displayName() / .toString()
+ private static final Pattern NAME_OF_DECODE_AS = Pattern.compile(
+
"\\b(?:getDecodeAs|decodeAs|supersetOf)\\s*\\((?:[^()]|\\([^()]*\\))*\\)\\s*\\.\\s*"
+ + "(?:name|displayName|toString)\\s*\\(");
+
+ private static final String SRC_MAIN =
+ File.separator + "src" + File.separator + "main" + File.separator
+ "java"
+ + File.separator;
+
+ @Test
+ public void noDecodeAsNameInMainSources() throws IOException {
+ Path root = repoRoot();
+ assumeTrue(root != null, "repo root (tika-core + tika-parsers) not
found");
+ List<String> hits = new ArrayList<>();
+ Files.walkFileTree(root, new SimpleFileVisitor<Path>() {
+ @Override
+ public FileVisitResult preVisitDirectory(Path dir,
BasicFileAttributes attrs) {
+ String n = dir.getFileName() == null ? "" :
dir.getFileName().toString();
+ return !dir.equals(root) && (n.startsWith(".") ||
n.equals("target")
+ || n.equals("node_modules")) ?
FileVisitResult.SKIP_SUBTREE
+ : FileVisitResult.CONTINUE;
+ }
+
+ @Override
+ public FileVisitResult visitFile(Path p, BasicFileAttributes
attrs) throws IOException {
+ if (p.toString().endsWith(".java") &&
p.toString().contains(SRC_MAIN)) {
+ String src = Files.readString(p, StandardCharsets.UTF_8);
+ Matcher m = NAME_OF_DECODE_AS.matcher(src);
+ while (m.find()) {
+ hits.add(root.relativize(p) + ":" + line(src,
m.start()) + ": "
+ + m.group().replaceAll("\\s+", " "));
+ }
+ }
+ return FileVisitResult.CONTINUE;
+ }
+ });
+ assertTrue(hits.isEmpty(), "A decode-as charset's name may not resolve
by name; use "
+ + "CharsetSupersets.jdkDecodeAs(detected) for APIs that take a
charset name:\n "
+ + String.join("\n ", hits));
+ }
+
+ @Test
+ public void patternCatchesTheShapes() {
+ String[] bad = {"r.get(0).getDecodeAs().name()",
"CharsetSupersets.decodeAs(cs).name()",
+ "decodeAs(Charset.forName(x)) .\n name()",
"supersetOf(c).toString()",
+ "result.getDecodeAs().displayName()"};
+ for (String s : bad) {
+ assertTrue(NAME_OF_DECODE_AS.matcher(s).find(), s);
+ }
+ String[] ok = {"CharsetSupersets.jdkDecodeAs(cs).name()",
"decodeAs.name()",
+ "getDecodeAs()"};
+ for (String s : ok) {
+ assertTrue(!NAME_OF_DECODE_AS.matcher(s).find(), s);
+ }
+ }
+
+ private static int line(String src, int offset) {
+ int n = 1;
+ for (int i = 0; i < offset; i++) {
+ if (src.charAt(i) == '\n') {
+ n++;
+ }
+ }
+ return n;
+ }
+
+ private static Path repoRoot() {
+ Path p = Paths.get("").toAbsolutePath();
+ while (p != null) {
+ if (Files.isDirectory(p.resolve("tika-core"))
+ && Files.isDirectory(p.resolve("tika-parsers"))) {
+ return p;
+ }
+ p = p.getParent();
+ }
+ return null;
+ }
+}
diff --git
a/tika-core/src/test/java/org/apache/tika/detect/PatchedCharsetTest.java
b/tika-core/src/test/java/org/apache/tika/detect/PatchedCharsetTest.java
new file mode 100644
index 0000000000..fcff3fa1fb
--- /dev/null
+++ b/tika-core/src/test/java/org/apache/tika/detect/PatchedCharsetTest.java
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.detect;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Reader;
+import java.nio.charset.Charset;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HexFormat;
+import java.util.Random;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.metadata.TikaCoreProperties;
+
+public class PatchedCharsetTest {
+
+ private static final Charset HKSCS = Charset.forName("Big5-HKSCS");
+ private static final Charset BIG5 =
CharsetSupersets.decodeAs(Charset.forName("Big5"));
+ private static final Charset EUCJP =
CharsetSupersets.decodeAs(Charset.forName("EUC-JP"));
+
+ private static String dec(Charset cs, int... bytes) {
+ byte[] b = new byte[bytes.length];
+ for (int i = 0; i < b.length; i++) {
+ b[i] = (byte) bytes[i];
+ }
+ return new String(b, cs);
+ }
+
+ @Test
+ public void tablesLoad() {
+ assertFalse(CharsetSupersets.patched().isEmpty());
+ assertTrue(BIG5 instanceof PatchedCharset);
+ assertEquals("x-Big5-HKSCS-WHATWG", BIG5.name());
+ assertEquals(144, ((PatchedCharset) BIG5).patchedUnits().length);
+ assertEquals("x-eucJP-Open-WHATWG", EUCJP.name());
+ assertEquals(9, ((PatchedCharset) EUCJP).patchedUnits().length);
+ }
+
+ /** The table's unit model is the base decoder's: what makes unit-boundary
matching sound. */
+ @Test
+ public void unitLengthsMatchTheBase() {
+ for (Charset c : CharsetSupersets.patched()) {
+ PatchedCharset cs = (PatchedCharset) c;
+ int[] len = WhatwgPatchGenerator.unitLengths(cs.base());
+ assertNotNull(len, cs.name());
+ for (int b = 0; b < 256; b++) {
+ assertEquals(len[b], cs.unitLength(b), cs.name() + " " +
Integer.toHexString(b));
+ }
+ }
+ }
+
+ /**
+ * Every one- and two-byte input, followed by ASCII: the base decodes it
unit by unit, and the
+ * patched charset differs only on patched units.
+ */
+ @Test
+ public void everyShortInputIsPatchOrBase() {
+ for (Charset c : CharsetSupersets.patched()) {
+ PatchedCharset cs = (PatchedCharset) c;
+ for (int b1 = 0; b1 <= 0xFF; b1++) {
+ byte[] one = {(byte) b1, 'A'};
+ assertEquals(new String(one, cs.base()), unitwise(cs, one,
false));
+ assertEquals(unitwise(cs, one, true), new String(one, cs));
+ for (int b2 = 0; b2 <= 0xFF; b2++) {
+ byte[] two = {(byte) b1, (byte) b2, 'A'};
+ String hex = HexFormat.of().formatHex(two);
+ assertEquals(new String(two, cs.base()), unitwise(cs, two,
false), hex);
+ assertEquals(unitwise(cs, two, true), new String(two, cs),
hex);
+ if (cs.unitLength(b1) == 3) {
+ for (int b3 = 0; b3 <= 0xFF; b3++) {
+ byte[] three = {(byte) b1, (byte) b2, (byte) b3,
'A'};
+ assertEquals(new String(three, cs.base()),
unitwise(cs, three, true),
+ HexFormat.of().formatHex(three));
+ assertEquals(unitwise(cs, three, true), new
String(three, cs));
+ }
+ }
+ }
+ }
+ }
+ }
+
+ @Test
+ public void everyPatchChangesTheDecode() {
+ for (Charset c : CharsetSupersets.patched()) {
+ PatchedCharset cs = (PatchedCharset) c;
+ for (int unit : cs.patchedUnits()) {
+ byte[] b = unit > 0xFF ? new byte[]{(byte) (unit >>> 8),
(byte) unit}
+ : new byte[]{(byte) unit};
+ assertFalse(new String(b, cs).equals(new String(b, cs.base())),
+ cs.name() + " " + Integer.toHexString(unit));
+ }
+ }
+ }
+
+ /** Values from the WHATWG index (pointer = (lead - 0x81) * 157 + trail
offset), by hand. */
+ @Test
+ public void big5SpotChecksAgainstWhatwgIndex() {
+ assertEquals("€", dec(BIG5, 0xA3, 0xE1)); // pointer 5465
+ assertEquals("‧", dec(BIG5, 0xA1, 0x45)); // 5029
+ assertEquals("～", dec(BIG5, 0xA1, 0xE3)); // 5153
+ assertEquals("／", dec(BIG5, 0xA1, 0xFE)); // 5180
+ assertEquals("廴", dec(BIG5, 0xC6, 0xCF)); // 10942
+ assertEquals("箸", dec(BIG5, 0x8E, 0x69)); // 2082
+ assertEquals("Ê̄", dec(BIG5, 0x88, 0x62)); // 1133, two code points
+ assertEquals("ê̌", dec(BIG5, 0x88, 0xA5)); // 1166
+ assertEquals("中", dec(BIG5, 0xA4, 0xA4)); // unpatched
+ assertNull(((PatchedCharset) BIG5).patch(0xA4, 0xA4));
+ }
+
+ /** EUC-JP gets the forms windows-31j gives the same JIS X 0208 characters
in Shift_JIS. */
+ @Test
+ public void eucJpMatchesWindows31j() {
+ Charset sjis = Charset.forName("windows-31j");
+ assertEquals("～", dec(EUCJP, 0xA1, 0xC1));
+ assertEquals(dec(sjis, 0x81, 0x60), dec(EUCJP, 0xA1, 0xC1)); // wave
dash
+ assertEquals(dec(sjis, 0x81, 0x7C), dec(EUCJP, 0xA1, 0xDD)); // minus
+ assertEquals(dec(sjis, 0x81, 0x91), dec(EUCJP, 0xA1, 0xF1)); // cent
+ assertEquals(dec(sjis, 0x81, 0xCA), dec(EUCJP, 0xA2, 0xCC)); // not
sign
+ assertEquals("日本", dec(EUCJP, 0xC6, 0xFC, 0xCB, 0xDC));
+ // JIS X 0212 (three bytes) is the base's, patched pair or not after it
+ assertEquals(dec(Charset.forName("x-eucJP-Open"), 0x8F, 0xB0, 0xA1) +
"～",
+ dec(EUCJP, 0x8F, 0xB0, 0xA1, 0xA1, 0xC1));
+ assertEquals("\u2170", dec(EUCJP, 0x8F, 0xF3, 0xF3));
+ assertEquals("\uFFFD", dec(EUCJP, 0x8F, 0xA1, 0xC1));
+ }
+
+ @Test
+ public void big5LoneLeadAtEndIsBaseMalformed() {
+ assertEquals("a�", dec(HKSCS, 'a', 0xA3));
+ assertEquals("a�", dec(BIG5, 'a', 0xA3));
+ assertEquals("€�", dec(BIG5, 0xA3, 0xE1, 0xA3));
+ }
+
+ /** Random mixes of patched units, ASCII and arbitrary bytes, read in
random chunks. */
+ @Test
+ public void streamedDecodeMatchesUnitwiseReference() throws IOException {
+ long seed = new Random().nextLong();
+ Random r = new Random(seed);
+ for (Charset c : CharsetSupersets.patched()) {
+ PatchedCharset cs = (PatchedCharset) c;
+ int[] units = cs.patchedUnits();
+ for (int iter = 0; iter < 300; iter++) {
+ byte[] b = new byte[1 + r.nextInt(20000)];
+ for (int i = 0; i < b.length; i++) {
+ int k = r.nextInt(10);
+ int u = units[r.nextInt(units.length)];
+ if (k < 3 && u > 0xFF && i + 1 < b.length) {
+ b[i++] = (byte) (u >>> 8);
+ b[i] = (byte) u;
+ } else if (k < 3) {
+ b[i] = (byte) u;
+ } else if (k < 6) {
+ b[i] = (byte) r.nextInt(0x80);
+ } else {
+ b[i] = (byte) r.nextInt(0x100);
+ }
+ }
+ String expected = unitwise(cs, b, true);
+ assertEquals(expected, new String(b, cs), cs.name() + " seed "
+ seed);
+ assertEquals(expected, read(new ChunkedStream(b, r), cs, 1 +
r.nextInt(64)),
+ cs.name() + " seed " + seed);
+ }
+ }
+ }
+
+ /** Lead byte of a patched pair as the last byte of every read and every
reader buffer. */
+ @Test
+ public void big5PatchedPairSplitAcrossBuffers() throws IOException {
+ for (int prefix = 0; prefix < 40; prefix++) {
+ byte[] b = new byte[prefix + 4];
+ Arrays.fill(b, 0, prefix, (byte) 'x');
+ b[prefix] = (byte) 0xA3;
+ b[prefix + 1] = (byte) 0xE1;
+ b[prefix + 2] = (byte) 0xA1;
+ b[prefix + 3] = (byte) 0x45;
+ String expected = "x".repeat(prefix) + "€‧";
+ for (int chunk = 1; chunk <= 8; chunk++) {
+ assertEquals(expected, read(new FixedChunkStream(b, chunk),
BIG5, chunk));
+ }
+ }
+ }
+
+ @Test
+ public void encodesAsTheBase() {
+ String s = "中文 abc";
+ assertArrayEquals(s.getBytes(HKSCS), s.getBytes(BIG5));
+ }
+
+ @Test
+ public void autoDetectReaderDecodesPatchedReportsJdkCharset() throws
Exception {
+ byte[] b = {'e', (byte) 0xA3, (byte) 0xE1};
+ EncodingDetector big5 = (tis, md, ctx) ->
+ Collections.singletonList(new
EncodingResult(Charset.forName("Big5"), 1.0f));
+ Metadata metadata = new Metadata();
+ try (AutoDetectReader reader =
+ new AutoDetectReader(TikaInputStream.get(b), metadata,
big5)) {
+ assertEquals("Big5-HKSCS", reader.getCharset().name());
+ assertEquals("e€", readAll(reader));
+ }
+ assertEquals("x-Big5-HKSCS-WHATWG",
metadata.get(TikaCoreProperties.DECODED_CHARSET));
+ }
+
+ /** Decodes unit by unit with the base, applying the patch table if {@code
patched}. */
+ private static String unitwise(PatchedCharset cs, byte[] b, boolean
patched) {
+ StringBuilder sb = new StringBuilder();
+ for (int i = 0; i < b.length; ) {
+ int n = Math.min(cs.unitLength(b[i]), b.length - i);
+ String patch = !patched || n != cs.unitLength(b[i]) ? null
+ : n == 1 ? cs.patch(b[i]) : n == 2 ? cs.patch(b[i], b[i +
1]) : null;
+ sb.append(patch != null ? patch : new String(b, i, n, cs.base()));
+ i += n;
+ }
+ return sb.toString();
+ }
+
+ private static String read(InputStream in, Charset cs, int bufSize) throws
IOException {
+ try (Reader reader = new InputStreamReader(in, cs)) {
+ StringBuilder sb = new StringBuilder();
+ char[] buf = new char[bufSize];
+ int n;
+ while ((n = reader.read(buf)) != -1) {
+ sb.append(buf, 0, n);
+ }
+ return sb.toString();
+ }
+ }
+
+ private static String readAll(Reader reader) throws IOException {
+ StringBuilder sb = new StringBuilder();
+ int c;
+ while ((c = reader.read()) != -1) {
+ sb.append((char) c);
+ }
+ return sb.toString();
+ }
+
+ private static final class ChunkedStream extends FilterInputStream {
+ private final Random r;
+
+ ChunkedStream(byte[] b, Random r) {
+ super(new ByteArrayInputStream(b));
+ this.r = r;
+ }
+
+ @Override
+ public int read(byte[] buf, int off, int len) throws IOException {
+ return super.read(buf, off, Math.min(len, 1 + r.nextInt(7)));
+ }
+ }
+
+ private static final class FixedChunkStream extends FilterInputStream {
+ private final int chunk;
+
+ FixedChunkStream(byte[] b, int chunk) {
+ super(new ByteArrayInputStream(b));
+ this.chunk = chunk;
+ }
+
+ @Override
+ public int read(byte[] buf, int off, int len) throws IOException {
+ return super.read(buf, off, Math.min(len, chunk));
+ }
+ }
+}
diff --git
a/tika-core/src/test/java/org/apache/tika/detect/WhatwgPatchGenerator.java
b/tika-core/src/test/java/org/apache/tika/detect/WhatwgPatchGenerator.java
new file mode 100644
index 0000000000..c26250c6f6
--- /dev/null
+++ b/tika-core/src/test/java/org/apache/tika/detect/WhatwgPatchGenerator.java
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.detect;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.CharBuffer;
+import java.nio.charset.Charset;
+import java.nio.charset.CharsetDecoder;
+import java.nio.charset.CoderResult;
+import java.nio.charset.CodingErrorAction;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HexFormat;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.TreeMap;
+
+/**
+ * Compares a WHATWG Encoding Standard decoder with a JDK charset, unit by
unit, and writes the
+ * difference as a {@link PatchedCharset} table. Not a test.
+ *
+ * <p>Download the index files from https://encoding.spec.whatwg.org/
(index-*.txt) into one
+ * directory, then run with {@code
tika-core/target/test-classes:tika-core/target/classes} on
+ * the classpath:
+ * <pre>
+ * WhatwgPatchGenerator ENCODING JDK_BASE INDEX_DIR [PATCHED_NAME]
+ * </pre>
+ * ENCODING is a WHATWG name: {@code big5}, {@code shift_jis}, {@code euc-jp},
{@code euc-kr},
+ * {@code gb18030}, or a single-byte encoding with an {@code
index-ENCODING.txt}. With
+ * PATCHED_NAME it prints the table resource; without, the differing units as
TSV. A summary by
+ * category goes to stderr either way.
+ *
+ * <p>Units are the byte sequences the WHATWG decoder reads as one code point
(or two, for the
+ * four big5 pointers that emit two). A pointer with no index entry decodes as
U+FFFD, followed
+ * by the trail when it is ASCII (the WHATWG decoder reads it again). The
table leaves error
+ * handling to the base: units whose WHATWG decode has a U+FFFD are not
patched.
+ */
+public final class WhatwgPatchGenerator {
+
+ private static final String FFFD = "�";
+
+ private WhatwgPatchGenerator() {
+ }
+
+ public static void main(String[] args) throws IOException {
+ String encoding = args[0];
+ Charset base = Charset.forName(args[1]);
+ Path dir = Paths.get(args[2]);
+ Map<String, String[]> diff = diff(encoding, base, dir);
+ Map<String, Integer> categories = new TreeMap<>();
+ for (String[] d : diff.values()) {
+ categories.merge(d[2], 1, Integer::sum);
+ }
+ System.err.println(encoding + " vs " + base.name() + ": " + diff.size()
+ + " differing units " + categories);
+ if (args.length > 3) {
+ System.out.print(resource(encoding, base, dir, args[3], diff));
+ } else {
+ for (Map.Entry<String, String[]> e : diff.entrySet()) {
+ System.out.println(e.getKey() + "\t" + cps(e.getValue()[0]) +
"\t"
+ + cps(e.getValue()[1]) + "\t" + e.getValue()[2]);
+ }
+ }
+ }
+
+ /** Differing units: hex bytes to {JDK decode, WHATWG decode, category}. */
+ static Map<String, String[]> diff(String encoding, Charset base, Path dir)
throws IOException {
+ Map<String, String[]> out = new LinkedHashMap<>();
+ for (Map.Entry<String, String> e : whatwgUnits(encoding,
dir).entrySet()) {
+ String jdk = decode(base, HexFormat.of().parseHex(e.getKey()));
+ String web = e.getValue();
+ if (!jdk.equals(web)) {
+ out.put(e.getKey(), new String[]{jdk, web, category(jdk,
web)});
+ }
+ }
+ return out;
+ }
+
+ /** Every unit the WHATWG decoder reads for {@code encoding}, hex bytes to
its decode. */
+ static Map<String, String> whatwgUnits(String encoding, Path dir) throws
IOException {
+ Map<String, String> u = new LinkedHashMap<>();
+ switch (encoding) {
+ case "big5": {
+ Map<Integer, String> index = index(dir, "big5");
+ index.put(1133, "Ê̄");
+ index.put(1135, "Ê̌");
+ index.put(1164, "ê̄");
+ index.put(1166, "ê̌");
+ for (int l = 0x81; l <= 0xFE; l++) {
+ for (int t = 0x40; t <= 0xFE; t++) {
+ if (t <= 0x7E || t >= 0xA1) {
+ int p = (l - 0x81) * 157 + (t < 0x7F ? t - 0x40 :
t - 0x62);
+ u.put(hex(l, t), index.getOrDefault(p, error(t)));
+ }
+ }
+ }
+ break;
+ }
+ case "shift_jis": {
+ Map<Integer, String> index = index(dir, "jis0208");
+ u.put(hex(0x80), "\u0080");
+ for (int b = 0xA1; b <= 0xDF; b++) {
+ u.put(hex(b), cp(0xFF61 + b - 0xA1));
+ }
+ for (int l = 0x81; l <= 0xFC; l++) {
+ if (l >= 0xA0 && l <= 0xDF) {
+ continue;
+ }
+ for (int t = 0x40; t <= 0xFC; t++) {
+ if (t == 0x7F) {
+ continue;
+ }
+ int p = (l - (l < 0xA0 ? 0x81 : 0xC1)) * 188 + t - (t
< 0x7F ? 0x40 : 0x41);
+ String v = p >= 8836 && p <= 10715 ? cp(0xE000 + p -
8836)
+ : index.getOrDefault(p, error(t));
+ u.put(hex(l, t), v);
+ }
+ }
+ break;
+ }
+ case "euc-jp": {
+ Map<Integer, String> jis0208 = index(dir, "jis0208");
+ Map<Integer, String> jis0212 = index(dir, "jis0212");
+ for (int b = 0xA1; b <= 0xDF; b++) {
+ u.put(hex(0x8E, b), cp(0xFF61 + b - 0xA1));
+ }
+ for (int l = 0xA1; l <= 0xFE; l++) {
+ for (int t = 0xA1; t <= 0xFE; t++) {
+ int p = (l - 0xA1) * 94 + t - 0xA1;
+ u.put(hex(l, t), jis0208.getOrDefault(p, FFFD));
+ u.put(hex(0x8F, l, t), jis0212.getOrDefault(p, FFFD));
+ }
+ }
+ break;
+ }
+ case "euc-kr": {
+ Map<Integer, String> index = index(dir, "euc-kr");
+ for (int l = 0x81; l <= 0xFE; l++) {
+ for (int t = 0x41; t <= 0xFE; t++) {
+ u.put(hex(l, t), index.getOrDefault((l - 0x81) * 190 +
t - 0x41, error(t)));
+ }
+ }
+ break;
+ }
+ case "gb18030": {
+ Map<Integer, String> index = index(dir, "gb18030");
+ TreeMap<Integer, Integer> ranges = new TreeMap<>();
+ for (Map.Entry<Integer, String> e : index(dir,
"gb18030-ranges").entrySet()) {
+ ranges.put(e.getKey(), e.getValue().codePointAt(0));
+ }
+ u.put(hex(0x80), "€");
+ for (int l = 0x81; l <= 0xFE; l++) {
+ for (int t = 0x40; t <= 0xFE; t++) {
+ if (t != 0x7F) {
+ int p = (l - 0x81) * 190 + t - (t < 0x7F ? 0x40 :
0x41);
+ u.put(hex(l, t), index.getOrDefault(p, error(t)));
+ }
+ }
+ }
+ for (int b1 = 0x81; b1 <= 0xFE; b1++) {
+ for (int b2 = 0x30; b2 <= 0x39; b2++) {
+ for (int b3 = 0x81; b3 <= 0xFE; b3++) {
+ for (int b4 = 0x30; b4 <= 0x39; b4++) {
+ int p = (b1 - 0x81) * 12600 + (b2 - 0x30) *
1260
+ + (b3 - 0x81) * 10 + b4 - 0x30;
+ String v;
+ if ((p > 39419 && p < 189000) || p > 1237575) {
+ v = FFFD;
+ } else if (p == 7457) {
+ v = "";
+ } else {
+ Map.Entry<Integer, Integer> r =
ranges.floorEntry(p);
+ v = cp(r.getValue() + p - r.getKey());
+ }
+ u.put(hex(b1, b2, b3, b4), v);
+ }
+ }
+ }
+ }
+ break;
+ }
+ default: {
+ Map<Integer, String> index = index(dir, encoding);
+ for (int b = 0x80; b <= 0xFF; b++) {
+ u.put(hex(b), index.getOrDefault(b - 0x80, FFFD));
+ }
+ }
+ }
+ return u;
+ }
+
+ /** WHATWG decode of a lead and unmapped trail: an ASCII trail is read
again as itself. */
+ private static String error(int trail) {
+ return trail < 0x80 ? FFFD + (char) trail : FFFD;
+ }
+
+ /** Bytes of the WHATWG unit starting at {@code i} (1 if not a multibyte
lead). */
+ static int unitLength(String encoding, byte[] b, int i) {
+ int x = b[i] & 0xFF;
+ int next = i + 1 < b.length ? b[i + 1] & 0xFF : -1;
+ switch (encoding) {
+ case "big5":
+ case "euc-kr":
+ return x >= 0x81 && x <= 0xFE ? 2 : 1;
+ case "shift_jis":
+ return (x >= 0x81 && x <= 0x9F) || (x >= 0xE0 && x <= 0xFC) ?
2 : 1;
+ case "euc-jp":
+ return x == 0x8F ? 3 : x == 0x8E || (x >= 0xA1 && x <= 0xFE) ?
2 : 1;
+ case "gb18030":
+ return x >= 0x81 && x <= 0xFE ? (next >= 0x30 && next <= 0x39
? 4 : 2) : 1;
+ default:
+ return 1;
+ }
+ }
+
+ /**
+ * Bytes in the unit {@code cs} reads starting with each byte (1, 2 or 3,
whatever follows),
+ * or {@code null} if some byte has no such fixed length.
+ */
+ static int[] unitLengths(Charset cs) {
+ int[] len = new int[256];
+ CharsetDecoder d = cs.newDecoder();
+ for (int b = 0; b < 256; b++) {
+ if (consumed(d, (byte) b) == 1) {
+ len[b] = 1;
+ continue;
+ }
+ int n = consumed(d, (byte) b, (byte) 0) == 2 ? 2 : 3;
+ for (int x = 0; x < 256; x++) {
+ if (n == 2 ? consumed(d, (byte) b, (byte) x) != 2
+ : !allConsume3(d, (byte) b, (byte) x)) {
+ return null;
+ }
+ }
+ len[b] = n;
+ }
+ return len;
+ }
+
+ private static boolean allConsume3(CharsetDecoder d, byte b, byte x) {
+ for (int y = 0; y < 256; y++) {
+ if (consumed(d, b, x, (byte) y) != 3) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /** Bytes the first unit of {@code b} takes (0 if it needs more input). */
+ private static int consumed(CharsetDecoder d, byte... b) {
+ for (int k = 1; k <= b.length; k++) {
+ d.reset();
+ ByteBuffer in = ByteBuffer.wrap(b, 0, k);
+ CoderResult cr = d.decode(in, CharBuffer.allocate(8), false);
+ if (cr.isError()) {
+ return in.position() + cr.length();
+ }
+ if (in.position() > 0) {
+ return in.position();
+ }
+ }
+ return 0;
+ }
+
+ private static String resource(String encoding, Charset base, Path dir,
String name,
+ Map<String, String[]> diff) throws
IOException {
+ int[] len = unitLengths(base);
+ if (len == null) {
+ throw new IllegalStateException(base.name() + " has units
PatchedCharset cannot walk");
+ }
+ StringBuilder sb = new StringBuilder();
+ String file = encoding.equals("shift_jis") ||
encoding.equals("euc-jp") ? "jis0208"
+ : encoding;
+ for (String line : Files.readAllLines(dir.resolve("index-" + file +
".txt"))) {
+ if (line.startsWith("# Identifier") || line.startsWith("# Date")) {
+ sb.append(line).append('\n');
+ }
+ }
+ sb.insert(0, String.format(Locale.ROOT, "# %s decoded as the WHATWG
Encoding Standard"
+ + " %s decoder, where they differ.%n# Generated by
WhatwgPatchGenerator from"
+ + " https://encoding.spec.whatwg.org/index-%s.txt%n";,
base.name(), encoding, file));
+ sb.append(BSD_NOTICE);
+ sb.append("name\t").append(name).append('\n');
+ sb.append("base\t").append(base.name()).append('\n');
+ for (int n = 2; n <= 3; n++) {
+ String ranges = ranges(len, n);
+ if (!ranges.isEmpty()) {
+ sb.append(n == 2 ? "lead" :
"lead3").append('\t').append(ranges).append('\n');
+ }
+ }
+ for (Map.Entry<String, String[]> e : diff.entrySet()) {
+ int first = Integer.parseInt(e.getKey().substring(0, 2), 16);
+ if (e.getKey().length() != 2 * len[first]) {
+ throw new IllegalStateException(e.getKey() + " is not one " +
base.name() + " unit");
+ }
+ if (e.getValue()[1].indexOf('\uFFFD') >= 0) {
+ // error handling stays the base's
+ continue;
+ }
+ if (len[first] > 2) {
+ throw new IllegalStateException(e.getKey() + ": PatchedCharset
patches 1- and 2-byte units");
+ }
+
sb.append(e.getKey()).append('\t').append(cps(e.getValue()[1])).append('\n');
+ }
+ return sb.toString();
+ }
+
+ private static String ranges(int[] len, int n) {
+ StringBuilder sb = new StringBuilder();
+ for (int b = 0; b < 256; b++) {
+ if (len[b] == n && (b == 0 || len[b - 1] != n)) {
+ int e = b;
+ while (e + 1 < 256 && len[e + 1] == n) {
+ e++;
+ }
+ sb.append(sb.length() > 0 ? "," : "")
+ .append(String.format(Locale.ROOT, "%02X-%02X", b, e));
+ }
+ }
+ return sb.toString();
+ }
+
+ private static String category(String jdk, String web) {
+ if (jdk.indexOf('\uFFFD') >= 0 && web.indexOf('\uFFFD') >= 0) {
+ // both report an error; they differ only in how many bytes it
takes
+ return "error handling";
+ }
+ if (jdk.chars().allMatch(c -> c == 0xFFFD)) {
+ return isC1(web) ? "FFFD->C1" : "FFFD->char";
+ }
+ if (web.equals(FFFD)) {
+ return isC1(jdk) ? "C1->FFFD" : "char->FFFD";
+ }
+ if (isC1(jdk)) {
+ return "C1->char";
+ }
+ if (isPua(jdk) || isPua(web)) {
+ return "PUA";
+ }
+ return "char->different char";
+ }
+
+ private static boolean isC1(String s) {
+ return s.length() == 1 && s.charAt(0) >= 0x80 && s.charAt(0) <= 0x9F;
+ }
+
+ private static boolean isPua(String s) {
+ return s.codePoints().anyMatch(c -> Character.getType(c) ==
Character.PRIVATE_USE);
+ }
+
+ private static Map<Integer, String> index(Path dir, String name) throws
IOException {
+ Map<Integer, String> m = new HashMap<>();
+ for (String line : Files.readAllLines(dir.resolve("index-" + name +
".txt"),
+ StandardCharsets.UTF_8)) {
+ if (line.startsWith("#") || line.isBlank()) {
+ continue;
+ }
+ String[] f = line.trim().split("\\s+");
+ m.put(Integer.parseInt(f[0]), cp(Integer.decode(f[1])));
+ }
+ return m;
+ }
+
+ private static String decode(Charset cs, byte[] b) {
+ try {
+ return cs.newDecoder().onMalformedInput(CodingErrorAction.REPLACE)
+ .onUnmappableCharacter(CodingErrorAction.REPLACE)
+ .decode(ByteBuffer.wrap(b)).toString();
+ } catch (IOException e) {
+ throw new IllegalStateException(e);
+ }
+ }
+
+ private static String hex(int... bytes) {
+ StringBuilder sb = new StringBuilder();
+ for (int b : bytes) {
+ sb.append(String.format(Locale.ROOT, "%02X", b));
+ }
+ return sb.toString();
+ }
+
+ private static String cp(int c) {
+ return new String(Character.toChars(c));
+ }
+
+ static String cps(String s) {
+ List<String> l = new ArrayList<>();
+ s.codePoints().forEach(c -> l.add(String.format(Locale.ROOT, "U+%04X",
c)));
+ return String.join(" ", l);
+ }
+
+ private static final String BSD_NOTICE = "#\n"
+ + "# Copyright (c) WHATWG (Apple, Google, Mozilla, Microsoft).\n"
+ + "#\n"
+ + "# Redistribution and use in source and binary forms, with or
without\n"
+ + "# modification, are permitted provided that the following
conditions are met:\n"
+ + "# 1. Redistributions of source code must retain the above
copyright notice,\n"
+ + "# this list of conditions and the following disclaimer.\n"
+ + "# 2. Redistributions in binary form must reproduce the above
copyright notice,\n"
+ + "# this list of conditions and the following disclaimer in
the documentation\n"
+ + "# and/or other materials provided with the distribution.\n"
+ + "# 3. Neither the name of the copyright holder nor the names of
its\n"
+ + "# contributors may be used to endorse or promote products
derived from\n"
+ + "# this software without specific prior written permission.\n"
+ + "#\n"
+ + "# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
CONTRIBUTORS \"AS IS\"\n"
+ + "# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE\n"
+ + "# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE\n"
+ + "# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
CONTRIBUTORS BE\n"
+ + "# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR\n"
+ + "# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF\n"
+ + "# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
OR BUSINESS\n"
+ + "# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN\n"
+ + "# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE)\n"
+ + "# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE\n"
+ + "# POSSIBILITY OF SUCH DAMAGE.\n"
+ + "#\n";
+}
diff --git
a/tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/JunkFilterEncodingDetector.java
b/tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/JunkFilterEncodingDetector.java
index 88eb6fd90f..cb03b3cb61 100644
---
a/tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/JunkFilterEncodingDetector.java
+++
b/tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/JunkFilterEncodingDetector.java
@@ -671,7 +671,7 @@ public class JunkFilterEncodingDetector implements
MetaEncodingDetector {
}
Long count = LEGACY_FREQUENCY.get(cs.name());
if (count == null) {
- count =
LEGACY_FREQUENCY.getOrDefault(CharsetSupersets.decodeAs(cs).name(), 0L);
+ count =
LEGACY_FREQUENCY.getOrDefault(CharsetSupersets.jdkDecodeAs(cs).name(), 0L);
}
if (count > bestCount) {
best = cs;
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-html-module/src/main/java/org/apache/tika/parser/html/JSoupParser.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-html-module/src/main/java/org/apache/tika/parser/html/JSoupParser.java
index 5e5da5456b..d04ddbb477 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-html-module/src/main/java/org/apache/tika/parser/html/JSoupParser.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-html-module/src/main/java/org/apache/tika/parser/html/JSoupParser.java
@@ -225,8 +225,14 @@ public class JSoupParser extends
AbstractEncodingDetectorParser {
Document document;
try {
BOMDetector.skipForeignBom(tis, decodeAs);
- document = Jsoup.parse(tis, decodeAs.name(), "",
- Parser.htmlParser().tagSet(tagSet));
+ Parser parser = Parser.htmlParser().tagSet(tagSet);
+ if (Charset.isSupported(decodeAs.name())) {
+ document = Jsoup.parse(tis, decodeAs.name(), "", parser);
+ } else {
+ // jsoup takes a charset by name; a decode-as charset outside
the JDK has none
+ document = parser.parseInput(new InputStreamReader(tis,
decodeAs), "");
+ document.outputSettings().charset(decodeAs);
+ }
} finally {
tis.removeCloseShield();
}
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-html-module/src/test/java/org/apache/tika/parser/html/HtmlParserTest.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-html-module/src/test/java/org/apache/tika/parser/html/HtmlParserTest.java
index b7365087fd..9060991c8f 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-html-module/src/test/java/org/apache/tika/parser/html/HtmlParserTest.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-html-module/src/test/java/org/apache/tika/parser/html/HtmlParserTest.java
@@ -323,6 +323,27 @@ public class HtmlParserTest extends TikaTest {
"no replacement chars expected under the MS949 superset
decode");
}
+ /** Big5 decodes as browsers do (A3E1 is the euro sign), through a charset
jsoup cannot name. */
+ @Test
+ public void testBig5DecodedWithWhatwgPatch() throws Exception {
+ byte[] head = "<html><head><meta charset=\"big5\"
/><title>t</title></head><body><p>"
+ .getBytes(UTF_8);
+ byte[] body = {(byte) 0xA3, (byte) 0xE1, (byte) 0xA4, (byte) 0xA4};
+ byte[] tail = "</p></body></html>".getBytes(UTF_8);
+ byte[] html = new byte[head.length + body.length + tail.length];
+ System.arraycopy(head, 0, html, 0, head.length);
+ System.arraycopy(body, 0, html, head.length, body.length);
+ System.arraycopy(tail, 0, html, head.length + body.length,
tail.length);
+ Metadata metadata = new Metadata();
+ BodyContentHandler handler = new BodyContentHandler();
+ try (TikaInputStream tis = TikaInputStream.get(html)) {
+ new JSoupParser().parse(tis, handler, metadata, new
ParseContext());
+ }
+ assertEquals("Big5", metadata.get(HttpHeaders.CONTENT_ENCODING));
+ assertEquals("x-Big5-HKSCS-WHATWG",
metadata.get(TikaCoreProperties.DECODED_CHARSET));
+ assertContains("\u20AC\u4E2D", handler.toString());
+ }
+
/**
* Test case for TIKA-334
*
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/OutlookExtractor.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/OutlookExtractor.java
index 3caa324415..c73608595e 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/OutlookExtractor.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/OutlookExtractor.java
@@ -67,6 +67,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.xml.sax.SAXException;
+import org.apache.tika.detect.CharsetSupersets;
import org.apache.tika.detect.DefaultEncodingDetector;
import org.apache.tika.detect.EncodingDetector;
import org.apache.tika.detect.EncodingResult;
@@ -867,7 +868,8 @@ public class OutlookExtractor extends
AbstractPOIFSExtractor {
try (TikaInputStream tis = TikaInputStream.get(body)) {
List<EncodingResult> results = encodingDetector.detect(tis,
metadata, context);
if (!results.isEmpty() && results.get(0).getConfidence() >
0.35f
- && tryToSet7BitEncoding(msg,
results.get(0).getDecodeAs().name())) {
+ && tryToSet7BitEncoding(msg,
+
CharsetSupersets.jdkDecodeAs(results.get(0).getCharset()).name())) {
return;
}
} catch (IOException e) {
diff --git
a/tika-server/tika-server-standard/src/main/appended-resources/META-INF/LICENSE
b/tika-server/tika-server-standard/src/main/appended-resources/META-INF/LICENSE
index 05c659a0d3..0cfff36b2d 100644
---
a/tika-server/tika-server-standard/src/main/appended-resources/META-INF/LICENSE
+++
b/tika-server/tika-server-standard/src/main/appended-resources/META-INF/LICENSE
@@ -186,6 +186,35 @@ ASM library (asm)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
THE POSSIBILITY OF SUCH DAMAGE.
+Big5 and EUC-JP decode patch tables in tika-core
(org/apache/tika/detect/*-WHATWG.tsv),
+derived from index-big5.txt and index-jis0208.txt of the WHATWG Encoding
Standard
+(https://encoding.spec.whatwg.org/)
+
+ Copyright (c) WHATWG (Apple, Google, Mozilla, Microsoft).
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are met:
+ 1. Redistributions of source code must retain the above copyright notice,
+ this list of conditions and the following disclaimer.
+ 2. Redistributions in binary form must reproduce the above copyright
notice,
+ this list of conditions and the following disclaimer in the
documentation
+ and/or other materials provided with the distribution.
+ 3. Neither the name of the copyright holder nor the names of its
+ contributors may be used to endorse or promote products derived from
+ this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ POSSIBILITY OF SUCH DAMAGE.
+
MIME type information from file-4.26.tar.gz (http://www.darwinsys.com/file/)
Copyright (c) Ian F. Darwin 1986, 1987, 1989, 1990, 1991, 1992, 1994, 1995.
