This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new e7dce6fb0c TIKA-5046: look for non-ascii farther into the file (#3594)
e7dce6fb0c is described below
commit e7dce6fb0c25b313696a36f756775e6ac7da6d12
Author: Tim Allison <[email protected]>
AuthorDate: Mon Nov 9 15:52:23 2026 -0500
TIKA-5046: look for non-ascii farther into the file (#3594)
---
CHANGES.txt | 9 +
.../pages/advanced/charset-detection-design.adoc | 27 ++
.../org/apache/tika/detect/EncodingProbeCache.java | 30 +-
.../apache/tika/detect/EncodingProbeCacheTest.java | 38 ++
.../apache/tika/ml/chardetect/EvidenceProbe.java | 530 +++++++++++++++++++++
.../ml/chardetect/MojibusterEncodingDetector.java | 9 +-
.../tika/ml/chardetect/EvidenceProbeTest.java | 373 +++++++++++++++
.../ml/junkdetect/JunkFilterEncodingDetector.java | 8 +-
.../ml/junkdetect/JunkFilterCandidatesTest.java | 21 +
.../tika/config/TikaEncodingDetectorTest.java | 18 +-
10 files changed, 1039 insertions(+), 24 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index b8f5f3a343..bf993b76f6 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -431,6 +431,15 @@ Release 4.3.0 - ???
tika-eval:languagenessCoverage. Languageness is -99 (unknown) for text
with no letters
or with under 90% of its letters in scripts the model was trained on
(TIKA-5027).
+ * Charset detection finds non-ASCII text past the first 16 KB, as in an
email with a
+ long ASCII header block, a log or a CSV: those were reported as
windows-1252. When the
+ leading sample is not conclusive, detection reads on (up to 16 MB of a
file or byte
+ array, 1 MB of a stream) and adds the bytes around each non-ASCII byte
and each
+ ISO-2022, HZ or UTF-7 sequence to the sample, stopping at 50 non-ASCII
bytes or 16
+ UTF-8 characters. Fewer than 8 scattered non-ASCII bytes leave the sample
as it was.
+ The junk filter scores the same sample. EncodingProbeCache keys on a
probe kind
+ (TIKA-5046).
+
OTHER CHANGES
* The file-to-markdown-docker agent skill turns OCR off with an empty
text-recognizers
diff --git a/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
b/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
index 5a82d538a6..f9ef4f079f 100644
--- a/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
+++ b/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
@@ -124,6 +124,33 @@ DECLARATIVE.
Pure-ASCII probes stop early; every other probe collects candidates from the
structural checks and the model, which are then ranked STRUCTURAL first.
+=== The probe
+
+The probe starts as the leading bytes, read until about 16 KB of tag-stripped
+content is present (at most 512 KB raw; `AdaptiveProbe`). When that prefix is
+not conclusive (fewer than 50 bytes ≥ 0x80 and fewer than 16 clean UTF-8
+characters) and holds no BOM, NUL or ISO-2022 escape, `EvidenceProbe` reads on
+and appends windows from the rest of the input, each after a line feed:
+
+* about 32 bytes either side of each byte ≥ 0x80, and around each 7-bit
+ signature (`ESC $`, SO/SI, HZ `~{`, UTF-7 `+...-`); a signature window runs
to
+ the closing marker, at most 8 KB, and signature windows stop after 32 KB of
+ them (base64 and binary are full of false signatures);
+* a window starts after two ASCII bytes and ends at two ASCII bytes, so a
+ double-byte character whose trail byte is ASCII-range is never cut; windows
+ that touch are merged;
+* the scan stops at 50 bytes ≥ 0x80 or 16 clean UTF-8 characters, at EOF, or at
+ 16 MB (file or byte array) or 1 MB (a stream, whose look-ahead is buffered in
+ memory).
+
+The windows are used only when the scan stopped, the evidence is clean UTF-8,
+it holds at least 8 bytes ≥ 0x80, or it is signature-only with no byte ≥ 0x80;
+otherwise the prefix alone is the probe. The prefix is always kept whole:
+7-bit evidence is ASCII-range, and EBCDIC Hebrew and Arabic letters are all
+below 0x80. `JunkFilterEncodingDetector` reads the same probe, shared through
+`EncodingProbeCache`. `detect(byte[])` callers (e.g. zip entry names) score
+the bytes they pass.
+
=== Pure ASCII and ISO-2022
A probe with no byte ≥ 0x80 and no NUL is first checked for ISO-2022-JP/KR/CN.
diff --git
a/tika-core/src/main/java/org/apache/tika/detect/EncodingProbeCache.java
b/tika-core/src/main/java/org/apache/tika/detect/EncodingProbeCache.java
index a4d783fb02..be698bc46d 100644
--- a/tika-core/src/main/java/org/apache/tika/detect/EncodingProbeCache.java
+++ b/tika-core/src/main/java/org/apache/tika/detect/EncodingProbeCache.java
@@ -31,24 +31,29 @@ package org.apache.tika.detect;
* <p>
* Not thread-safe: a single detection runs its detectors sequentially on one
* thread. The cache is keyed by the probe parameters — {@link #get} returns
the
- * cached probe only when both {@code contentTarget} and {@code rawCap} match
what
- * it was stored with, so a detector that wants a differently-sized probe
+ * cached probe only when its {@code kind}, {@code contentTarget} and {@code
rawCap}
+ * all match what it was stored with, so a detector that wants a different
probe
* transparently reads (and caches) its own.
* <p>
* The cached array is shared read-only state; callers must not mutate it in
place.
*/
public class EncodingProbeCache {
+ /** Kind of the plain leading-bytes probe, used by the kind-less methods.
*/
+ public static final String PREFIX = "prefix";
+
private byte[] probe;
+ private String kind;
private int contentTarget = -1;
private int rawCap = -1;
/**
- * @return the cached probe if one was stored with the same {@code
contentTarget} and
- * {@code rawCap}; otherwise {@code null}
+ * @return the cached probe if one was stored with the same {@code kind},
+ * {@code contentTarget} and {@code rawCap}; otherwise {@code null}
*/
- public byte[] get(int contentTarget, int rawCap) {
- if (probe != null && this.contentTarget == contentTarget &&
this.rawCap == rawCap) {
+ public byte[] get(String kind, int contentTarget, int rawCap) {
+ if (probe != null && this.kind.equals(kind) && this.contentTarget ==
contentTarget
+ && this.rawCap == rawCap) {
return probe;
}
return null;
@@ -57,9 +62,20 @@ public class EncodingProbeCache {
/**
* Stores the probe bytes read with the given parameters.
*/
- public void put(byte[] probe, int contentTarget, int rawCap) {
+ public void put(byte[] probe, String kind, int contentTarget, int rawCap) {
this.probe = probe;
+ this.kind = kind;
this.contentTarget = contentTarget;
this.rawCap = rawCap;
}
+
+ /** {@link #get(String, int, int)} for a {@link #PREFIX} probe. */
+ public byte[] get(int contentTarget, int rawCap) {
+ return get(PREFIX, contentTarget, rawCap);
+ }
+
+ /** {@link #put(byte[], String, int, int)} for a {@link #PREFIX} probe. */
+ public void put(byte[] probe, int contentTarget, int rawCap) {
+ put(probe, PREFIX, contentTarget, rawCap);
+ }
}
diff --git
a/tika-core/src/test/java/org/apache/tika/detect/EncodingProbeCacheTest.java
b/tika-core/src/test/java/org/apache/tika/detect/EncodingProbeCacheTest.java
new file mode 100644
index 0000000000..881af83ef2
--- /dev/null
+++ b/tika-core/src/test/java/org/apache/tika/detect/EncodingProbeCacheTest.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.detect;
+
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
+
+import org.junit.jupiter.api.Test;
+
+public class EncodingProbeCacheTest {
+
+ @Test
+ public void kindIsPartOfTheKey() {
+ EncodingProbeCache cache = new EncodingProbeCache();
+ byte[] probe = {'a'};
+ cache.put(probe, "evidence", 16384, 524288);
+ assertSame(probe, cache.get("evidence", 16384, 524288));
+ assertNull(cache.get(16384, 524288));
+ assertNull(cache.get("evidence", 16384, 1024));
+ cache.put(probe, 16384, 524288);
+ assertSame(probe, cache.get(EncodingProbeCache.PREFIX, 16384, 524288));
+ assertNull(cache.get("evidence", 16384, 524288));
+ }
+}
diff --git
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/EvidenceProbe.java
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/EvidenceProbe.java
new file mode 100644
index 0000000000..18e0c9f4c8
--- /dev/null
+++
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/EvidenceProbe.java
@@ -0,0 +1,530 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.ml.chardetect;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.VarHandle;
+import java.nio.ByteOrder;
+import java.util.Arrays;
+
+import org.apache.commons.io.IOUtils;
+
+import org.apache.tika.detect.BOMDetector;
+import org.apache.tika.io.TikaInputStream;
+
+/**
+ * Encoding-detection probe that adds evidence from past the {@link
AdaptiveProbe} prefix.
+ *
+ * <p>A document whose first non-ASCII byte sits past the prefix (an email
with a long ASCII
+ * header block, a log, a CSV) gives the detectors nothing to work with. When
the prefix is
+ * not already decisive, this scans on, takes windows around each byte ≥
0x80 and around
+ * the 7-bit signatures of ISO-2022 ({@code ESC $}, SO/SI), HZ
(<code>~{</code>) and UTF-7
+ * ({@code +...-}), and appends them to the prefix, each after a {@code '\n'}.
A prefix
+ * with a BOM, a NUL or an ISO-2022 escape is evidence enough and is returned
as is.
+ *
+ * <ul>
+ * <li>The scan stops at {@value #STOP_HIGH} high bytes or {@value
#STOP_UTF8} clean UTF-8
+ * sequences, at EOF, or at the cap: {@value #FILE_SCAN_CAP} bytes when the
stream's length
+ * is measured (file, byte array), {@value #STREAM_SCAN_CAP} otherwise,
since a passthrough
+ * stream buffers what the mark covers.</li>
+ * <li>Evidence is used only if it reached a stop, is clean UTF-8, holds at
least
+ * {@value #ACCEPT_HIGH} high bytes, or is signature-only with no high byte
at all;
+ * otherwise the prefix is returned unchanged.</li>
+ * <li>Signature windows stop being taken after {@value #SIG_BUDGET} bytes
of them.</li>
+ * <li>The prefix is always kept whole: 7-bit evidence is ASCII-range, and
EBCDIC
+ * Hebrew/Arabic letters are all below 0x80.</li>
+ * </ul>
+ *
+ * <p>Windows start after two ASCII bytes and end at two ASCII bytes, so no
multibyte
+ * character is cut (DBCS trail bytes can be ASCII-range); touching windows
merge; a
+ * signature window runs to its closing marker, at most {@value #SIG_MAX}
bytes.
+ */
+public final class EvidenceProbe {
+
+ /** {@link org.apache.tika.detect.EncodingProbeCache} kind for this probe.
*/
+ public static final String KIND = "evidence";
+
+ static final long FILE_SCAN_CAP = 16L << 20;
+ static final long STREAM_SCAN_CAP = 1L << 20;
+ static final int STOP_HIGH = 50;
+ static final int STOP_UTF8 = 16;
+ static final int ACCEPT_HIGH = 8;
+ static final int WINDOW = 32;
+ static final int MAX_WINDOW = 256;
+ static final int ALIGN_BACK = 256;
+ static final int SIG_MAX = 8192;
+ // base64 and binary are full of false 7-bit signatures; high-byte windows
are bounded by STOP_HIGH
+ static final int SIG_BUDGET = 32 * 1024;
+
+ private static final int CHUNK = 64 * 1024;
+ // covers the longest reach of one window past its trigger byte (a
signature run + WINDOW)
+ private static final int LOOKAHEAD = 12 * 1024;
+ private static final int LOOKBACK = ALIGN_BACK + WINDOW + 16;
+
+ private static final VarHandle LONGS =
+ MethodHandles.byteArrayViewVarHandle(long[].class,
ByteOrder.LITTLE_ENDIAN);
+ private static final long ONES = 0x0101010101010101L;
+ private static final long HIGHS = 0x8080808080808080L;
+ /** First bytes {@link #sigEnd} can match. */
+ private static final boolean[] SIG_START = new boolean[256];
+
+ private static final boolean[] BASE64 = new boolean[256];
+
+ static {
+ for (int c = 0; c < 256; c++) {
+ BASE64[c] = isBase64((byte) c);
+ }
+ SIG_START[0x1B] = true;
+ SIG_START[0x0E] = true;
+ SIG_START['~'] = true;
+ SIG_START['+'] = true;
+ }
+
+ private EvidenceProbe() {
+ }
+
+ /**
+ * Reads the probe from {@code tis}; the stream position is unchanged on
return.
+ * {@code contentTarget} and {@code rawCap} size the prefix as in {@link
AdaptiveProbe}.
+ */
+ public static byte[] read(TikaInputStream tis, int contentTarget, int
rawCap)
+ throws IOException {
+ return scan(tis, contentTarget, rawCap,
+ tis.hasReliableLength() ? FILE_SCAN_CAP :
STREAM_SCAN_CAP).probe;
+ }
+
+ static Result scan(TikaInputStream tis, int contentTarget, int rawCap,
long scanCap)
+ throws IOException {
+ byte[] prefix = AdaptiveProbe.read(tis, contentTarget, rawCap);
+ int pl = prefix.length;
+ Result r = new Result(prefix, pl);
+ if (pl == 0 || hasBomOrNul(prefix)) {
+ return r;
+ }
+ Tally st = new Tally();
+ st.add(prefix, 0, pl);
+ if (st.decisive() || hasIso2022Escape(prefix)) {
+ return r;
+ }
+ if (pl >= scanCap || (tis.hasReliableLength() && tis.getLength() <=
pl)) {
+ return r;
+ }
+ st.sigs = countSigs(prefix, pl);
+
+ Scanner sc = new Scanner(tis, prefix, (int) scanCap);
+ tis.mark((int) scanCap);
+ try {
+ IOUtils.skip(tis, pl);
+ boolean stopped = sc.extract(st);
+ r.read = sc.pos;
+ if (sc.windows == 0) {
+ return r;
+ }
+ boolean utf8Clean = st.utf8 > 0 && st.utf8Bad == 0;
+ if (!stopped && st.high < ACCEPT_HIGH && !utf8Clean
+ && !(st.high == 0 && st.sigs > 0)) {
+ return r;
+ }
+ byte[] ev = sc.out.toByteArray();
+ byte[] probe = Arrays.copyOf(prefix, pl + ev.length);
+ System.arraycopy(ev, 0, probe, pl, ev.length);
+ r.probe = probe;
+ return r;
+ } finally {
+ tis.reset();
+ }
+ }
+
+ static final class Result {
+ byte[] probe;
+ /** End offset of the scan (bytes from the mark). */
+ long read;
+
+ Result(byte[] probe, long read) {
+ this.probe = probe;
+ this.read = read;
+ }
+ }
+
+ private static boolean hasBomOrNul(byte[] p) {
+ if (BOMDetector.bomLength(p) > 0) {
+ return true;
+ }
+ for (byte b : p) {
+ if (b == 0) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static boolean hasIso2022Escape(byte[] p) {
+ for (int i = 0; i + 1 < p.length; i++) {
+ if (p[i] == 0x1B && (p[i + 1] == '$' || p[i + 1] == '(')) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static int countSigs(byte[] p, int hi) {
+ int n = 0;
+ for (int i = 0; i < hi; i++) {
+ if (!SIG_START[p[i] & 0xFF]) {
+ continue;
+ }
+ int e = sigEnd(p, 0, i, hi);
+ if (e > 0) {
+ n++;
+ i = e - 1;
+ } else if (e < -1) {
+ i = -e - 3;
+ }
+ }
+ return n;
+ }
+
+ /**
+ * End (exclusive, absolute) of a 7-bit signature starting at absolute
offset {@code i}
+ * of {@code b} (whose index 0 is offset {@code base}), bounded by {@code
hi}; -1 if none,
+ * or {@code -(j + 2)} for a failed UTF-7 candidate whose base64 run ends
at {@code j}, so
+ * the caller can skip the run instead of rescanning it from each inner
'+'.
+ */
+ static int sigEnd(byte[] b, int base, int i, int hi) {
+ int c = b[i - base] & 0xFF;
+ if (c == 0x1B && i + 1 < hi && "$()*+.NO".indexOf(b[i + 1 - base]) >=
0) {
+ for (int j = i + 2; j + 1 < hi && j < i + SIG_MAX; j++) {
+ if ((b[j - base] == 0x1B && b[j + 1 - base] == '(') || b[j -
base] == 0x0F) {
+ return Math.min(hi, j + 3);
+ }
+ }
+ return Math.min(hi, i + SIG_MAX);
+ }
+ if (c == 0x0E) {
+ for (int j = i + 1; j < hi && j < i + SIG_MAX; j++) {
+ if (b[j - base] == 0x0F) {
+ return j + 1;
+ }
+ }
+ return Math.min(hi, i + SIG_MAX);
+ }
+ if (c == '~' && i + 1 < hi && b[i + 1 - base] == '{') {
+ for (int j = i + 2; j + 1 < hi && j < i + SIG_MAX; j++) {
+ if (b[j - base] == '~' && b[j + 1 - base] == '}') {
+ return j + 2;
+ }
+ }
+ return Math.min(hi, i + SIG_MAX);
+ }
+ if (c == '+') {
+ int j = i + 1;
+ while (j < hi && j < i + MAX_WINDOW && isBase64(b[j - base])) {
+ j++;
+ }
+ if (j - i - 1 >= 2 && j < hi && b[j - base] == '-') {
+ return j + 1;
+ }
+ return j > i + 1 ? -(j + 2) : -1;
+ }
+ return -1;
+ }
+
+ private static boolean isBase64(byte x) {
+ return (x >= 'A' && x <= 'Z') || (x >= 'a' && x <= 'z') || (x >= '0'
&& x <= '9')
+ || x == '+' || x == '/';
+ }
+
+ private static boolean isHigh(byte x) {
+ return (x & 0xFF) >= 0x80;
+ }
+
+ /** High-byte and UTF-8 sequence counts. */
+ static final class Tally {
+ int high;
+ int utf8;
+ int utf8Bad;
+ int sigs;
+
+ boolean decisive() {
+ return (utf8 >= STOP_UTF8 && utf8Bad == 0) || high >= STOP_HIGH;
+ }
+
+ void add(byte[] b, int s, int e) {
+ int i = s;
+ while (i < e) {
+ int x = b[i] & 0xFF;
+ if (x < 0x80) {
+ i++;
+ continue;
+ }
+ int len = x >= 0xC2 && x <= 0xDF ? 2 : x >= 0xE0 && x <= 0xEF
? 3
+ : x >= 0xF0 && x <= 0xF4 ? 4 : 0;
+ boolean ok = len > 0 && i + len <= e;
+ for (int j = 1; ok && j < len; j++) {
+ ok = (b[i + j] & 0xC0) == 0x80;
+ }
+ if (ok && len == 3) {
+ int y = b[i + 1] & 0xFF;
+ ok = !((x == 0xE0 && y < 0xA0) || (x == 0xED && y > 0x9F));
+ }
+ if (ok && len == 4) {
+ int y = b[i + 1] & 0xFF;
+ ok = !((x == 0xF0 && y < 0x90) || (x == 0xF4 && y > 0x8F));
+ }
+ if (ok) {
+ utf8++;
+ high += len;
+ i += len;
+ } else {
+ utf8Bad++;
+ high++;
+ i++;
+ }
+ }
+ }
+ }
+
+ /** Sliding buffer over the stream past the prefix; offsets are absolute
from the mark. */
+ private static final class Scanner {
+ private final TikaInputStream tis;
+ private final int cap;
+ private final int lo;
+ private byte[] buf;
+ /** Absolute offset of buf[0]. */
+ private int base;
+ /** Absolute end of the valid bytes in buf. */
+ private int end;
+ private boolean eof;
+ final ByteArrayOutputStream out = new ByteArrayOutputStream();
+ int windows;
+ int pos;
+ private int sigBytes;
+
+ Scanner(TikaInputStream tis, byte[] prefix, int cap) {
+ this.tis = tis;
+ this.cap = cap;
+ this.lo = prefix.length;
+ // the first window may reach back across the prefix end, which
can split a character
+ int keep = Math.min(prefix.length, LOOKBACK);
+ this.buf = new byte[Math.max(CHUNK + LOOKAHEAD + LOOKBACK, keep)];
+ System.arraycopy(prefix, prefix.length - keep, buf, 0, keep);
+ this.base = prefix.length - keep;
+ this.end = prefix.length;
+ }
+
+ private byte at(int i) {
+ return buf[i - base];
+ }
+
+ /** Ensures bytes up to {@code upTo} are buffered unless EOF or the
cap comes first. */
+ private void fill(int upTo, int keepFrom) throws IOException {
+ if (eof || end >= upTo) {
+ return;
+ }
+ int shift = keepFrom - base;
+ if (shift > 0) {
+ System.arraycopy(buf, shift, buf, 0, end - keepFrom);
+ base = keepFrom;
+ }
+ while (!eof && end - base < buf.length) {
+ int want = Math.min(buf.length - (end - base), cap - end);
+ if (want <= 0) {
+ eof = true;
+ break;
+ }
+ int n = IOUtils.read(tis, buf, end - base, want);
+ end += n;
+ if (n < want) {
+ eof = true;
+ }
+ }
+ }
+
+ /** Windows over [lo, cap); true when the tally became decisive. */
+ boolean extract(Tally st) throws IOException {
+ int prevEnd = lo;
+ int i = lo;
+ int lastEnd = -1;
+ while (true) {
+ int sigEnd = -1;
+ while (true) {
+ if (!eof && i + LOOKAHEAD >= end) {
+ fill(i + LOOKAHEAD + 1, Math.max(base, i - LOOKBACK));
+ }
+ if (i >= end) {
+ pos = end;
+ return false;
+ }
+ int lim = eof ? end : end - LOOKAHEAD;
+ i = skipPlain(i, lim, sigBytes < SIG_BUDGET);
+ if (i >= lim) {
+ continue;
+ }
+ if (isHigh(at(i))) {
+ break;
+ }
+ if (sigBytes >= SIG_BUDGET) {
+ i++;
+ continue;
+ }
+ sigEnd = sigEnd(buf, base, i, end);
+ if (sigEnd > 0) {
+ break;
+ }
+ i = sigEnd < -1 ? -sigEnd - 2 : i + 1;
+ }
+ int start = Math.max(prevEnd, i - WINDOW);
+ while (start > prevEnd && isHigh(at(start - 1))) {
+ start--;
+ }
+ int floorBase = windows == 0 ? Math.max(Math.max(0, lo -
ALIGN_BACK), base)
+ : prevEnd;
+ int s = start;
+ int floor = Math.max(floorBase, start - ALIGN_BACK);
+ while (s > floor && !afterTwoAscii(s)) {
+ s--;
+ }
+ if ((s == prevEnd && windows > 0) || afterTwoAscii(s)) {
+ start = s;
+ }
+ int wEnd;
+ if (sigEnd > 0) {
+ wEnd = Math.min(end, sigEnd + WINDOW);
+ st.sigs++;
+ } else {
+ int last = i;
+ int k = i + 1;
+ wEnd = last + WINDOW + 1;
+ while (k < end && k < wEnd && k - start < MAX_WINDOW) {
+ if (isHigh(at(k))) {
+ last = k;
+ wEnd = last + WINDOW + 1;
+ }
+ k++;
+ }
+ wEnd = Math.min(wEnd, end);
+ }
+ while (wEnd < end && wEnd - start < MAX_WINDOW + 1024
+ && (isHigh(at(wEnd - 1)) || isHigh(at(wEnd - 2)))) {
+ wEnd++;
+ }
+ // a separator inside a 7-bit run, or between a lead byte and
its trail, corrupts decoding
+ if (windows > 0 && start <= lastEnd) {
+ out.write(buf, lastEnd - base, wEnd - lastEnd);
+ } else {
+ out.write('\n');
+ out.write(buf, start - base, wEnd - start);
+ windows++;
+ }
+ st.add(buf, start - base, wEnd - base);
+ if (sigEnd > 0) {
+ sigBytes += wEnd - start;
+ }
+ lastEnd = wEnd;
+ prevEnd = wEnd;
+ i = wEnd;
+ if (st.decisive()) {
+ pos = wEnd;
+ return true;
+ }
+ }
+ }
+
+ /**
+ * First offset in [i, lim) holding a high byte or, if {@code sigs}, a
signature start;
+ * {@code lim} if none. Eight bytes at a time: the zero-byte test's
lowest flag is
+ * always a true match (borrows only create flags above one).
+ */
+ private int skipPlain(int i, int lim, boolean sigs) {
+ byte[] b = buf;
+ int k = i - base;
+ int kl = lim - base;
+ int ke = end - base;
+ while (true) {
+ int hit = -1;
+ while (k + 8 <= kl) {
+ long v = (long) LONGS.get(b, k);
+ long m = v & HIGHS;
+ if (sigs) {
+ m |= zeroByte(v ^ (0x1BL * ONES)) | zeroByte(v ^
(0x0EL * ONES))
+ | zeroByte(v ^ ('~' * ONES)) | zeroByte(v ^
('+' * ONES));
+ }
+ if (m != 0) {
+ // little-endian: the lowest flagged byte is the first
match
+ hit = k + (Long.numberOfTrailingZeros(m) >>> 3);
+ break;
+ }
+ k += 8;
+ }
+ if (hit < 0) {
+ while (k < kl && b[k] >= 0 && !(sigs && SIG_START[b[k]])) {
+ k++;
+ }
+ hit = k;
+ }
+ k = hit;
+ if (k >= kl || b[k] < 0) {
+ return k + base;
+ }
+ int next = nextAfterNonSignature(b, k, ke);
+ if (next < 0) {
+ return k + base;
+ }
+ k = next;
+ }
+ }
+
+ /**
+ * Where {@link #sigEnd} would send the scan from {@code k} when no
signature starts
+ * there (k + 1, or past a failed UTF-7 run); -1 if one may start
there.
+ */
+ private static int nextAfterNonSignature(byte[] b, int k, int ke) {
+ int c = b[k];
+ if (c == 0x0E) {
+ return -1;
+ }
+ if (k + 1 >= ke) {
+ return k + 1;
+ }
+ byte n = b[k + 1];
+ if (c == '~') {
+ return n == '{' ? -1 : k + 1;
+ }
+ if (c == '+') {
+ int j = k + 1;
+ while (j < ke && j < k + MAX_WINDOW && BASE64[b[j] & 0xFF]) {
+ j++;
+ }
+ if (j - k - 1 >= 2 && j < ke && b[j] == '-') {
+ return -1;
+ }
+ return j > k + 1 ? j : k + 1;
+ }
+ return "$()*+.NO".indexOf(n) < 0 ? k + 1 : -1;
+ }
+
+ private static long zeroByte(long x) {
+ return (x - ONES) & ~x & HIGHS;
+ }
+
+ private boolean afterTwoAscii(int s) {
+ return s == 0 || (!isHigh(at(s - 1)) && (s - 2 < 0 || !isHigh(at(s
- 2))));
+ }
+ }
+}
diff --git
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/MojibusterEncodingDetector.java
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/MojibusterEncodingDetector.java
index 71249b4c57..c5fe40f2fb 100644
---
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/MojibusterEncodingDetector.java
+++
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/MojibusterEncodingDetector.java
@@ -72,7 +72,8 @@ public class MojibusterEncodingDetector implements
EncodingDetector {
// Probe sized by tag-stripped content (16 KB target), capped at 512 KB
raw.
// Markup-heavy pages whose distinguishing bytes (esp. UTF-8 multi-byte
// sequences) sit past a fixed 16 KB raw window would otherwise starve the
- // structural UTF-8 check and the model. See AdaptiveProbe.
+ // structural UTF-8 check and the model. See AdaptiveProbe; EvidenceProbe
adds
+ // windows from past this prefix when it is not decisive.
private static final int PROBE_CONTENT_TARGET =
AdaptiveProbe.DEFAULT_CONTENT_TARGET;
private static final int PROBE_RAW_CAP = AdaptiveProbe.DEFAULT_RAW_CAP;
@@ -675,14 +676,14 @@ public class MojibusterEncodingDetector implements
EncodingDetector {
parseContext == null ? null :
parseContext.get(EncodingDetectorContext.class);
EncodingProbeCache cache = context == null ? null :
context.getProbeCache();
if (cache != null) {
- byte[] cached = cache.get(PROBE_CONTENT_TARGET, PROBE_RAW_CAP);
+ byte[] cached = cache.get(EvidenceProbe.KIND,
PROBE_CONTENT_TARGET, PROBE_RAW_CAP);
if (cached != null) {
return cached;
}
}
- byte[] probe = AdaptiveProbe.read(tis, PROBE_CONTENT_TARGET,
PROBE_RAW_CAP);
+ byte[] probe = EvidenceProbe.read(tis, PROBE_CONTENT_TARGET,
PROBE_RAW_CAP);
if (cache != null) {
- cache.put(probe, PROBE_CONTENT_TARGET, PROBE_RAW_CAP);
+ cache.put(probe, EvidenceProbe.KIND, PROBE_CONTENT_TARGET,
PROBE_RAW_CAP);
}
return probe;
}
diff --git
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/EvidenceProbeTest.java
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/EvidenceProbeTest.java
new file mode 100644
index 0000000000..cbd2c6baae
--- /dev/null
+++
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/EvidenceProbeTest.java
@@ -0,0 +1,373 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.ml.chardetect;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Arrays;
+import java.util.Random;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import org.apache.tika.detect.EncodingDetectorContext;
+import org.apache.tika.detect.EncodingResult;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.ParseContext;
+
+public class EvidenceProbeTest {
+
+ private static final int CT = AdaptiveProbe.DEFAULT_CONTENT_TARGET;
+ private static final int RC = AdaptiveProbe.DEFAULT_RAW_CAP;
+ private static final Charset W1252 = Charset.forName("windows-1252");
+ private static final String RUSSIAN = "Съешь же ещё этих мягких
французских булок, да выпей чаю. ";
+
+ private static final MojibusterEncodingDetector DET = newDetector();
+
+ private static MojibusterEncodingDetector newDetector() {
+ try {
+ return new MojibusterEncodingDetector();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /** Header-like ASCII lines, {@code n} bytes. */
+ private static byte[] ascii(int n, long seed) {
+ String[] words = {"Received:", "from", "mail", "server", "by",
"relay", "with", "ESMTP",
+ "id", "for", "Tue,", "Nov", "2026", "Subject:", "report",
"status", "the", "and"};
+ Random r = new Random(seed);
+ StringBuilder sb = new StringBuilder();
+ while (sb.length() < n) {
+ sb.append(words[r.nextInt(words.length)]).append(r.nextInt(8) == 0
? "\r\n" : " ");
+ }
+ return sb.substring(0, n).getBytes(StandardCharsets.US_ASCII);
+ }
+
+ private static byte[] cat(byte[]... parts) {
+ ByteArrayOutputStream o = new ByteArrayOutputStream();
+ for (byte[] p : parts) {
+ o.writeBytes(p);
+ }
+ return o.toByteArray();
+ }
+
+ private static byte[] prefix(byte[] b) throws IOException {
+ try (TikaInputStream tis = TikaInputStream.get(b)) {
+ return AdaptiveProbe.read(tis, CT, RC);
+ }
+ }
+
+ private static byte[] evidence(byte[] b) throws IOException {
+ try (TikaInputStream tis = TikaInputStream.get(b)) {
+ return EvidenceProbe.read(tis, CT, RC);
+ }
+ }
+
+ private static String detect(byte[] b) throws IOException {
+ ParseContext pc = new ParseContext();
+ pc.set(EncodingDetectorContext.class, new EncodingDetectorContext());
+ try (TikaInputStream tis = TikaInputStream.get(b)) {
+ return DET.detect(tis, new Metadata(),
pc).get(0).getCharset().name();
+ }
+ }
+
+ private static int highBytes(byte[] b) {
+ int n = 0;
+ for (byte x : b) {
+ n += (x & 0xFF) >= 0x80 ? 1 : 0;
+ }
+ return n;
+ }
+
+ private static boolean contains(byte[] hay, byte[] needle) {
+ outer:
+ for (int i = 0; i + needle.length <= hay.length; i++) {
+ for (int j = 0; j < needle.length; j++) {
+ if (hay[i + j] != needle[j]) {
+ continue outer;
+ }
+ }
+ return true;
+ }
+ return false;
+ }
+
+ @Test
+ public void lateUtf8IsFoundPastAnAsciiPrefix() throws Exception {
+ byte[] b = cat(ascii(40_000, 1),
RUSSIAN.repeat(20).getBytes(StandardCharsets.UTF_8),
+ ascii(5_000, 2));
+ byte[] pre = prefix(b);
+ assertEquals(0, highBytes(pre));
+ assertEquals("windows-1252",
DET.detect(pre).get(0).getCharset().name());
+ byte[] p = evidence(b);
+ assertArrayEquals(pre, Arrays.copyOf(p, pre.length));
+ assertTrue(p.length < pre.length + 2_000, "probe length " + p.length);
+ assertEquals("UTF-8", detect(b));
+ }
+
+ @Test
+ public void evidenceFollowsThePrefixAfterANewline() throws Exception {
+ byte[] b = cat("café ".getBytes(W1252), ascii(40_000, 3),
+ RUSSIAN.repeat(20).getBytes(StandardCharsets.UTF_8));
+ byte[] pre = prefix(b);
+ byte[] p = evidence(b);
+ assertTrue(p.length > pre.length);
+ assertArrayEquals(pre, Arrays.copyOf(p, pre.length));
+ assertEquals('\n', p[pre.length]);
+ }
+
+ @Test
+ public void decisivePrefixIsUnchanged() throws Exception {
+ byte[] b = cat(RUSSIAN.repeat(5).getBytes(StandardCharsets.UTF_8),
ascii(40_000, 4),
+ "naïve".getBytes(W1252));
+ assertArrayEquals(prefix(b), evidence(b));
+ }
+
+ /** Below {@value EvidenceProbe#ACCEPT_HIGH} scattered high bytes the
evidence is too thin to use. */
+ @ParameterizedTest
+ @ValueSource(ints = {1, 7, 8})
+ public void sparseLateEvidenceFallsBackToThePrefix(int n) throws Exception
{
+ ByteArrayOutputStream o = new ByteArrayOutputStream();
+ o.writeBytes(ascii(40_000, 5));
+ for (int i = 0; i < n; i++) {
+ o.writeBytes(("Norrgård " + new String(ascii(400, i),
StandardCharsets.US_ASCII))
+ .getBytes(W1252));
+ }
+ byte[] b = o.toByteArray();
+ byte[] p = evidence(b);
+ if (n < EvidenceProbe.ACCEPT_HIGH) {
+ assertArrayEquals(prefix(b), p);
+ } else {
+ assertEquals(n, highBytes(p));
+ }
+ }
+
+ /** DBCS trail bytes can be ASCII-range (0x5C here): every window must
start on a character. */
+ @ParameterizedTest
+ @ValueSource(ints = {0, 1, 2, 3})
+ public void windowsStartOnDbcsCharacterBoundaries(int split) throws
Exception {
+ Charset sjis = Charset.forName("windows-31j");
+ String chars = "表ソ能予あい日本";
+ Random r = new Random(split);
+ StringBuilder sb = new StringBuilder();
+ while (sb.length() < 600) {
+ for (int i = 0, k = 1 + r.nextInt(6); i < k; i++) {
+ sb.append(chars.charAt(r.nextInt(chars.length())));
+ if (r.nextInt(3) == 0) {
+ sb.append('a');
+ }
+ }
+ sb.append(r.nextInt(4) == 0 ? " ".repeat(40) : " ");
+ }
+ String text = sb.toString();
+ // the prefix ends `split` bytes into the Japanese text
+ byte[] b = cat(ascii(CT - split, 6), text.getBytes(sjis));
+ byte[] pre = prefix(b);
+ byte[] p = evidence(b);
+ assertTrue(p.length > pre.length);
+ String tail = new String(p, pre.length, p.length - pre.length, sjis);
+ for (String w : tail.split("\n")) {
+ if (!w.isEmpty()) {
+ assertTrue(text.contains(w) || new String(b,
sjis).contains(w), "misaligned: " + w);
+ }
+ }
+ }
+
+ private static String japanese() {
+ return "日本語のテスト文章です。"
+ + "これは七ビットの文書です。";
+ }
+
+ @Test
+ public void iso2022JpPastThePrefixIsWindowedToItsClosingEscape() throws
Exception {
+ byte[] jp = ("Dear user, " + japanese().repeat(10) + "
thanks.\n").getBytes("ISO-2022-JP");
+ byte[] b = cat(ascii(20_000, 7), jp, ascii(5_000, 8));
+ assertEquals("windows-1252", detect(cat(ascii(20_000, 7), ascii(5_000,
8))));
+ byte[] p = evidence(b);
+ int esc = indexOf(jp, new byte[] {0x1B, '$', 'B'});
+ int close = lastIndexOf(jp, new byte[] {0x1B, '(', 'B'});
+ assertTrue(contains(p, Arrays.copyOfRange(jp, esc, close + 3)));
+ assertEquals("ISO-2022-JP", detect(b));
+ }
+
+ /** Signature-only evidence is used only when there is no high byte at
all. */
+ @Test
+ public void iso2022JpWithAStrayLateHighByteKeepsThePrefix() throws
Exception {
+ byte[] jp = japanese().repeat(10).getBytes("ISO-2022-JP");
+ byte[] late = cat(ascii(20_000, 9), jp, ascii(5_000, 10),
"café".getBytes(W1252));
+ assertArrayEquals(prefix(late), evidence(late));
+ // escapes inside the prefix: the prefix is evidence enough, whatever
follows
+ byte[] early = cat(jp, ascii(20_000, 11), "café".getBytes(W1252));
+ assertArrayEquals(prefix(early), evidence(early));
+ assertEquals("ISO-2022-JP", detect(early));
+ }
+
+ @Test
+ public void hzRunIsWindowedWhole() throws Exception {
+ // HZ: ~{ GB2312 bytes with the high bit cleared ~}
+ byte[] gb = "中文测试文本".repeat(40).getBytes("GB2312");
+ ByteArrayOutputStream hz = new ByteArrayOutputStream();
+ hz.writeBytes("~{".getBytes(StandardCharsets.US_ASCII));
+ for (byte x : gb) {
+ hz.write(x & 0x7F);
+ }
+ hz.writeBytes("~}".getBytes(StandardCharsets.US_ASCII));
+ byte[] run = hz.toByteArray();
+ byte[] b = cat(ascii(20_000, 12), run, ascii(5_000, 13));
+ assertTrue(contains(evidence(b), run));
+ byte[] stray = cat(b, "café".getBytes(W1252));
+ assertArrayEquals(prefix(stray), evidence(stray));
+ }
+
+ @Test
+ public void utf7RunIsWindowed() throws Exception {
+ byte[] run = "+BB8EQAQ4BDIENQRC-".getBytes(StandardCharsets.US_ASCII);
+ byte[] b = cat(ascii(20_000, 14), run, ascii(5_000, 15));
+ assertTrue(contains(evidence(b), run));
+ byte[] stray = cat(b, "café".getBytes(W1252));
+ assertArrayEquals(prefix(stray), evidence(stray));
+ }
+
+ /** The scan skips ASCII a word at a time: evidence must be found at every
byte alignment. */
+ @ParameterizedTest
+ @ValueSource(ints = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9})
+ public void evidenceIsFoundAtEveryAlignment(int shift) throws Exception {
+ byte[] hz = "~{<:WS~}".getBytes(StandardCharsets.US_ASCII);
+ byte[] fake = "a+b c~d ~~ +0100 ".getBytes(StandardCharsets.US_ASCII);
+ byte[] latin = "éèêëàâçô".getBytes(W1252);
+ byte[] b = cat(ascii(30_000 + shift, 19), fake, ascii(3_000, 20), hz,
ascii(3_000, 21));
+ assertTrue(contains(evidence(b), hz), "hz at shift " + shift);
+ byte[] h = cat(ascii(30_000 + shift, 22), fake, ascii(3_000, 23),
latin, ascii(3_000, 24));
+ assertTrue(contains(evidence(h), latin), "high bytes at shift " +
shift);
+ }
+
+ /** False signatures (base64, binary) cannot grow the probe past the
signature budget. */
+ @Test
+ public void signatureWindowsAreBounded() throws Exception {
+ ByteArrayOutputStream o = new ByteArrayOutputStream();
+ o.writeBytes(ascii(20_000, 18));
+ for (int i = 0; i < 3_000; i++) {
+ o.writeBytes(("~{" + "ab".repeat(15) +
"~}").getBytes(StandardCharsets.US_ASCII));
+ o.writeBytes(ascii(100, i));
+ }
+ byte[] b = o.toByteArray();
+ int extra = evidence(b).length - prefix(b).length;
+ assertTrue(extra > EvidenceProbe.SIG_BUDGET && extra <
EvidenceProbe.SIG_BUDGET + 1_000,
+ "evidence bytes " + extra);
+ }
+
+ private static final String HEBREW = "שלום עולם "
+ + "זהו מסמך בעברית "
+ + "על המחשב והספרים "
+ + "של המדינה היום. ";
+ private static final String ARABIC = "مرحبا بالعالم "
+ + "هذا نص عربي عن الحاسوب "
+ + "والكتب في المدينة اليوم. ";
+ private static final String LATIN = "\nPage 12 of 345 - Report ABC 2024 -
Ref 9876\n";
+
+ /**
+ * EBCDIC Hebrew letters are all below 0x80, so an IBM424 prefix can hold
no high byte;
+ * digits and Latin letters later on are.
+ */
+ @Test
+ public void ibm424WithLateLatinKeepsThePrefix() throws Exception {
+ Charset ibm424 = Charset.forName("IBM424");
+ byte[] heb = HEBREW.repeat(300).getBytes(ibm424);
+ byte[] b = cat(heb, LATIN.repeat(3).getBytes(ibm424));
+ byte[] pre = prefix(b);
+ assertEquals(0, highBytes(pre));
+ byte[] p = evidence(b);
+ assertTrue(p.length > pre.length);
+ assertArrayEquals(pre, Arrays.copyOf(p, pre.length));
+ assertEquals("IBM424", detect(b));
+ }
+
+ @Test
+ public void ibm420PrefixIsKeptWhole() throws Exception {
+ Charset ibm420 = Charset.forName("IBM420");
+ byte[] b = cat(ARABIC.repeat(300).getBytes(ibm420),
LATIN.repeat(3).getBytes(ibm420));
+ byte[] pre = prefix(b);
+ byte[] p = evidence(b);
+ assertArrayEquals(pre, Arrays.copyOf(p, pre.length));
+ assertEquals("IBM420", detect(b));
+ }
+
+ /** A passthrough stream is scanned no further than {@value
EvidenceProbe#STREAM_SCAN_CAP} bytes. */
+ @Test
+ public void streamsUseTheSmallerCap(@TempDir Path tmp) throws Exception {
+ byte[] lateUtf8 = RUSSIAN.repeat(20).getBytes(StandardCharsets.UTF_8);
+ byte[] b = cat(ascii(1_200_000, 16), lateUtf8);
+ try (TikaInputStream tis = TikaInputStream.get(new
ByteArrayInputStream(b))) {
+ assertArrayEquals(prefix(b), EvidenceProbe.read(tis, CT, RC));
+ assertEquals(0, tis.getPosition());
+ }
+ Path f = tmp.resolve("late.txt");
+ Files.write(f, b);
+ try (TikaInputStream tis = TikaInputStream.get(f)) {
+ assertTrue(contains(EvidenceProbe.read(tis, CT, RC),
Arrays.copyOf(lateUtf8, 64)));
+ assertEquals(0, tis.getPosition());
+ byte[] all = tis.readAllBytes();
+ assertArrayEquals(b, all);
+ }
+ }
+
+ /** The detector caches the evidence probe under its own kind. */
+ @Test
+ public void probeIsCachedUnderItsKind() throws Exception {
+ byte[] b = cat(ascii(40_000, 17),
RUSSIAN.repeat(20).getBytes(StandardCharsets.UTF_8));
+ EncodingDetectorContext ctx = new EncodingDetectorContext();
+ ParseContext pc = new ParseContext();
+ pc.set(EncodingDetectorContext.class, ctx);
+ try (TikaInputStream tis = TikaInputStream.get(b)) {
+ EncodingResult top = DET.detect(tis, new Metadata(), pc).get(0);
+ assertEquals("UTF-8", top.getCharset().name());
+ }
+ assertArrayEquals(evidence(b),
ctx.getProbeCache().get(EvidenceProbe.KIND, CT, RC));
+ assertEquals(null, ctx.getProbeCache().get(CT, RC));
+ }
+
+ private static int indexOf(byte[] b, byte[] n) {
+ for (int i = 0; i + n.length <= b.length; i++) {
+ if (Arrays.equals(b, i, i + n.length, n, 0, n.length)) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ private static int lastIndexOf(byte[] b, byte[] n) {
+ for (int i = b.length - n.length; i >= 0; i--) {
+ if (Arrays.equals(b, i, i + n.length, n, 0, n.length)) {
+ return i;
+ }
+ }
+ return -1;
+ }
+}
diff --git
a/tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/JunkFilterEncodingDetector.java
b/tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/JunkFilterEncodingDetector.java
index c6f6599101..88eb6fd90f 100644
---
a/tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/JunkFilterEncodingDetector.java
+++
b/tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/JunkFilterEncodingDetector.java
@@ -43,6 +43,7 @@ import org.apache.tika.detect.MetaEncodingDetector;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.ml.chardetect.AdaptiveProbe;
+import org.apache.tika.ml.chardetect.EvidenceProbe;
import org.apache.tika.ml.chardetect.StructuralEncodingRules;
import org.apache.tika.parser.ParseContext;
@@ -551,17 +552,18 @@ public class JunkFilterEncodingDetector implements
MetaEncodingDetector {
private byte[] readProbe(TikaInputStream tis, EncodingDetectorContext
context)
throws IOException {
// readLimit is the tag-stripped content target; cap raw reads at 512
KB.
+ // Same kind as Mojibuster's probe: the filter must weigh the evidence
the model saw.
int rawCap = AdaptiveProbe.DEFAULT_RAW_CAP;
EncodingProbeCache cache = context == null ? null :
context.getProbeCache();
if (cache != null) {
- byte[] cached = cache.get(readLimit, rawCap);
+ byte[] cached = cache.get(EvidenceProbe.KIND, readLimit, rawCap);
if (cached != null) {
return cached.length == 0 ? null : cached;
}
}
- byte[] probe = AdaptiveProbe.read(tis, readLimit, rawCap);
+ byte[] probe = EvidenceProbe.read(tis, readLimit, rawCap);
if (cache != null) {
- cache.put(probe, readLimit, rawCap);
+ cache.put(probe, EvidenceProbe.KIND, readLimit, rawCap);
}
return probe.length == 0 ? null : probe;
}
diff --git
a/tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/JunkFilterCandidatesTest.java
b/tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/JunkFilterCandidatesTest.java
index b568ecaa33..74454dc70d 100644
---
a/tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/JunkFilterCandidatesTest.java
+++
b/tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/JunkFilterCandidatesTest.java
@@ -16,6 +16,7 @@
*/
package org.apache.tika.ml.junkdetect;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -30,6 +31,8 @@ import org.apache.tika.detect.EncodingDetectorContext;
import org.apache.tika.detect.EncodingResult;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
+import org.apache.tika.ml.chardetect.AdaptiveProbe;
+import org.apache.tika.ml.chardetect.EvidenceProbe;
import org.apache.tika.parser.ParseContext;
/** {@link JunkFilterEncodingDetector#candidates} reports what {@link
JunkFilterEncodingDetector#detect} acts on. */
@@ -130,4 +133,22 @@ public class JunkFilterCandidatesTest {
assertEquals(List.of(UTF8, W1252), c.getBandCharsets());
assertTrue(c.getTexts().keySet().containsAll(c.getBandCharsets()));
}
+
+ /** The filter weighs the same evidence probe as the charset model, not
the bare prefix. */
+ @Test
+ public void filterSeesTheEvidenceProbe() throws Exception {
+ byte[] b = ("x".repeat(40_000) + " Съешь же ещё этих мягких
французских булок".repeat(10))
+ .getBytes(UTF8);
+ byte[] evidence;
+ try (TikaInputStream tis = TikaInputStream.get(b)) {
+ evidence = EvidenceProbe.read(tis,
AdaptiveProbe.DEFAULT_CONTENT_TARGET,
+ AdaptiveProbe.DEFAULT_RAW_CAP);
+ }
+ EncodingDetectorContext ctx = context(r(UTF8,
EncodingResult.ResultType.STATISTICAL),
+ r(W1252, EncodingResult.ResultType.STATISTICAL));
+ assertArrayEquals(evidence, candidates(b, ctx).getProbe());
+ assertArrayEquals(evidence, ctx.getProbeCache().get(EvidenceProbe.KIND,
+ AdaptiveProbe.DEFAULT_CONTENT_TARGET,
AdaptiveProbe.DEFAULT_RAW_CAP));
+ assertTrue(candidates(b,
ctx).getTexts().get(UTF8).contains("французских"));
+ }
}
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/config/TikaEncodingDetectorTest.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/config/TikaEncodingDetectorTest.java
index 2f7160d63e..2d1566e2fa 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/config/TikaEncodingDetectorTest.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/config/TikaEncodingDetectorTest.java
@@ -228,8 +228,8 @@ public class TikaEncodingDetectorTest extends TikaTest {
public void testMarkLimitIntegration() throws Exception {
StringBuilder sb = new StringBuilder();
sb.append("<html><head><script>");
- // ~600 KB of script: past mojibuster's 512 KB probe and the html mark
- // limit, so the buried meta + UTF-8 body aren't reached by default.
+ // ~600 KB of script: past mojibuster's 512 KB prefix and the html mark
+ // limit, so the buried meta isn't reached by default.
for (int i = 0; i < 120000; i++) {
sb.append("blah ");
}
@@ -241,19 +241,17 @@ public class TikaEncodingDetectorTest extends TikaTest {
byte[] bytes = sb.toString().getBytes(StandardCharsets.UTF_8);
- // Default: meta buried at ~byte 600,000, past mojibuster's probe and
the
- // html mark limit, so mojibuster sees pure ASCII and returns
windows-1252.
- // Entities (ø) survive; raw UTF-8 (ø in "økologisk") is garbled.
- // Raised mark limit fixes it (see below).
+ // Default: the meta at ~byte 600,000 is past the html mark limit, but
+ // mojibuster's evidence probe reaches the UTF-8 body (TIKA-5046).
Parser p = AUTO_DETECT_PARSER;
Metadata metadata = new Metadata();
String xml = getXML(TikaInputStream.get(bytes), p, metadata).xml;
- assertContains("gr\u00F8nd", xml); // ø entity — correct
regardless
- assertNotContained("\u00f8kologisk", xml); // raw UTF-8 bytes —
garbled by default
- assertNotContained("gr\u00F8nt", xml);
- assertNotContained("g\u00E5 til", xml);
+ assertContains("gr\u00F8nd", xml);
+ assertContains("\u00f8kologisk", xml);
+ assertContains("gr\u00F8nt", xml);
+ assertContains("g\u00E5 til", xml);
// Raised mark limit reaches the meta and decodes UTF-8.
p =
TikaLoaderHelper.getLoader("TIKA-2485-encoding-detector-mark-limits.json").loadAutoDetectParser();
