This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new 74249f86e6 TIKA-5027: Junk retrain v113 (#3571)
74249f86e6 is described below
commit 74249f86e6ad0bd69a1bb073312758bddb90a25b
Author: Tim Allison <[email protected]>
AuthorDate: Mon Nov 9 08:54:21 2026 -0500
TIKA-5027: Junk retrain v113 (#3571)
* TIKA-5052, TIKA-5027: charset model trained on fixed data; no declared
anchor with U+FFFD
Charset model (seed 7 of the data-fix A/B, gold v1.12 recipe):
scattered-NUL text dropped
from the binary negatives; UTF-16 positives with NUL padding, U+0000
separators, C0 code
units and vocalized Arabic (Arabic Wikisource); Iberian Wikisource in
windows-1252,
Kurmanji in windows-1254; sparse single NULs labelled by charset; tier B
identical
decodes trained as windows-1252. Junk arbitration does not defer to a
declared charset
whose decode has U+FFFD the champion lacks. Junk trainer misreads each
text with two
more charsets the chain can offer. Training script reads gold v1.13.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
CHANGES.txt | 12 +++++
.../pages/advanced/charset-detection-design.adoc | 5 +-
.../ml/chardetect/MojibusterEncodingDetector.java | 4 +-
.../org/apache/tika/ml/chardetect/hashed-ngram.hng | Bin 2742509 -> 2742509
bytes
.../tika/ml/chardetect/hashed-ngram.hng.provenance | 19 +++++---
.../HashedNgramEncodingDetectorTest.java | 26 +++++++---
.../tika/ml/chardetect/MacRomanDetectionTest.java | 53 +++++++++++++++++++++
.../tika/ml/chardetect/hashed-ngram-parity.tsv | 24 +++++-----
.../tika/ml/junkdetect/tools/JunkFamilies.java | 28 ++++++++++-
.../tika-ml-junkdetect-tools/train-junk-model.sh | 2 +-
.../tika/ml/junkdetect/HashedArbitration.java | 9 ++++
.../org/apache/tika/ml/junkdetect/junkdetect.bin | Bin 3060035 -> 3058253
bytes
.../tika/ml/junkdetect/junkdetect.bin.provenance | 46 +++++++++---------
.../tika/ml/junkdetect/HashedArbitrationTest.java | 47 ++++++++++++++++++
14 files changed, 221 insertions(+), 54 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 716c93528a..bdb9ab009f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -376,6 +376,18 @@ Release 4.3.0 - ???
cut at the end of the sample is allowed); its next most likely charset is
used. Short
CJK, Greek and Thai samples were often called binary (TIKA-5052).
+ * Charset detection reads BOM-less UTF-16 with trailing NUL padding, U+0000
separators,
+ stray control code units or fully vocalized Arabic as UTF-16, not binary;
Spanish,
+ Portuguese and Catalan text with ¡ ¿ º ª as windows-1252, not ISO-8859-3;
Kurmanji
+ as windows-1254; and text with a few stray NUL bytes as text (TIKA-5052).
+
+ * Charset arbitration no longer defers to a declared charset whose decode
has more
+ replacement characters than the best-scoring decode, such as a page
declared
+ iso-8859-3 but written in windows-1252 (TIKA-5027).
+
+ * Charset detection keeps x-MacRoman instead of relabelling it windows-1252
when the
+ text has MacRoman curly quotes (TIKA-5052).
+
* XHTML output no longer contains empty or whitespace-only
b/strong/i/em/sup/sub
elements, and whitespace at their edges is written outside them (<b>Note:
</b>
becomes <b>Note:</b> ); u and s are unchanged. The Markdown writer does
the same for
diff --git a/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
b/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
index ba0095c4e7..5a82d538a6 100644
--- a/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
+++ b/docs/modules/ROOT/pages/advanced/charset-detection-design.adoc
@@ -232,8 +232,9 @@ Otherwise the whole probe is scored.
* **Probes under 2 bytes**, or an empty pool → windows-1252 at 0.1 confidence.
The WHATWG default; the detector never returns an empty result.
* **Latin-sibling rewrite** — if the top candidate is a Western Latin sibling
- (x-MacRoman, IBM850, ISO-8859-15) and the probe has at least as many cased
- letters at high-byte positions under windows-1252, rewrite it to
windows-1252.
+ (IBM850, ISO-8859-15) and the probe has at least as many cased letters at
+ high-byte positions under windows-1252, rewrite it to windows-1252.
x-MacRoman
+ is exempt: its curly quotes (0xD2-0xD5) are letters under windows-1252.
==== CJK decode-failure veto (`CjkDecodeValidator`)
diff --git
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/MojibusterEncodingDetector.java
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/MojibusterEncodingDetector.java
index ef3fb44b0b..71249b4c57 100644
---
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/MojibusterEncodingDetector.java
+++
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/java/org/apache/tika/ml/chardetect/MojibusterEncodingDetector.java
@@ -544,7 +544,9 @@ public class MojibusterEncodingDetector implements
EncodingDetector {
// letter-count compare ties on most of their Latin letters
// because Unicode classifies both decodings as Letters,
// misleading the rule into a wrong flip).
- if (!CharsetConfusables.WESTERN_LATIN_FAMILY.contains(topName)) {
+ // MacRoman's curly quotes (0xD2-0xD5) are letters under windows-1252,
so the count
+ // below always favours windows-1252 on real MacRoman.
+ if (!CharsetConfusables.WESTERN_LATIN_FAMILY.contains(topName) ||
"x-MacRoman".equals(topName)) {
return ranked;
}
Charset win1252 = Charset.forName(WIN1252);
diff --git
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/hashed-ngram.hng
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/hashed-ngram.hng
index bbfdc9b193..4c2617d6aa 100644
Binary files
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/hashed-ngram.hng
and
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/hashed-ngram.hng
differ
diff --git
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/hashed-ngram.hng.provenance
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/hashed-ngram.hng.provenance
index 6903be5187..b4f63e32e9 100644
---
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/hashed-ngram.hng.provenance
+++
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/main/resources/org/apache/tika/ml/chardetect/hashed-ngram.hng.provenance
@@ -13,11 +13,11 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-model_sha256 2342e4c26e1a09371c9d80f616e4a4e89397484e24fa883b5d72298f35060b5f
-model_md5 9915a5d2d98f7128d7419c82127fdf08
+model_sha256 b1a64c01b6404c1dbe9644a054907e4975e9cb527b12c31895612806f72ebeb4
+model_md5 7cd1606fdff904d026678e966c18dab6
model_bytes 2742509
format HNG2 (see tika-ml/tika-ml-chardetect/README.md)
-status final-v112 "R", seed 11, 2026-11-05; trained on gold v1.12
+status datafix "ALL", seed 7, 2026-11-07; trained on gold v1.12 (R-s11 recipe
plus the data fixes below)
Architecture
43 classes: 37 legacy/UTF-8 charsets (IBM420/IBM424 as -ltr/-rtl pairs),
IBM437, binary,
@@ -33,11 +33,16 @@ Training data (nothing from the 1.2M regression corpus)
Gold of record v1.12 (Common Crawl HTML, Common Crawl non-HTML, CC v3) fit
part only:
tier A, re-encoded non-HTML UTF-8 units (weight 0.2); tier B, real
legacy-charset bytes (weight 0.2).
UTF-16LE/BE positives (generated, all scripts, with/without BOM, emoji
rates, lengths from 16 bytes,
- even-aligned crops).
+ even-aligned crops), plus rows with trailing U+0000 padding, U+0000
separators and C0 code units, and
+ fully vocalized Arabic (Arabic Wikisource, train buckets by book).
+ Vocalized Arabic in windows-1256 / ISO-8859-6;
Spanish/Portuguese/Catalan/Galician Wikisource and Spanish
+ Wikiquote in windows-1252 (dialogue/literary text with ¡ ¿ º ª); Kurmanji
Wikipedia in windows-1254.
+ Text with sparse single NULs (<= 2%) labelled by its charset.
+ Tier B rows whose label set decodes identically take windows-1252 as a hard
target.
Negatives: generated UTF-32LE/BE rows (own classes); generated binary (NUL
records, deflate, bytecode,
- integer arrays, random, executables, scattered NULs) plus the gold v1.12 fit
binary-verdict documents
+ integer arrays, random, executables; the scattered-NUL text family was
removed) plus the gold v1.12 fit binary-verdict documents
(1,123) as class "binary".
Web probes went through the production HTML strip rule (maybeStripHtml, no
content type).
- Selection on held-out train-side sets (gold v1.12 train sel split, Wikipedia
bucket 8) by a rule fixed
- before training; gold v1.12 dev scored once, after the freeze. Test and
sealed splits and Wikipedia
+ Selection on held-out train-side sets only (gold v1.12 train sel split,
Wikipedia/Wikisource bucket 8,
+ real regression-corpus failures as eval only); seed chosen by the sel score.
Dev not scored. Test and sealed splits and Wikipedia
buckets 0 and 9 were not read.
diff --git
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/HashedNgramEncodingDetectorTest.java
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/HashedNgramEncodingDetectorTest.java
index c88bc54215..b355c0535e 100644
---
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/HashedNgramEncodingDetectorTest.java
+++
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/HashedNgramEncodingDetectorTest.java
@@ -46,12 +46,19 @@ public class HashedNgramEncodingDetectorTest {
@Test
public void greekCropCutMidCharacterIsUtf8() {
- assertUtf8NotBinary(crop("Η Ελλάδα είναι χώρα της νοτιοανατολικής
Ευρώπης.", 32));
+ assertUtf8(crop("Η Ελλάδα είναι χώρα της νοτιοανατολικής Ευρώπης.",
32));
}
@Test
public void japaneseCropCutMidCharacterIsUtf8() {
- assertUtf8NotBinary(crop("日本は東アジアに位置する島国であり、首都は東京である。", 16));
+ assertUtf8(crop("日本は東アジアに位置する島国であり、首都は東京である。", 16));
+ }
+
+ @Test
+ public void textLikeSymbolRunIsVetoedToUtf8() {
+ byte[] probe = "↦⇙⇙ ⇽⇻".getBytes(StandardCharsets.UTF_8);
+ assertEquals("binary", rawTop(probe), "probe no longer exercises the
veto");
+ assertUtf8(probe);
}
@Test
@@ -71,8 +78,16 @@ public class HashedNgramEncodingDetectorTest {
public void utf8WithDisallowedControlMayBeBinary(int control) {
byte[] probe = crop("Η Ελλάδα είναι χώρα της νοτιοανατολικής
Ευρώπης.", 32);
probe[probe.length - 1] = (byte) control;
- assertEquals("binary", rawTop(probe));
- assertTrue(detector.detect(probe).isEmpty());
+ assertFalse(textLike(probe));
+ // not text-like: the veto leaves the model's call alone
+ List<EncodingResult> results = detector.detect(probe);
+ if ("binary".equals(rawTop(probe))) {
+ assertTrue(results.isEmpty());
+ } else {
+ assertFalse(results.isEmpty());
+
assertEquals(HashedNgramEncodingDetector.toJavaCharsetName(rawTop(probe)),
+ results.get(0).getCharset().name());
+ }
}
@Test
@@ -98,8 +113,7 @@ public class HashedNgramEncodingDetectorTest {
assertFalse(textLike(new byte[] {(byte) 0xE4, 'a', (byte) 0xE4, (byte)
0xB8}));
}
- private static void assertUtf8NotBinary(byte[] probe) {
- assertEquals("binary", rawTop(probe), "probe no longer exercises the
veto");
+ private static void assertUtf8(byte[] probe) {
List<EncodingResult> results = detector.detect(probe);
assertFalse(results.isEmpty());
assertEquals(StandardCharsets.UTF_8, results.get(0).getCharset(),
results.toString());
diff --git
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/MacRomanDetectionTest.java
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/MacRomanDetectionTest.java
new file mode 100644
index 0000000000..4be9cfc497
--- /dev/null
+++
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/java/org/apache/tika/ml/chardetect/MacRomanDetectionTest.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.ml.chardetect;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+
+import java.nio.charset.Charset;
+import java.util.List;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.tika.detect.EncodingResult;
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.ParseContext;
+
+public class MacRomanDetectionTest {
+
+ /**
+ * MacRoman curly quotes (0xD2-0xD5) are letters (Ò Ó Ô Õ) in
windows-1252; the model's
+ * MacRoman call must not be rewritten to windows-1252 on that account.
+ */
+ @Test
+ public void curlyQuotesStayMacRoman() throws Exception {
+ Charset mac = Charset.forName("x-MacRoman");
+ byte[] probe = ("<html><body><p>The Earth’s climate system is driven
by energy from the Sun."
+ + " Scientists call this the “greenhouse effect”, and it’s why
the planet"
+ + " doesn’t freeze at night. The ‘carbon cycle’ moves carbon
between the"
+ + " atmosphere, the oceans and the land; it isn’t a closed
loop, and today’s"
+ + " emissions add to it faster than it can
respond.</p></body></html>").getBytes(mac);
+ List<EncodingResult> results;
+ try (TikaInputStream tis = TikaInputStream.get(probe)) {
+ results = new MojibusterEncodingDetector().detect(tis, new
Metadata(), new ParseContext());
+ }
+ assertFalse(results.isEmpty());
+ assertEquals(mac, results.get(0).getCharset(), results.toString());
+ }
+}
diff --git
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/resources/org/apache/tika/ml/chardetect/hashed-ngram-parity.tsv
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/resources/org/apache/tika/ml/chardetect/hashed-ngram-parity.tsv
index f77694eef8..f0a107191a 100644
---
a/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/resources/org/apache/tika/ml/chardetect/hashed-ngram-parity.tsv
+++
b/tika-encoding-detectors/tika-encoding-detector-mojibuster/src/test/resources/org/apache/tika/ml/chardetect/hashed-ngram-parity.tsv
@@ -14,16 +14,16 @@
# limitations under the License.
# Parity fixture for HashedNgramModelParityTest: probe name, probe bytes
(hex), reference logits.
# Regenerate with the reference scorer whenever the model changes.
-model-md5 9915a5d2d98f7128d7419c82127fdf08
+model-md5 7cd1606fdff904d026678e966c18dab6
ascii-abstain
506c61696e204153434949206f6e6c792c206e6f7468696e6720746f2073636f72652e0d0a
abstain
-windows-1252
4c6520636166e920657374207472e87320626f6e2c206d657263692062656175636f757020706f757220766f747265206163637565696c20e02050617269732e
-4.4688854,-1.1962241,-6.8784375,-7.0891438,-6.8860435,-7.0042973,-6.295397,-6.677034,-9.23016,-1.0490059,-5.162087,-6.4058795,-6.264429,2.5200534,-2.1321251,2.4542825,0.8207326,-5.9856725,-3.1178734,-3.8439894,-1.8840295,-9.749587,-3.3997228,-3.0048518,-0.6965329,4.650988,-5.737924,4.389301,-2.887215,-6.7846203,-3.017495,1.2942129,-3.3867366,-1.73
[...]
-UTF-8
4772c3bcc39f6520617573204dc3bc6e6368656e20e2809420736368c3b66e65204772c3bcc39f652c20e697a5e69cace8aa9e2e
-0.35387138,-9.074315,1.1373283,-13.87538,-7.63868,-7.873353,-20.866707,-20.624247,-16.06933,-8.568505,-14.014507,-1.2421216,-3.984268,-17.662807,-8.218629,-9.616811,-7.3049345,-11.481492,-14.388479,1.8598406,-5.0547886,-6.5030265,14.140795,-9.224589,-11.732056,-19.556551,-11.205365,-11.195001,-5.115281,-14.344123,-13.072593,-10.280072,1.1228199,-2.8632386,-6.5386357,-6.6816797,
[...]
-Shift_JIS
93fa967b8cea82cc8365834c8358836782c582b7814282b182ea82cd83658358836782c582b78142
-4.478262,-15.82093,-6.2672405,-7.190492,-5.392521,-5.0362372,-1.287788,-1.3407742,-4.5284324,-6.595331,-5.2293854,-1.6678491,-5.823333,-13.604709,-9.75838,-34.490944,-16.08255,-3.512752,-14.028969,-6.1464624,-8.216362,19.860409,-3.5907722,2.1841571,-4.472017,-20.135654,-2.7374072,-19.430166,-2.996345,0.6498822,-15.1307535,-15.825123,-8.410225,-2.0418582,-10.0603285,-0.35763884,-20.054491,-18.91693
[...]
-ISO-8859-7 b6f1e8f1e120e3e9e120f4e7ed20f0f1fcebe7f8e720f4e7f220e2dfe1f22e
-6.633405,-3.55985,0.32556352,2.3095713,-8.9727125,-8.471653,-11.198448,-11.261342,-0.20689417,-13.464438,-7.7593527,-0.1951622,-1.6554296,-10.55582,-0.86260265,-5.61697,-0.4340254,-2.9107993,13.115026,0.9292874,1.1878953,-6.9674454,-4.865222,-11.827286,-2.68292,-13.912327,7.360378,-22.053299,-0.65438294,-3.4692101,-10.655784,-7.612983,-1.4319738,-1.4997764,-18.036194,-0.57111627,0.187817,-13.4236,6.083128,-3.94730
[...]
-windows-1251
cdeee2e0ff20eff0eee3f0e0ecece020efee20eff0eef4e8ebe0eaf2e8eae520ede0f1e8ebe8ff2e
-16.236979,-8.145807,-7.0906777,-1.3168094,-16.799265,-16.29536,-9.73359,-10.40736,-4.8865776,-11.628687,-9.201777,-2.9745672,-5.111637,-13.309752,-13.9698305,-17.580523,-1.7027507,-3.1087925,-0.46587533,-4.288209,-1.3876741,-11.989142,-8.694642,-17.264072,18.559114,-7.856275,-2.6807668,-18.323345,-1.0936506,-4.8127966,-6.321303,-1.7308524,-4.5960383,-4.6365757,-9.124981,5.289506,-2.18139,-22.19
[...]
-IBM500
c38583894085a2a340a49540a385a7a38540859540c5c2c3c4c9c36b4081a58583408485a2408183838595a3a27a4051404440484b
-3.498919,-5.3964705,0.079976976,11.45065,7.0824103,7.108503,11.099829,11.175525,12.368509,-9.478295,0.6079704,-9.127392,-4.7296834,-8.077731,-13.934918,-10.112591,-11.850968,-14.913987,-2.6195552,-8.415146,-7.271687,-0.77081317,-7.362428,-11.254117,-6.073449,-13.7920265,-3.2923772,-12.256852,-13.210939,-3.03515,-14.304068,-11.834185,-11.33522,-6.8807254,-7.1075034,-3.325489,
[...]
-UTF-16LE-emoji
53006d0069006c0069006e0067002000660061006300650020003dd800de2c0020006e0075006d00620065007200200031002c0030007800300039002c002a000d000a00
-21.26249,-21.522182,-3.3044417,-18.740128,-19.147,-18.562986,-24.104496,-23.933065,-17.311255,-16.61437,-13.140314,-7.819072,-18.208033,-25.037186,-23.779665,-30.634981,-7.983687,-25.697887,-14.498555,-1.036621,-8.248244,-20.242086,-4.0537415,-14.631187,-4.8367653,-14.573564,-7.4878645,-16.821867,-7.06777,-15.4407425,-15.122156,-17.5609,
[...]
-UTF-16BE-thai 0e2a0e270e310e2a0e140e350e040e230e310e1a
-11.833257,-17.552015,-0.97335124,-4.677023,-3.8056273,-4.18433,-4.960505,-5.033229,-3.4578817,-21.214819,-6.918234,-6.2184033,-22.431446,-22.143135,-17.479528,-27.809132,-7.8366156,-13.93506,-10.067974,-2.1847234,-5.630471,-11.53564,-7.65749,-9.863378,-4.56702,-10.290909,-1.4330521,-10.970513,-15.698202,-2.6214354,-18.792316,-12.538482,-6.9780416,-2.9793653,-12.760799,-13.273045,-9.636212,-23.195158,11.2868395,-1.1321154,23.425142,-
[...]
-UTF-32LE
48000000650000006c0000006c0000006f00000020000000770000006f000000720000006c000000640000002000000000f60100
-22.281254,-10.983885,-14.642101,-14.84429,-13.07636,-12.862815,-19.802914,-20.313585,-23.503405,-19.49819,-28.929182,-19.41122,-20.78039,-5.6425943,-16.548326,-29.965017,-15.5855665,-21.226997,-18.902283,-5.3892255,-12.7358055,-20.30458,-7.2728114,-6.146436,-10.311509,-15.186714,-15.478616,-9.066836,0.056835424,-17.214111,-14.747776,-16.287947,-12.326821,-5.9938016,-3.913244
[...]
-UTF-16LE-long
4c006f006e006700200065006e006f00750067006800200074006f00200070006100730073002000740068006500200035003100320020006200790074006500200063006f006c0075006d006e002000770069006e0064006f0077002e0020004c006f006e006700200065006e006f00750067006800200074006f00200070006100730073002000740068006500200035003100320020006200790074006500200063006f006c0075006d006e002000770069006e0064006f0077002e0020004c006f006e006700200065006e006f00750067006800200074006f0020007000610073007300200074006800650020
[...]
-random
2dbf9f901511db1b9499ddf6b72b5c3544cadbf085b0a22369a741858d27322dc88a3686eb481d5d30a2528592acb75beeed34672f90906a585185ab9bfaa434a38cded677964e3a9c3cd417e59921a304e491d98507bab2995b7f66121591971321ff775c9e026ca0aa5ee871cfe23fa67357f22aafc210cdb80826827b12d5a21c2a6461ebd0ed58838b9bb8f55aa769ab474b26c427c92a32878780877582901e974cef0ef669b3895fcd82d65b30d5db4fa866d00257fa2532702604a3db50b3c67e9451c85ef5d28c9059c1ea2ee7612b2f3be3480c1a96c02a905a43250647774d82f3e16e353d63fc0c001aac9041a
[...]
+windows-1252
4c6520636166e920657374207472e87320626f6e2c206d657263692062656175636f757020706f757220766f747265206163637565696c20e02050617269732e
-6.26243,-8.276049,-7.5950513,-15.685164,-18.508417,-18.136816,-11.677787,-11.436772,-11.429723,-1.6022514,-0.6278504,-2.9343238,-3.2420566,4.612255,-1.5458668,3.170168,-2.1325564,0.52321064,-1.605255,-4.855722,-6.62645,-14.765057,-5.6225595,-1.4943169,-2.3204155,7.4401264,-2.5516288,6.9778447,-0.6250175,-4.572941,-0.37654838,1.202957,-0.26864636,-
[...]
+UTF-8
4772c3bcc39f6520617573204dc3bc6e6368656e20e2809420736368c3b66e65204772c3bcc39f652c20e697a5e69cace8aa9e2e
-0.5827861,-6.6113305,0.045845836,-9.239936,-5.9060984,-5.5219054,-9.469475,-9.633721,-8.441814,-7.54745,-13.777663,-4.715149,-3.4075067,-15.77235,-16.508482,-18.398487,-12.1268635,-15.299498,-14.69563,0.13925156,-9.128676,-5.489176,14.18173,-10.594682,-4.7317257,-15.118749,-12.118952,-22.418028,-21.3163,-10.658545,-15.678664,-10.982281,-4.5749016,-1.7791243,-8.110289,-4.906046,
[...]
+Shift_JIS
93fa967b8cea82cc8365834c8358836782c582b7814282b182ea82cd83658358836782c582b78142
-7.482871,-6.7766385,4.062569,3.0263946,-7.778361,-7.413558,-3.5662513,-3.9060385,0.2243073,-13.015116,-18.522057,-14.260608,-3.560188,-18.54181,-17.02574,-40.14331,-13.6332655,-19.063444,-9.753614,-0.43981814,-7.344433,20.272371,-3.1040049,-1.7563919,-3.9684286,-15.076569,4.173208,-24.263617,-6.0777817,-9.948965,-29.124353,-13.09328,0.19826484,2.1880326,-17.013842,-5.945684,-19.390018,-12.687593,4
[...]
+ISO-8859-7 b6f1e8f1e120e3e9e120f4e7ed20f0f1fcebe7f8e720f4e7f220e2dfe1f22e
-0.8203472,-7.461944,-1.9833272,-2.4042392,-7.2492504,-7.193231,-8.779393,-9.237593,1.1447276,-15.360343,-27.163002,-3.4032931,-5.120804,-11.597296,-6.46171,-13.295239,-2.9843912,-4.196037,16.045206,-3.2323692,0.98040044,-2.3271854,-10.323003,-10.868758,-4.4015923,-11.062085,10.325654,-13.378147,-3.8846247,-4.175025,-19.496069,-7.166667,-6.739304,0.17751911,-13.286078,-5.898166,-4.800658,-19.094826,-2.1375165,-3.95
[...]
+windows-1251
cdeee2e0ff20eff0eee3f0e0ecece020efee20eff0eef4e8ebe0eaf2e8eae520ede0f1e8ebe8ff2e
-12.438238,-10.242093,-1.2998546,-5.962235,-13.427248,-12.878796,-16.230131,-17.201805,-1.0009803,-24.018206,-23.622522,-5.2191486,-4.1623316,-11.167844,-13.7162,-26.255186,-2.1109056,0.028141916,-3.6518037,1.7760476,-7.4237423,-13.722433,-3.9797523,-12.691779,21.79608,-13.254122,-0.24368778,-16.654087,-0.72549474,-7.4011917,-5.029045,-7.342651,-3.902788,-2.1215475,-17.628601,4.9885983,-9.793963
[...]
+IBM500
c38583894085a2a340a49540a385a7a38540859540c5c2c3c4c9c36b4081a58583408485a2408183838595a3a27a4051404440484b
-7.6649594,-5.1355267,-0.059511315,15.283144,6.6421494,6.853642,10.779206,10.726931,14.810395,-10.952078,-24.543068,-11.163412,0.65159315,-14.676382,-17.094334,-29.665552,-10.920232,-19.944944,-10.267533,1.8090932,-8.857017,5.5395036,0.2454361,-10.220937,-3.9662151,-13.020994,-1.419175,-23.684883,-2.9006724,-22.812782,-24.694105,-11.838569,-5.818771,-2.858652,-14.0329075,-2.4
[...]
+UTF-16LE-emoji
53006d0069006c0069006e0067002000660061006300650020003dd800de2c0020006e0075006d00620065007200200031002c0030007800300039002c002a000d000a00
-15.003139,-17.027376,-3.51826,-5.1978855,-10.307344,-10.29958,-16.53273,-16.69455,-3.9325695,-9.80203,-23.2849,0.0851226,-7.799486,-19.536577,-15.215047,-18.650007,-9.40576,-9.536814,-7.5072374,-3.7153225,-5.7540827,-12.313613,-8.299613,-12.684108,-5.6328754,-11.970228,-2.3121173,-12.124944,-4.9368258,-3.026019,-13.056631,-10.524815,-13.
[...]
+UTF-16BE-thai 0e2a0e270e310e2a0e140e350e040e230e310e1a
-11.964084,-20.762892,-3.1328537,-18.356253,-8.522627,-8.178322,-15.8935585,-16.665577,-19.966204,-11.75348,-4.8390594,-7.3139324,-13.296401,-18.660704,-11.799979,-15.680888,-11.938468,-4.528831,-15.936991,-7.375014,-5.5594296,-17.357899,-4.5168223,-5.2730837,-2.0711021,-9.265343,-7.2035103,-11.161968,-15.081558,2.4231293,-3.8285866,-6.273818,-14.419546,-13.408076,-6.7828083,-5.356897,-13.405363,-12.566665,8.712523,3.4173558,23.47274
[...]
+UTF-32LE
48000000650000006c0000006c0000006f00000020000000770000006f000000720000006c000000640000002000000000f60100
-17.05875,-14.528189,-15.666076,-0.61441404,-7.322275,-7.337266,-11.801676,-11.66604,3.42458,-15.463148,-25.011389,-3.5088234,-0.9584222,-26.449154,-28.674969,-18.125229,-9.030397,-15.152163,-14.510708,1.0818806,-3.7085783,-14.689876,-8.415785,-23.006014,-11.733483,-11.460316,-8.38566,-24.200026,-0.6148938,-12.609276,-14.809252,-22.7466,-22.387167,-9.81124,-7.145131,-5.720913
[...]
+UTF-16LE-long
4c006f006e006700200065006e006f00750067006800200074006f00200070006100730073002000740068006500200035003100320020006200790074006500200063006f006c0075006d006e002000770069006e0064006f0077002e0020004c006f006e006700200065006e006f00750067006800200074006f00200070006100730073002000740068006500200035003100320020006200790074006500200063006f006c0075006d006e002000770069006e0064006f0077002e0020004c006f006e006700200065006e006f00750067006800200074006f0020007000610073007300200074006800650020
[...]
+random
2dbf9f901511db1b9499ddf6b72b5c3544cadbf085b0a22369a741858d27322dc88a3686eb481d5d30a2528592acb75beeed34672f90906a585185ab9bfaa434a38cded677964e3a9c3cd417e59921a304e491d98507bab2995b7f66121591971321ff775c9e026ca0aa5ee871cfe23fa67357f22aafc210cdb80826827b12d5a21c2a6461ebd0ed58838b9bb8f55aa769ab474b26c427c92a32878780877582901e974cef0ef669b3895fcd82d65b30d5db4fa866d00257fa2532702604a3db50b3c67e9451c85ef5d28c9059c1ea2ee7612b2f3be3480c1a96c02a905a43250647774d82f3e16e353d63fc0c001aac9041a
[...]
diff --git
a/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/JunkFamilies.java
b/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/JunkFamilies.java
index 6487db5342..749e93e534 100644
---
a/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/JunkFamilies.java
+++
b/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/JunkFamilies.java
@@ -49,13 +49,28 @@ final class JunkFamilies {
static final List<String> CROSS = List.of("UTF-8", "windows-1252",
"windows-1250", "windows-1251",
"GBK", "Shift_JIS", "windows-1253");
+ /**
+ * The other ASCII-compatible labels the charset model can return; {@link
#EXTRA_DECODES} of
+ * these are added per text, so every candidate the chain offers appears
as a wrong decode
+ * (Shift_JIS tables were lost to x-MacCyrillic, which no CJK text was
misread with) without
+ * thinning the common pairs above.
+ */
+ static final List<String> EXTRA = List.of("windows-1254", "windows-1255",
"windows-1256",
+ "windows-1257", "windows-1258", "x-windows-874", "ISO-8859-2",
"ISO-8859-3", "ISO-8859-5",
+ "ISO-8859-6", "ISO-8859-7", "ISO-8859-16", "KOI8-R", "KOI8-U",
"IBM437", "IBM850", "IBM852",
+ "IBM855", "IBM866", "x-MacRoman", "x-MacCyrillic", "GB18030",
"Big5-HKSCS", "x-EUC-TW",
+ "EUC-JP", "x-windows-949");
+
+ static final int EXTRA_DECODES = 2;
+
private JunkFamilies() {
}
/**
* Misreadings of {@code text}: encode it in one plausible source charset
(UTF-8 or a
* charset of its script that can encode it), decode with every other
candidate, keep those
- * that differ; shuffled, at most {@code max}.
+ * that differ; shuffled, at most {@code max}, plus up to {@link
#EXTRA_DECODES} from
+ * {@link #EXTRA}.
*/
static List<String> wrongDecodes(String text, String script, Random r, int
max) {
List<String> family = SCRIPT_CHARSETS.getOrDefault(script, List.of());
@@ -71,6 +86,15 @@ final class JunkFamilies {
LinkedHashSet<String> decoders = new LinkedHashSet<>(family);
decoders.addAll(CROSS);
decoders.remove(source);
+ List<String> out = misreads(b, text, decoders, r, max);
+ LinkedHashSet<String> extra = new LinkedHashSet<>(EXTRA);
+ extra.removeAll(decoders);
+ extra.remove(source);
+ out.addAll(misreads(b, text, extra, r, EXTRA_DECODES));
+ return out;
+ }
+
+ private static List<String> misreads(byte[] b, String text,
Iterable<String> decoders, Random r, int max) {
List<String> out = new ArrayList<>();
for (String d : decoders) {
String m = new String(b, Charset.forName(d));
@@ -79,7 +103,7 @@ final class JunkFamilies {
}
}
Collections.shuffle(out, r);
- return out.size() > max ? out.subList(0, max) : out;
+ return out.size() > max ? new ArrayList<>(out.subList(0, max)) : out;
}
/** One junk version of {@code text} from {@code family}; {@code null} if
none applies. */
diff --git a/tika-ml/tika-ml-junkdetect-tools/train-junk-model.sh
b/tika-ml/tika-ml-junkdetect-tools/train-junk-model.sh
index ba0c05e28a..c349dcc0a2 100755
--- a/tika-ml/tika-ml-junkdetect-tools/train-junk-model.sh
+++ b/tika-ml/tika-ml-junkdetect-tools/train-junk-model.sh
@@ -30,7 +30,7 @@
set -euo pipefail
CORPUS=junk-clean-20261103 # Wikipedia
sentences per language
-GOLD=charsets/gold-v112-train-halves.tsv # gold v1.12
cc-html train, even/odd halves, damage spans
+GOLD=charsets/gold-v113-train-halves.tsv # gold v1.13
cc-html train, even/odd halves, damage spans
SAMPLE=charsets/cchtml-train-unanimous/sample-gen-20261103.tsv # cc-html
train pages for real-decision pairs
SAMPLE_DOCS=charsets/cchtml-train-unanimous/docs
diff --git
a/tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/HashedArbitration.java
b/tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/HashedArbitration.java
index 3142357587..30a93fc885 100644
---
a/tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/HashedArbitration.java
+++
b/tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/HashedArbitration.java
@@ -81,10 +81,19 @@ public final class HashedArbitration {
logit = ld[0];
difference = (long) ld[1];
}
+ // a declaration whose decode has more replacement characters than the
champion's is a failed decode, not an anchor
+ if (declared != null && texts.containsKey(declared)
+ && replacements(texts.get(declared)) >
replacements(championText)) {
+ declared = null;
+ }
return new Decision(champion, logit, difference, structural != null ?
structural : declared,
model.getDeferThreshold());
}
+ private static long replacements(String s) {
+ return s.chars().filter(c -> c == '�').count();
+ }
+
/** The champion, its margin, and the anchor it yields to on a thin
margin. */
public static final class Decision {
private final Charset champion;
diff --git
a/tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin
b/tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin
index 2fa20caa93..1a90b56965 100644
Binary files
a/tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin
and
b/tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin
differ
diff --git
a/tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin.provenance
b/tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin.provenance
index f9a8f04d05..5f9bd0132c 100644
---
a/tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin.provenance
+++
b/tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin.provenance
@@ -1,48 +1,48 @@
-model_sha256 c84ffc3d45b76e7ff3d3993e6b695abd1e54a4f42d5e193351854680b7503cdd
-trained_at 2026-11-05T14:55:39.297407548Z
+model_sha256 0f4d6535d1cae74b74892adc43298732e399393cde8d0b2235bf729cae7388a6
+trained_at 2026-11-08T01:29:35.867917407Z
source_corpus junk-clean-20261103
source_manifest_sha256
cad620363ea1a4eceac5775197b62068795531930a759493b644caafe930ca4a
source_content_sha256
32c371b35d7b00291e7cf1d1f7c305cf1db56908c76d2f42a1ac80f8d0c530d2
-training_data 20261105-094302-7e7b9d60ab19/built
+training_data 20261107-201733-5bc3f3034004/built
training_data_manifest_sha256
176b567837c33ca0dd16b6f8879539b0e54193be259923616e946e0b5952a4b7
-git_commit 7e7b9d60ab19ffb85a7321585a1bf2b87891f1ac-dirty-f2b8d3686315
-args --corpus data/junk-clean-20261103 --built
20261105-094302-7e7b9d60ab19/built --gold charsets/gold-v112-train-halves.tsv
--sample cchtml-train-unanimous/sample-gen-20261103.tsv --sample-docs
cchtml-train-unanimous/docs --output junkdetect/junkdetect.bin --git-commit
7e7b9d60ab19ffb85a7321585a1bf2b87891f1ac-dirty-f2b8d3686315
+git_commit 5bc3f30340046c78c11a1bcb6eca75992a8ca695
+args --corpus data/junk-clean-20261103 --built
20261107-201733-5bc3f3034004/built --gold charsets/gold-v113-train-halves.tsv
--sample cchtml-train-unanimous/sample-gen-20261103.tsv --sample-docs
cchtml-train-unanimous/docs --output junkdetect/junkdetect.bin --git-commit
5bc3f30340046c78c11a1bcb6eca75992a8ca695
java_version 25.0.4.1
format JUNKHNG1
-gold charsets/gold-v112-train-halves.tsv
-gold_sha256 5bad69f8050f6b0596b3c0a3fac40cb6d9884d68520960e2a24f8e10b2b5a213
+gold charsets/gold-v113-train-halves.tsv
+gold_sha256 88c28a193f2b405ff668cec233e65dca9d1755b1e18af62d3fb4385246fe08c8
real_decision_sample cchtml-train-unanimous/sample-gen-20261103.tsv
real_decision_sample_sha256
6366919ca3ab598046b18cb3e6d1dc4b050ee14614c80ccbc47d4fa59a7ef1e1
source_corpus_dir data/junk-clean-20261103
wiki_units 96667
wiki_units_upper_case_added 6144
wiki_units_without_language 0.0000
-pairs_wiki 365788
-gold_pages_even 13268
-gold_pages_odd 13225
+pairs_wiki 548682
+gold_pages_even 13293
+gold_pages_odd 13239
chain
BOMDetector,MetadataCharsetDetector,MojibusterEncodingDetector,HtmlEncodingDetector
real_decision_pages 66022
real_decision_variants 280708
real_decision_errors 0
-pairs_real_decision_fit 159485
-pairs_real_decision_select 19219
-pairs_gold_decoders_odd 253834
-pairs_gold_chain_odd 7481
-defer_sweep_pages 13268
-defer_sweep_decisive_pages 106
-defer_sweep_curve 0.25:+10/-0 0.50:+12/-0 0.75:+18/-0 1.00:+21/-0
1.25:+29/-1 1.50:+33/-2 1.75:+34/-2 2.00:+36/-2 2.25:+37/-4 2.50:+37/-6
2.75:+39/-8 3.00:+39/-8 3.25:+41/-9 3.50:+42/-10 3.75:+44/-11 4.00:+44/-12
4.25:+44/-15 4.50:+44/-19 4.75:+44/-19 5.00:+47/-19 5.25:+48/-22 5.50:+48/-23
5.75:+48/-26 6.00:+48/-30 6.25:+48/-37 6.50:+48/-39 6.75:+48/-39 7.00:+48/-42
7.25:+48/-44 7.50:+48/-45 7.75:+48/-46 8.00:+48/-46 8.25:+48/-48 8.50:+48/-50
8.75:+48/-51 9.00:+49/-52 9.25:+49/-52 9.50:+ [...]
-defer_sweep_at_threshold +36/-2
-pairs_gold_decoders_even 253773
-pairs_gold_chain_even 7230
+pairs_real_decision_fit 7540
+pairs_real_decision_select 973
+pairs_gold_decoders_odd 254125
+pairs_gold_chain_odd 825
+defer_sweep_pages 13293
+defer_sweep_decisive_pages 96
+defer_sweep_curve 0.25:+1/-0 0.50:+5/-0 0.75:+8/-0 1.00:+11/-1
1.25:+12/-2 1.50:+15/-2 1.75:+16/-2 2.00:+20/-2 2.25:+21/-5 2.50:+22/-5
2.75:+22/-7 3.00:+25/-7 3.25:+26/-8 3.50:+26/-9 3.75:+29/-10 4.00:+29/-14
4.25:+30/-14 4.50:+32/-15 4.75:+33/-15 5.00:+33/-21 5.25:+34/-22 5.50:+34/-22
5.75:+34/-25 6.00:+34/-28 6.25:+34/-35 6.50:+34/-38 6.75:+34/-41 7.00:+34/-45
7.25:+34/-46 7.50:+34/-48 7.75:+34/-49 8.00:+34/-51 8.25:+35/-55 8.50:+35/-55
8.75:+35/-57 9.00:+35/-57 9.25:+35/-58 9.50:+35/- [...]
+defer_sweep_at_threshold +20/-2
+pairs_gold_decoders_even 254294
+pairs_gold_chain_even 761
point_clean_wiki 102857
-point_clean_web 12682
+point_clean_web 12692
point_damaged_windows 1698
point_damaged_clean_rest 196
point_damaged_stale_pages 1
-point_examples 246342
+point_examples 246362
trained_scripts [LATIN, GREEK, CYRILLIC, ARMENIAN, HEBREW, ARABIC,
DEVANAGARI, BENGALI, GURMUKHI, GUJARATI, ORIYA, TAMIL, TELUGU, KANNADA,
MALAYALAM, SINHALA, THAI, LAO, TIBETAN, MYANMAR, GEORGIAN, HANGUL, ETHIOPIC,
KHMER, HIRAGANA, KATAKANA, HAN, NKO, OL_CHIKI]
deferral_threshold 2.0000
-train_seconds 666
+train_seconds 651
config.HashedTrainingConfig.LOG2_SIZE 21
config.HashedTrainingConfig.MAX_N 3
config.HashedTrainingConfig.CLIP_CHARS 6000
diff --git
a/tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/HashedArbitrationTest.java
b/tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/HashedArbitrationTest.java
index 0635a56596..b856b420da 100644
---
a/tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/HashedArbitrationTest.java
+++
b/tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/HashedArbitrationTest.java
@@ -82,6 +82,53 @@ public class HashedArbitrationTest {
assertEquals(s.pick(model.getDeferThreshold()), s.pick());
}
+ /**
+ * From a CC page declared iso-8859-3 but written in Latin-1 with UTF-8
fragments: 0xC3 is undefined
+ * in ISO-8859-3, so its decode has U+FFFD the windows-1252 decode lacks.
+ */
+ @Test
+ public void declarationWithReplacementCharactersIsNotAnAnchor() throws
Exception {
+ Charset iso3 = Charset.forName("ISO-8859-3");
+ byte[] latin1 = ("Correo Electrónico. ¡Por favor, consulte la
Composición Municipal y los"
+ + " Trámites Oficiales. ").getBytes(L1);
+ byte[] utf8 = "Información del municipio, días de
atención.".getBytes(UTF8);
+ byte[] page = new byte[latin1.length + utf8.length];
+ System.arraycopy(latin1, 0, page, 0, latin1.length);
+ System.arraycopy(utf8, 0, page, latin1.length, utf8.length);
+ Map<Charset, String> texts = new LinkedHashMap<>();
+ texts.put(W1252, new String(page, W1252));
+ texts.put(iso3, new String(page, iso3));
+ assertTrue(texts.get(iso3).indexOf('�') >= 0);
+
+ HashedJunkModel bundled = JunkDetector.loadFromClasspath().getModel();
+ HashedArbitration.Decision d = HashedArbitration.decide(bundled,
texts, null, iso3);
+ assertEquals(W1252, d.getChampion());
+ assertNull(d.getAnchor());
+ assertEquals(W1252, d.pick(Double.MAX_VALUE));
+
+ // a champion with as many replacement characters leaves the
declaration as the anchor
+ Map<Charset, String> same = new LinkedHashMap<>();
+ same.put(W1252, "caf� und mehr");
+ same.put(iso3, "ca�s und mehr");
+ HashedArbitration.Decision b = HashedArbitration.decide(model, same,
null, iso3);
+ if (!b.getChampion().equals(iso3)) {
+ assertEquals(iso3, b.getAnchor());
+ }
+ // one with fewer does not: 0x81 is undefined in windows-1252, a C1
control in ISO-8859-3
+ byte[] spanish = ("¡Bienvenidos al Ayuntamiento! ¿Necesita ayuda?
").getBytes(L1);
+ byte[] stray = new byte[spanish.length + page.length + 1];
+ System.arraycopy(spanish, 0, stray, 0, spanish.length);
+ System.arraycopy(page, 0, stray, spanish.length, page.length);
+ stray[stray.length - 1] = (byte) 0x81;
+ Map<Charset, String> fewer = new LinkedHashMap<>();
+ fewer.put(W1252, new String(stray, W1252));
+ fewer.put(iso3, new String(stray, iso3));
+ assertTrue(fewer.get(W1252).indexOf('�') >= 0);
+ HashedArbitration.Decision f = HashedArbitration.decide(bundled,
fewer, null, iso3);
+ assertEquals(W1252, f.getChampion());
+ assertNull(f.getAnchor());
+ }
+
@Test
public void identicalTextsHaveNoMarginAndNeverDefer() {
Map<Charset, String> texts = new LinkedHashMap<>();
