This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new e2fd85f94a TIKA-5029 (#3533)
e2fd85f94a is described below
commit e2fd85f94a01b22d3e45bb078ee6a502e9f3aaed
Author: Tim Allison <[email protected]>
AuthorDate: Fri Nov 6 13:18:48 2026 -0500
TIKA-5029 (#3533)
* TIKA-5029: fail the mime-types load on a duplicate definition; merge
tika-mimetypes.xml duplicates
* TIKA-5029: detect WOFF, SquashFS, ASE, PalmDOC and binary OpenPGP
signatures by content
* TIKA-5029: detect FEC filings, gettext catalogs, Garmin GDB, MPQ/Warcraft
III, VCDIFF, Xdelta and PSP by content
* TIKA-5029: detect HTML that opens with a fragment, and ASX without an XML
declaration
* TIKA-5029: fail the build on mime-type rules that can never match; fix
SubRip CRLF
* TIKA-5029: detect Elasto Mania, PGP keys, StuffIt, MPEG-TS, Symbian 9,
FLC/FLI, MS Reader, lrzip, Glulx, S3M and APE
* TIKA-5029: detect niche formats and DER CRLs, certificates and time-stamp
replies by content
* TIKA-5029: detect glTF, Blender, FBX, Jw_cad, FastCAD, Source maps,
Unreal UZ, SoundFont, zsync and WFDB by content
* TIKA-5029: detect printer jobs, MO:DCA, Windows dumps, StarView metafiles
and four crawl clusters by content
* TIKA-5029: detect gcov, Python bytecode and pickles, Lucene indexes, git
packs, ACPI tables, TZif, mozLz4 and JPEG XR by content
* TIKA-5029: let a .xml name pick xml over the html-fragment magic
* TIKA-5029: detect Python 3.14+ bytecode and Blender 5 files; use
established MIME names; drop shared globs
* TIKA-5029: detect new-format OpenPGP packets, v1 CRLs, gettext revision 1
and SDCH VCDIFF; cite evidence for crawl-derived rules
* TIKA-5029: tighten Python 3 bytecode magic; gate new regex rules behind
literal prefixes
* TIKA-5029: choose the PJL job's language by literal ENTER LANGUAGE
matches over 8K instead of a 1K regex
* TIKA-5029: more from recent cc detect NEXRAD Level III, UCSC
bigBed/bigWig/2bit, BAM/BAI, OSM PBF, LIGO frames, qcow, ImageDisk, MAME CHD,
devicetree/FIT, U-Boot, UBI, Amiga hunk executables, DDS, YM, VST presets and
older Jw_cad
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
CHANGES.txt | 80 ++
.../java/org/apache/tika/mime/MimeTypesReader.java | 14 +
.../org/apache/tika/mime/tika-mimetypes.xml | 1085 ++++++++++++++++++--
.../org/apache/tika/mime/MimeDetectionTest.java | 210 ++++
.../org/apache/tika/mime/MimeTypesLintTest.java | 167 +++
.../org/apache/tika/mime/MimeTypesReaderTest.java | 35 +
.../apache/tika/mime/NicheFormatDetectionTest.java | 364 +++++++
.../org/apache/tika/parser/ParsingReaderTest.java | 2 +-
.../tika-parser-cad-module/pom.xml | 9 +
9 files changed, 1897 insertions(+), 69 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 70246f11e1..866d6b1430 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -22,6 +22,14 @@ Release 4.3.0 - ???
the forks never saw such a fetcher, so every fetch through it failed. Use
file or
ignite (TIKA-5001).
+ * A mime-types XML file that defines the same type twice, directly or
through an
+ alias, fails to load with "Duplicate definition of ..."; the second
definition's
+ magic and globs were silently added to the first, so tightening a rule in
one copy
+ had no effect. A custom-mimetypes.xml that repeats a type within itself
now stops
+ Tika from loading its types; merge the repeated blocks into one. A custom
file may
+ still add globs or magic to a type defined in tika-mimetypes.xml. The
eleven
+ duplicates in tika-mimetypes.xml are merged; detection is unchanged
(TIKA-5029).
+
* 3MF packages are detected as model/3mf, their IANA type, instead of
application/vnd.ms-package.3dmanufacturing-3dmodel+xml, which is the
content type of
the model part inside the package. The old name is an alias, so it still
resolves as
@@ -81,6 +89,78 @@ Release 4.3.0 - ???
EXTRACTION AND FORMATS
+ * WOFF and WOFF2 fonts (font/woff, font/woff2), SquashFS images
+ (application/vnd.squashfs), Adobe Swatch Exchange files, PalmDOC e-books
and binary
+ OpenPGP signatures (application/pgp-signature) are detected by content;
they were
+ application/octet-stream. Together they are about a sixth of the Common
Crawl files
+ Tika could not identify. MXF and Java serialization streams are detected
by content
+ too (they had globs only), EOT fonts of version 0x00010000 and 0x00020001
are
+ detected (only 0x00020002 was), and torrent files whose first key is not
"announce"
+ are detected (TIKA-5029).
+
+ * US FEC electronic filings (application/x-fec, a kind of text/plain, so
their text is
+ extracted), gettext message catalogs, Garmin MapSource databases,
Blizzard MPQ
+ archives and Warcraft III maps and replays, VCDIFF and Xdelta patches and
Paint Shop
+ Pro images are detected by content; they were application/octet-stream.
SID tunes
+ in RSID format are detected, and PSID needs a valid version after the
magic
+ (TIKA-5029).
+
+ * HTML that opens with a fragment instead of <html> (a <meta>, <script>,
<p>, <br>,
+ <table> or similar tag, as in pages that start with PHP warnings) is
detected as
+ text/html, and Windows Media playlists without an XML declaration are
detected as
+ application/x-ms-asx; they were application/octet-stream or text/plain. A
PDF or
+ XHTML document with such a fragment in front keeps its type (TIKA-5029).
+
+ * A new test fails the build on tika-mimetypes.xml rules that load but can
never
+ match as written (TIKA-5029).
+
+ * Elasto Mania levels, binary OpenPGP keys, StuffIt (SIT!) archives, MPEG
transport
+ streams (video/mp2t, now also *.m2ts and *.m2t), Symbian 9 installers,
FLC and FLI
+ animations, Microsoft Reader e-books, lrzip archives, Glulx games, Scream
Tracker 3
+ modules and Monkey's Audio files are detected by content, and so are MARC
records
+ with a blank bibliographic level; they were application/octet-stream
(TIKA-5029).
+
+ * NoteWorthy Composer songs, Finale (Enigma binary) scores, ReCycle REX2
loops, i-mode
+ MFi melodies, TrackMania/ManiaPlanet GameBox files, PlayStation 3
packages and themes,
+ Trainz content packages, ZX Spectrum RZX recordings, Quake and Half-Life
sprites,
+ NuFX (ShrinkIt) and Amiga LZX archives, MikroTik RouterOS packages and
ClamAV
+ signature databases and updates, and Casio FX-Interface files are
detected by
+ content; they were application/octet-stream (TIKA-5029).
+
+ * DER certificate revocation lists (application/pkix-crl), DER X.509
certificates
+ signed with any algorithm (only some RSA, DSA and ECDSA ones were
matched, so SM2
+ and Ed25519 certificates were missed) and RFC 3161 time-stamp replies are
detected
+ by content; they were application/octet-stream (TIKA-5029).
+
+ * glTF binary models, Blender scenes, Autodesk FBX (binary), Jw_cad and
FastCAD drawings,
+ Valve Source maps, Unreal Tournament compressed packages, SoundFont 2
banks, zsync
+ control files and PhysioNet WFDB annotations are detected by content;
they were
+ application/octet-stream in a recent Common Crawl sample (TIKA-5029).
+
+ * NEXRAD Level III radar products, UCSC bigBed, bigWig and 2bit files,
uncompressed BAM
+ and BAM indexes, OpenStreetMap PBF, LIGO/Virgo frame files, QEMU qcow
disks, ImageDisk
+ floppy images, MAME CHD, devicetree blobs and FIT images, U-Boot legacy
images, UBI
+ images, AmigaOS executables, DirectDraw Surface images
(image/vnd.ms-dds), YM chiptunes,
+ VST presets and older Jw_cad drawings are detected by content; they were
+ application/octet-stream in a recent Common Crawl sample (TIKA-5029).
+
+ * HP printer jobs are detected by content: PJL jobs by the language they
enter (PCL,
+ PCL XL or PostScript), and raw PCL5 and PCL XL streams. So are IBM MO:DCA
(AFP)
+ documents, Windows minidumps and kernel dumps, StarView metafiles, Lotus
1-2-3 files
+ with format version 0x1005, Promethean ActivInspire flipcharts, Casio
calculator
+ (g1m family) files, Haihaisoft DRM-X protected files and Navitel maps;
they were
+ application/octet-stream (TIKA-5029).
+
+ * gcov coverage data and notes, Python bytecode and pickles, Apache Lucene
index files,
+ git packfiles, ACPI tables, TZif time zone files (application/tzif),
Firefox mozLz4
+ files and JPEG XR images (image/jxr) are detected by content; they were
+ application/octet-stream (TIKA-5029).
+
+ * More variants are detected by content: OpenPGP keys and signatures in
new-format
+ packets (as written for RFC 9580 v6 keys), v1 DER certificate revocation
lists (no
+ version field), gettext catalogs with minor revision 1 and VCDIFF deltas
in
+ open-vcdiff's SDCH format; they were application/octet-stream (TIKA-5029).
+
* EpubParser resolves manifest hrefs as URI references relative to the OPF:
a '+' is
no longer read as a space, and '../' is followed, so such chapters and
images are no
longer skipped. encryption.xml URIs are resolved the same way. The new
diff --git a/tika-core/src/main/java/org/apache/tika/mime/MimeTypesReader.java
b/tika-core/src/main/java/org/apache/tika/mime/MimeTypesReader.java
index a350939f32..7bc318ae8c 100644
--- a/tika-core/src/main/java/org/apache/tika/mime/MimeTypesReader.java
+++ b/tika-core/src/main/java/org/apache/tika/mime/MimeTypesReader.java
@@ -22,7 +22,9 @@ import java.net.URI;
import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.Collections;
+import java.util.HashSet;
import java.util.List;
+import java.util.Set;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantReadWriteLock;
@@ -139,6 +141,9 @@ public class MimeTypesReader extends DefaultHandler
implements MimeTypesReaderMe
protected StringBuilder characters = null;
private ClauseRecord current = new ClauseRecord(null);
+ // per document: a later file (custom-mimetypes.xml) may extend a type,
one file may not repeat it
+ private final Set<MediaType> definedInDocument = new HashSet<>();
+
protected MimeTypesReader(MimeTypes types) {
this.types = types;
}
@@ -253,6 +258,11 @@ public class MimeTypesReader extends DefaultHandler
implements MimeTypesReaderMe
return new
InputSource(UnsynchronizedByteArrayInputStream.nullInputStream());
}
+ @Override
+ public void startDocument() {
+ definedInDocument.clear();
+ }
+
@Override
public void startElement(String uri, String localName, String qName,
Attributes attributes)
throws SAXException {
@@ -264,6 +274,10 @@ public class MimeTypesReader extends DefaultHandler
implements MimeTypesReaderMe
try {
type = types.forName(name);
type.setInterpreted(interpreted);
+ if (!definedInDocument.add(type.getType())) {
+ throw new MimeTypeException(
+ "Duplicate definition of " + type.getType() +
" (as " + name + ")");
+ }
} catch (MimeTypeException e) {
handleMimeError(name, e, qName, attributes);
}
diff --git
a/tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml
b/tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml
index 0d3d07e5ec..36badd6abc 100644
--- a/tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml
+++ b/tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml
@@ -383,6 +383,10 @@
<mime-type type="application/x-tika-unix-dump"/>
<mime-type type="application/java-serialized-object">
+ <magic priority="50">
+ <!-- STREAM_MAGIC, STREAM_VERSION 5, then a TC_* (0x70-0x7E) -->
+ <match value="0xACED000570" mask="0xFFFFFFFFF0" type="string"
offset="0"/>
+ </magic>
<glob pattern="*.ser"/>
</mime-type>
@@ -544,6 +548,14 @@
</magic>
</mime-type>
+ <mime-type type="application/x-mame-chd">
+ <_comment>MAME compressed hunks of data</_comment>
+ <magic priority="50">
+ <match value="MComprHD" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.chd"/>
+ </mime-type>
+
<mime-type type="application/x-java-jnilib">
<_comment>Java Native Library for OSX</_comment>
<glob pattern="*.jnilib"/>
@@ -594,7 +606,8 @@
<match value="[0-9]{5,5}" type="regex" offset="0">
<match value="45" type="string" offset="20">
<!-- bibliographic -->
- <match value="[acdnp][acdefgijkmoprt][abcdims]" type="regex"
offset="5"/>
+ <!-- leader/07 is often left blank -->
+ <match value="[acdnp][acdefgijkmoprt][abcdims ]" type="regex"
offset="5"/>
<!-- authority-->
<match value="[acdnosx]z" type="regex" offset="5"/>
<!-- holdings -->
@@ -678,11 +691,6 @@
<mime-type type="application/mediaservercontrol+xml">
<glob pattern="*.mscml"/>
</mime-type>
- <mime-type type="application/vnd.mif">
- <_comment>Adobe MIF File</_comment>
- <glob pattern="*.mif"/>
- <sub-class-of type="text/plain"/>
- </mime-type>
<mime-type type="application/mikey"/>
<mime-type type="application/moss-keys"/>
<mime-type type="application/moss-signature"/>
@@ -748,6 +756,10 @@
</mime-type>
<mime-type type="application/mxf">
+ <magic priority="50">
+ <!-- SMPTE 377 header partition pack key -->
+ <match value="0x060E2B34020501010D0102010102" type="string" offset="0"/>
+ </magic>
<glob pattern="*.mxf"/>
</mime-type>
<mime-type type="application/nasdata"/>
@@ -899,9 +911,73 @@
<glob pattern="*.gpg"/>
</mime-type>
- <mime-type type="application/pgp-keys"/>
+ <mime-type type="application/pgp-keys">
+ <magic priority="50">
+ <!-- binary old-format v4 key packet (RFC 4880 5.5): public (tag 6) or
secret (tag 5)
+ key with a 1- or 2-byte length, version, 4-byte time, then
RSA/DSA/Elgamal/ECDH/ECDSA/EdDSA -->
+ <match value="0x94" type="string" offset="0">
+ <match value="0x04" type="string" offset="2">
+ <match value="0x01" type="string" offset="7"/>
+ <match value="0x11" mask="0xFE" type="string" offset="7"/>
+ <match value="0x12" mask="0xFE" type="string" offset="7"/>
+ <match value="0x16" type="string" offset="7"/>
+ </match>
+ </match>
+ <match value="0x98" type="string" offset="0">
+ <match value="0x04" type="string" offset="2">
+ <match value="0x01" type="string" offset="7"/>
+ <match value="0x11" mask="0xFE" type="string" offset="7"/>
+ <match value="0x12" mask="0xFE" type="string" offset="7"/>
+ <match value="0x16" type="string" offset="7"/>
+ </match>
+ </match>
+ <match value="0x95" type="string" offset="0">
+ <match value="0x04" type="string" offset="3">
+ <match value="0x01" type="string" offset="8"/>
+ <match value="0x11" mask="0xFE" type="string" offset="8"/>
+ <match value="0x12" mask="0xFE" type="string" offset="8"/>
+ <match value="0x16" type="string" offset="8"/>
+ </match>
+ </match>
+ <match value="0x99" type="string" offset="0">
+ <match value="0x04" type="string" offset="3">
+ <match value="0x01" type="string" offset="8"/>
+ <match value="0x11" mask="0xFE" type="string" offset="8"/>
+ <match value="0x12" mask="0xFE" type="string" offset="8"/>
+ <match value="0x16" type="string" offset="8"/>
+ </match>
+ </match>
+ <!-- new-format secret (0xC5) or public (0xC6) key packet (RFC 9580 4.2,
5.5.2): 1- or 2-byte
+ length, v4 or v6, 4-byte time, then a known public-key algorithm -->
+ <match value="0xC4" mask="0xFC" type="string" offset="0">
+ <match
value="(?s)[\\xc5\\xc6](?:[\\x00-\\xbf]|[\\xc0-\\xdf].)[\\x04\\x06]....[\\x01\\x10-\\x13\\x16\\x19-\\x1c]"
type="regex" offset="0"/>
+ </match>
+ </magic>
+ </mime-type>
<mime-type type="application/pgp-signature">
+ <magic priority="50">
+ <!-- binary old-format signature packet (RFC 4880 5.2): tag 2 with a 1-
or 2-byte
+ length, then v4 + binary/text sig type + RSA/DSA/ECDSA/EdDSA, or v3
+ hashed length 5 -->
+ <match value="0x88" type="string" offset="0">
+ <match value="0x040001" mask="0xFFFEFF" type="string" offset="2"/>
+ <match value="0x040011" mask="0xFFFEFF" type="string" offset="2"/>
+ <match value="0x040013" mask="0xFFFEFF" type="string" offset="2"/>
+ <match value="0x040016" mask="0xFFFEFF" type="string" offset="2"/>
+ <match value="0x030500" mask="0xFFFFFE" type="string" offset="2"/>
+ </match>
+ <match value="0x89" type="string" offset="0">
+ <match value="0x040001" mask="0xFFFEFF" type="string" offset="3"/>
+ <match value="0x040011" mask="0xFFFEFF" type="string" offset="3"/>
+ <match value="0x040013" mask="0xFFFEFF" type="string" offset="3"/>
+ <match value="0x040016" mask="0xFFFEFF" type="string" offset="3"/>
+ <match value="0x030500" mask="0xFFFFFE" type="string" offset="3"/>
+ </match>
+ <!-- new-format signature packet (0xC2): v4 or v6, binary/text sig type,
known algorithm -->
+ <match value="0xC2" type="string" offset="0">
+ <match
value="(?s)\\xc2(?:[\\x00-\\xbf]|[\\xc0-\\xdf].)[\\x04\\x06][\\x00\\x01][\\x01\\x11\\x13\\x16\\x1b\\x1c]"
type="regex" offset="0"/>
+ </match>
+ </magic>
<glob pattern="*.asc"/>
<glob pattern="*.sig"/>
</mime-type>
@@ -965,6 +1041,20 @@
</mime-type>
<mime-type type="application/pkix-crl">
<glob pattern="*.crl"/>
+ <magic priority="50">
+ <!-- DER v2 CertificateList: SEQUENCE { SEQUENCE { INTEGER 1,
AlgorithmIdentifier { OID ...
+ any signature algorithm; long-form outer length, short or long
inner.
+ A v1 certificate with serial 1 starts the same way; its validity
SEQUENCE { Time, Time }
+ (never in a CRL) rules it out -->
+ <match value="0x3080" mask="0xFFFC" type="string" offset="0">
+ <match
value="(?s)\\x30(?:\\x81.|\\x82..|\\x83...)\\x30(?:[\\x00-\\x7f]|\\x81.|\\x82..|\\x83...)\\x02\\x01\\x01\\x30[\\x00-\\x7f]\\x06(?!.{0,512}?\\x30(?:\\x1e\\x17\\x0d|\\x20(?:\\x17\\x0d|\\x18\\x0f)|\\x22\\x18\\x0f))"
type="regex" offset="0"/>
+ </match>
+ <!-- DER v1 CertificateList, no version: SEQUENCE { SEQUENCE {
AlgorithmIdentifier, issuer Name
+ (SEQUENCE { SET ...), then thisUpdate. A certificate's TBS opens
with [0] or an INTEGER -->
+ <match value="0x3080" mask="0xFFFC" type="string" offset="0">
+ <match
value="(?s)\\x30(?:\\x81.|\\x82..|\\x83...)\\x30(?:[\\x00-\\x7f]|\\x81.|\\x82..|\\x83...)\\x30[\\x00-\\x7f]\\x06[\\x03-\\x0f].{3,20}?\\x30(?:[\\x00-\\x7f]|\\x81.|\\x82..)\\x31.{0,1024}?(?:\\x17\\x0d[0-9]{12}Z|\\x18\\x0f[0-9]{14}Z)"
type="regex" offset="0"/>
+ </match>
+ </magic>
</mime-type>
<mime-type type="application/pkix-pkipath">
<glob pattern="*.pkipath"/>
@@ -979,6 +1069,13 @@
<mime-type type="application/postscript">
<_comment>PostScript</_comment>
+ <magic priority="60">
+ <!-- PJL job that switches to PostScript; driver headers put the command
up to ~2K in -->
+ <match value="\x1b%-12345X" type="stringignorecase" offset="0">
+ <match value="ENTER LANGUAGE=POSTSCRIPT" type="stringignorecase"
offset="0:8192"/>
+ <match value="ENTER LANGUAGE = POSTSCRIPT" type="stringignorecase"
offset="0:8192"/>
+ </match>
+ </magic>
<magic priority="50">
<match value="%!" type="string" offset="0" />
<match value="\004%!" type="string" offset="0" />
@@ -1186,8 +1283,26 @@
<glob pattern="*.ssml"/>
</mime-type>
<mime-type type="application/timestamp-query"/>
- <mime-type type="application/timestamp-reply"/>
+ <mime-type type="application/timestamp-reply">
+ <magic priority="50">
+ <!-- RFC 3161 TimeStampResp: SEQUENCE { PKIStatusInfo { INTEGER status,
... },
+ then the token, a CMS signedData ContentInfo -->
+ <match value="0x3080" mask="0xFFFC" type="string" offset="0">
+ <match
value="(?s)\\x30(?:\\x81.|\\x82..|\\x83...)\\x30[\\x03-\\x7f]\\x02\\x01[\\x00-\\x05].{0,160}?\\x06\\x09\\x2a\\x86\\x48\\x86\\xf7\\x0d\\x01\\x07\\x02"
type="regex" offset="0"/>
+ </match>
+ </magic>
+ <glob pattern="*.tsr"/>
+ </mime-type>
<mime-type type="application/tve-trigger"/>
+ <mime-type type="application/tzif">
+ <_comment>Time Zone Information Format (RFC 8536)</_comment>
+ <magic priority="50">
+ <match value="TZif\x00" type="string" offset="0"/>
+ <match value="TZif2" type="string" offset="0"/>
+ <match value="TZif3" type="string" offset="0"/>
+ <match value="TZif4" type="string" offset="0"/>
+ </magic>
+ </mime-type>
<mime-type type="application/ulpfec"/>
<mime-type type="application/vemmi"/>
<mime-type type="application/vividence.scriptfile"/>
@@ -1413,6 +1528,20 @@
<_comment>frictionless data package zip package</_comment>
<sub-class-of type="application/zip"/>
</mime-type>
+
+ <mime-type type="application/x-vst-preset">
+ <_comment>VST plug-in preset or bank</_comment>
+ <magic priority="50">
+ <match value="CcnK" type="string" offset="0">
+ <match value="FxCk" type="string" offset="8"/>
+ <match value="FPCh" type="string" offset="8"/>
+ <match value="FxBk" type="string" offset="8"/>
+ <match value="FBCh" type="string" offset="8"/>
+ </match>
+ </magic>
+ <glob pattern="*.fxp"/>
+ <glob pattern="*.fxb"/>
+ </mime-type>
<mime-type type="application/x-ipynb+json">
<_comment>Jupyter Notebook</_comment>
<tika:link>https://nbformat.readthedocs.io/en/latest/format_description.html</tika:link>
@@ -1724,10 +1853,36 @@
<glob pattern="*.jlt"/>
</mime-type>
<mime-type type="application/vnd.hp-pcl">
+ <magic priority="55">
+ <!-- PJL job (UEL, then @PJL commands) that switches to PCL; below PCL
XL, whose command
+ this prefix also matches -->
+ <match value="\x1b%-12345X" type="stringignorecase" offset="0">
+ <match value="ENTER LANGUAGE=PCL" type="stringignorecase"
offset="0:8192"/>
+ <match value="ENTER LANGUAGE = PCL" type="stringignorecase"
offset="0:8192"/>
+ </match>
+ </magic>
+ <magic priority="50">
+ <!-- UEL straight into PCL, or a raw PCL5 stream: reset, then a
parameterized command -->
+ <match value="\x1b%-12345X\x1bE" type="string" offset="0"/>
+ <match value="\x1bE\x1b" type="string" offset="0">
+ <match value="\x1bE\x1b[&*(%]" type="regex" offset="0"/>
+ </match>
+ </magic>
<glob pattern="*.pcl"/>
</mime-type>
<mime-type type="application/vnd.hp-pclxl">
+ <magic priority="60">
+ <match value="\x1b%-12345X" type="stringignorecase" offset="0">
+ <match value="ENTER LANGUAGE=PCLXL" type="stringignorecase"
offset="0:8192"/>
+ <match value="ENTER LANGUAGE = PCLXL" type="stringignorecase"
offset="0:8192"/>
+ </match>
+ </magic>
+ <magic priority="50">
+ <!-- stream header, e.g. ") HP-PCL XL;3;0;" -->
+ <match value=") HP-PCL XL;" type="string" offset="0"/>
+ </magic>
<glob pattern="*.pclxl"/>
+ <glob pattern="*.pxl"/>
</mime-type>
<mime-type type="application/vnd.httphone"/>
<mime-type type="application/vnd.hydrostatix.sof-data">
@@ -1742,6 +1897,12 @@
<glob pattern="*.mpy"/>
</mime-type>
<mime-type type="application/vnd.ibm.modcap">
+ <magic priority="50">
+ <!-- MO:DCA structured field: 5A, 2-byte length, D3 class, A8 = Begin
(document, resource group, ...) -->
+ <match value="0x5A" type="string" offset="0">
+ <match value="0xD3A8" type="string" offset="3"/>
+ </match>
+ </magic>
<glob pattern="*.afp"/>
<glob pattern="*.listafp"/>
<glob pattern="*.list3820"/>
@@ -1960,6 +2121,9 @@
<magic priority="50">
<match value="0x00001A0003100400" type="string" offset="0">
</match>
+ <!-- 0x1005: not in PRONOM; seen in .123 attachments to LibreOffice bugs
53713, 54655, 55066
+ and 114000 and OpenOffice bugs 79309 and 122841 -->
+ <match value="0x00001A0005100400" type="string" offset="0"/>
</magic>
<glob pattern="*.123"/>
</mime-type>
@@ -2023,6 +2187,7 @@
<_comment>FrameMaker Interchange Format</_comment>
<alias type="application/x-mif"/>
<alias type="application/x-frame"/>
+ <sub-class-of type="text/plain"/>
<magic priority="50">
<match value="<MakerFile" type="string" offset="0" />
<match value="<MIFFile" type="string" offset="0" />
@@ -2213,6 +2378,13 @@
<mime-type type="application/vnd.ms-fontobject">
<magic priority="50">
+ <!-- version 0x00010000, 0x00020001 or 0x00020002, then the "LP" magic
-->
+ <match value="\x00\x00\x01\x00" type="string" offset="8">
+ <match value="\x4C\x50" type="string" offset="34"/>
+ </match>
+ <match value="\x01\x00\x02\x00" type="string" offset="8">
+ <match value="\x4C\x50" type="string" offset="34"/>
+ </match>
<match value="\x02\x00\x02\x00" type="string" offset="8">
<match value="\x4C\x50" type="string" offset="34"/>
</match>
@@ -2824,6 +2996,15 @@
<glob pattern="*.pqa"/>
<glob pattern="*.oprc"/>
</mime-type>
+ <mime-type type="application/x-aportisdoc">
+ <_comment>PalmDOC (AportisDoc) e-book</_comment>
+ <alias type="application/x-palmdoc"/>
+ <sub-class-of type="application/vnd.palm"/>
+ <magic priority="60">
+ <!-- Palm database type + creator -->
+ <match value="TEXtREAd" type="string" offset="60"/>
+ </magic>
+ </mime-type>
<mime-type type="application/vnd.paos.xml"/>
<mime-type type="application/vnd.pg.format">
@@ -3030,10 +3211,6 @@
<sub-class-of type="text/plain"/>
</mime-type>
- <mime-type type="application/vnd.sun.xml.writer">
- <sub-class-of type="application/zip"/>
- <glob pattern="*.sxw"/>
- </mime-type>
<!-- can't currently find any diff in contents btwn writer
and template. Must rely on extension -->
<mime-type type="application/vnd.sun.xml.writer.template">
@@ -3058,21 +3235,12 @@
</mime-type>
<mime-type type="application/vnd.street-stream"/>
- <mime-type type="application/vnd.sun.xml.calc">
- <glob pattern="*.sxc"/>
- </mime-type>
<mime-type type="application/vnd.sun.xml.calc.template">
<glob pattern="*.stc"/>
</mime-type>
- <mime-type type="application/vnd.sun.xml.draw">
- <glob pattern="*.sxd"/>
- </mime-type>
<mime-type type="application/vnd.sun.xml.draw.template">
<glob pattern="*.std"/>
</mime-type>
- <mime-type type="application/vnd.sun.xml.impress">
- <glob pattern="*.sxi"/>
- </mime-type>
<mime-type type="application/vnd.sun.xml.impress.template">
<glob pattern="*.sti"/>
</mime-type>
@@ -3083,6 +3251,7 @@
<mime-type type="application/vnd.sun.xml.writer">
<alias type="application/x-vnd.sun.xml.writer"/>
<_comment>OpenOffice v1.0: Writer Document</_comment>
+ <sub-class-of type="application/zip"/>
<magic>
<match type="string" offset="0" value="PK">
<match type="string" offset="30"
@@ -3095,9 +3264,6 @@
<mime-type type="application/vnd.sun.xml.writer.global">
<glob pattern="*.sxg"/>
</mime-type>
- <mime-type type="application/vnd.sun.xml.writer.template">
- <glob pattern="*.stw"/>
- </mime-type>
<mime-type type="application/vnd.sun.wadl+xml"/>
<mime-type type="application/vnd.sus-calendar">
<glob pattern="*.sus"/>
@@ -3111,6 +3277,8 @@
<mime-type type="application/vnd.symbian.install">
<magic priority="50">
<match value="0x10000419" type="little32" offset="8" />
+ <!-- SIS 9.x (sisx): UID1 -->
+ <match value="0x10201A7A" type="little32" offset="0" />
</magic>
<glob pattern="*.sis"/>
<glob pattern="*.sisx"/>
@@ -3451,6 +3619,13 @@
<glob pattern="*.wspolicy"/>
</mime-type>
+ <mime-type type="image/x-svm">
+ <_comment>StarView Metafile</_comment>
+ <magic priority="50">
+ <match value="VCLMTF" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.svm"/>
+ </mime-type>
<mime-type type="image/x-tga">
<alias type="image/x-targa"/>
<_comment>Targa image data</_comment>
@@ -3532,6 +3707,17 @@
<mime-type type="application/x-ace-compressed">
<glob pattern="*.ace"/>
</mime-type>
+ <mime-type type="application/x-intel-aml">
+ <_comment>ACPI table (DSDT, SSDT, FACP, APIC, ...)</_comment>
+ <alias type="application/x-acpi-table"/>
+ <magic priority="50">
+ <!-- signature, 4-byte length, revision, checksum, then a printable
6-byte OEM ID -->
+ <match value="0x00" mask="0xF8" type="string" offset="8">
+ <match
value="(?s)(?:DSDT|SSDT|FACP|APIC|HPET|MCFG|SLIC|BERT|WAET)....[\\x00-\\x06].[\\x20-\\x7e]{6}"
type="regex" offset="0"/>
+ </match>
+ </magic>
+ <glob pattern="*.aml"/>
+ </mime-type>
<mime-type type="application/x-activemime">
<magic priority="60">
<!-- ActiveMime\u0000\u0000 -->
@@ -3569,6 +3755,31 @@
</magic>
</mime-type>
+ <mime-type type="application/x-bam">
+ <_comment>Sequence Alignment/Map, binary (BGZF-decompressed)</_comment>
+ <magic priority="50">
+ <match value="BAM\x01" type="string" offset="0"/>
+ </magic>
+ </mime-type>
+
+ <mime-type type="application/x-bam-index">
+ <_comment>BAM index</_comment>
+ <magic priority="50">
+ <match value="BAI\x01" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.bai"/>
+ </mime-type>
+
+
+ <mime-type type="application/x-adobe-swatch-exchange">
+ <acronym>ASE</acronym>
+ <_comment>Adobe Swatch Exchange</_comment>
+ <!-- no *.ase glob: Aseprite uses it too -->
+ <magic priority="50">
+ <match value="ASEF\x00\x01\x00\x00" type="string" offset="0"/>
+ </magic>
+ </mime-type>
+
<mime-type type="application/x-adobe-indesign">
<acronym>INDD</acronym>
<_comment>Adobe InDesign document</_comment>
@@ -3851,19 +4062,53 @@
<sub-class-of type="text/plain"/>
</mime-type>
+ <mime-type type="application/x-bigbed">
+ <_comment>UCSC bigBed track</_comment>
+ <magic priority="50">
+ <!-- 0x8789F2EB in either byte order, then version 1-5 -->
+ <match value="(?s)\\xeb\\xf2\\x89\\x87[\\x01-\\x05]\\x00" type="regex"
offset="0"/>
+ <match value="(?s)\\x87\\x89\\xf2\\xeb\\x00[\\x01-\\x05]" type="regex"
offset="0"/>
+ </magic>
+ <glob pattern="*.bigbed"/>
+ </mime-type>
+
+ <mime-type type="application/x-bigwig">
+ <_comment>UCSC bigWig track</_comment>
+ <magic priority="50">
+ <!-- 0x888FFC26 in either byte order, then version 1-5 -->
+ <match value="(?s)\\x26\\xfc\\x8f\\x88[\\x01-\\x05]\\x00" type="regex"
offset="0"/>
+ <match value="(?s)\\x88\\x8f\\xfc\\x26\\x00[\\x01-\\x05]" type="regex"
offset="0"/>
+ </magic>
+ <glob pattern="*.bigwig"/>
+ </mime-type>
+
+
<mime-type type="application/x-bittorrent">
<magic priority="50">
+ <!-- bencoded dict; keys are sorted, so the first is one of these -->
<match value="d8:announce" type="string" offset="0"/>
+ <match value="d13:announce-list" type="string" offset="0"/>
+ <match value="d7:comment" type="string" offset="0"/>
+ <match value="d10:created by" type="string" offset="0"/>
+ <match value="d13:creation date" type="string" offset="0"/>
+ <match value="d8:encoding" type="string" offset="0"/>
+ <match value="d4:infod" type="string" offset="0"/>
</magic>
<glob pattern="*.torrent"/>
</mime-type>
-
<mime-type type="application/x-blender">
<acronym>Blend</acronym>
<_comment>Blender scene</_comment>
<tika:link>https://www.blender.org/</tika:link>
<magic priority="50">
- <match value="BLENDER" type="string" offset="0"/>
+ <!-- then pointer size (_ 32-bit, - 64-bit), endianness, version -->
+ <match value="BLENDER" type="string" offset="0">
+ <match value="BLENDER[_-][vV][0-9]" type="regex" offset="0"/>
+ </match>
+ <!-- 5.0+: header size, pointer size, format version, endianness,
4-digit version -->
+ <match value="BLENDER" type="string" offset="0">
+ <match value="BLENDER[0-9]{2}[_-][0-9]{2}[vV][0-9]{4}" type="regex"
offset="0"/>
+ </match>
</magic>
<glob pattern="*.blend"/>
<glob pattern="*.blender"/>
@@ -3898,6 +4143,26 @@
<sub-class-of type="application/xml"/>
</mime-type>
+ <mime-type type="application/x-casio-fxi">
+ <_comment>Casio calculator data (Yellow Computing FX-Interface)</_comment>
+ <magic priority="50">
+ <!-- "FX-INTERFACE" XOR 0xE7 -->
+ <match value="0xA1BFCAAEA9B3A2B5A1A6A4A2" type="string" offset="3"/>
+ </magic>
+ <glob pattern="*.fxi"/>
+ </mime-type>
+ <mime-type type="application/x-casio-g1m">
+ <_comment>Casio calculator file (g1m family)</_comment>
+ <magic priority="50">
+ <!-- "USBPower", bitwise inverted -->
+ <match value="0xAAACBDAF90889A8D" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.g1m"/>
+ <glob pattern="*.g1r"/>
+ <glob pattern="*.g2m"/>
+ <glob pattern="*.g2r"/>
+ <glob pattern="*.g3m"/>
+ </mime-type>
<mime-type type="application/x-cdf">
<_comment>CD Audio</_comment>
<magic priority="50">
@@ -3905,6 +4170,76 @@
<glob pattern="*.cda"/>
</magic>
</mime-type>
+ <mime-type type="application/x-fec">
+ <_comment>US Federal Election Commission electronic filing</_comment>
+
<tika:link>https://www.fec.gov/help-candidates-and-committees/filing-reports/fecfile-software/</tika:link>
+ <sub-class-of type="text/plain"/>
+ <magic priority="50">
+ <!-- header record, fields separated by FS (0x1C) -->
+ <match value="HDR\x1cFEC\x1c" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.fec"/>
+ </mime-type>
+ <mime-type type="application/x-garmin-gdb">
+ <_comment>Garmin MapSource GPS database</_comment>
+ <!-- no *.gdb glob: InterBase/Firebird and others use it -->
+ <magic priority="50">
+ <match value="MsRcf\x00" type="string" offset="0"/>
+ </magic>
+ </mime-type>
+ <mime-type type="application/x-gbx">
+ <_comment>Nadeo GameBox (TrackMania, ManiaPlanet)</_comment>
+ <magic priority="50">
+ <!-- version 3-6, then format and compression flags -->
+ <match value="GBX" type="string" offset="0">
+ <match value="GBX[\x03-\x06]\x00[BT][UC][UC]" type="regex" offset="0"/>
+ </match>
+ </magic>
+ <glob pattern="*.gbx"/>
+ </mime-type>
+ <mime-type type="application/x-gcov-data">
+ <_comment>gcov coverage data</_comment>
+ <magic priority="50">
+ <!-- "gcda" in either byte order, then a 4-character version -->
+ <match value="adcg" type="string" offset="0">
+ <match value="[0-9A-Z*]{4}" type="regex" offset="4"/>
+ </match>
+ <match value="gcda" type="string" offset="0">
+ <match value="[0-9A-Z*]{4}" type="regex" offset="4"/>
+ </match>
+ </magic>
+ <glob pattern="*.gcda"/>
+ </mime-type>
+ <mime-type type="application/x-gcov-notes">
+ <_comment>gcov notes</_comment>
+ <magic priority="50">
+ <match value="oncg" type="string" offset="0">
+ <match value="[0-9A-Z*]{4}" type="regex" offset="4"/>
+ </match>
+ <match value="gcno" type="string" offset="0">
+ <match value="[0-9A-Z*]{4}" type="regex" offset="4"/>
+ </match>
+ </magic>
+ <glob pattern="*.gcno"/>
+ </mime-type>
+ <mime-type type="application/x-gettext-translation">
+ <_comment>GNU gettext message catalog</_comment>
+ <magic priority="50">
+ <!-- magic in either byte order, then major revision 0, minor 0 or 1
(system-dependent strings) -->
+ <match value="0xDE12049500000000" mask="0xFFFFFFFFFEFFFFFF"
type="string" offset="0"/>
+ <match value="0x950412DE00000000" mask="0xFFFFFFFFFFFFFFFE"
type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.mo"/>
+ <glob pattern="*.gmo"/>
+ </mime-type>
+ <mime-type type="application/x-git-pack">
+ <_comment>git packfile</_comment>
+ <alias type="application/x-git"/>
+ <magic priority="50">
+ <match value="PACK\x00\x00\x00\x02" type="string" offset="0"/>
+ <match value="PACK\x00\x00\x00\x03" type="string" offset="0"/>
+ </magic>
+ </mime-type>
<mime-type type="application/x-gtar">
<_comment>GNU tar Compressed File Archive (GNU Tape Archive)</_comment>
<magic priority="50">
@@ -3940,6 +4275,14 @@
</magic>
</mime-type>
+ <mime-type type="application/x-amiga-executable">
+ <_comment>AmigaOS hunk executable</_comment>
+ <magic priority="50">
+ <!-- HUNK_HEADER, no resident libraries, then a nonzero hunk table size
-->
+ <match value="(?s)\\x00\\x00\\x03\\xf3\\x00{7}[\\x01-\\xff]"
type="regex" offset="0"/>
+ </magic>
+ </mime-type>
+
<mime-type type="application/x-brotli">
<glob pattern="*.br" />
<glob pattern="*.brotli" />
@@ -3986,6 +4329,22 @@
</magic>
<glob pattern="*.crx"/>
</mime-type>
+ <mime-type type="application/x-clamav-database">
+ <_comment>ClamAV signature database</_comment>
+ <alias type="application/x-clamav"/>
+ <magic priority="50">
+ <match value="ClamAV-VDB:" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.cvd"/>
+ <glob pattern="*.cld"/>
+ </mime-type>
+ <mime-type type="application/x-clamav-diff">
+ <_comment>ClamAV signature database update</_comment>
+ <magic priority="50">
+ <match value="ClamAV-Diff:" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.cdiff"/>
+ </mime-type>
<mime-type type="application/x-compress">
<magic priority="50">
@@ -4027,6 +4386,18 @@
<glob pattern="*.udeb"/>
</mime-type>
+ <mime-type type="application/x-device-tree-blob">
+ <_comment>Flattened devicetree (DTB, U-Boot FIT image)</_comment>
+ <magic priority="50">
+ <!-- magic, then version 16 or 17 at offset 20 -->
+ <match value="0xD00DFEED" type="string" offset="0">
+ <match value="0x00000010" mask="0xFFFFFFFE" type="string" offset="20"/>
+ </match>
+ </magic>
+ <glob pattern="*.dtbo"/>
+ <glob pattern="*.itb"/>
+ </mime-type>
+
<mime-type type="application/x-dex">
<acronym>DEX</acronym>
<_comment>Dalvik Executable Format</_comment>
@@ -4076,6 +4447,14 @@
<glob pattern="*.dvi"/>
</mime-type>
+ <mime-type type="application/x-elasto-mania-level">
+ <_comment>Elasto Mania level</_comment>
+ <magic priority="50">
+ <!-- Elma; POT06 is Across -->
+ <match value="POT14" type="string" offset="0"/>
+ <match value="POT06" type="string" offset="0"/>
+ </magic>
+ </mime-type>
<mime-type type="application/x-elc">
<_comment>Emacs Lisp bytecode</_comment>
<magic priority="50">
@@ -4143,6 +4522,13 @@
</match>
</magic>
</mime-type>
+ <mime-type type="application/x-fbx">
+ <_comment>Autodesk FBX (binary)</_comment>
+ <magic priority="50">
+ <match value="Kaydara FBX Binary \x00" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.fbx"/>
+ </mime-type>
<mime-type type="application/x-killustrator">
<_comment>KIllustrator File</_comment>
@@ -4170,6 +4556,17 @@
<glob pattern="*.scad"/>
<sub-class-of type="text/plain"/>
</mime-type>
+
+ <mime-type type="application/x-osm-pbf">
+ <_comment>OpenStreetMap PBF</_comment>
+ <magic priority="50">
+ <!-- big-endian BlobHeader length, then BlobHeader.type "OSMHeader" -->
+ <match value="0x0A094F534D486561646572" type="string" offset="4">
+ <match value="0x0000" type="string" offset="0"/>
+ </match>
+ </magic>
+ <glob pattern="*.osm.pbf"/>
+ </mime-type>
<mime-type type="application/x-extension-fcstd">
<_comment>FreeCAD document</_comment>
<tika:link>https://wiki.freecad.org/File_Format_FCStd</tika:link>
@@ -4186,6 +4583,13 @@
</match>
</magic>
</mime-type>
+ <mime-type type="application/x-fastcad">
+ <_comment>FastCAD drawing</_comment>
+ <magic priority="50">
+ <match value="FCW (FastCAD for Windows)" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.fcw"/>
+ </mime-type>
<mime-type type="application/x-sharedlib">
<sub-class-of type="application/x-elf"/>
<magic priority="50">
@@ -4214,6 +4618,20 @@
</magic>
<!-- too rare to use? <glob pattern="*.mmm"/> -->
</mime-type>
+ <mime-type type="application/x-dmp">
+ <_comment>Windows minidump</_comment>
+ <magic priority="50">
+ <match value="MDMP\x93\xa7" type="string" offset="0"/>
+ </magic>
+ </mime-type>
+ <mime-type type="application/x-windows-kernel-dump">
+ <_comment>Windows kernel memory dump</_comment>
+ <alias type="application/x-ms-dmp"/>
+ <magic priority="50">
+ <match value="PAGEDUMP" type="string" offset="0"/>
+ <match value="PAGEDU64" type="string" offset="0"/>
+ </magic>
+ </mime-type>
<mime-type type="application/x-dosexec">
<_comment>DOS/Windows executable (EXE)</_comment>
<!-- magic is MZ at offset=0, but that may not be unique enough -->
@@ -4238,6 +4656,12 @@
</magic>
<glob pattern="*.fp7" />
</mime-type>
+ <mime-type type="application/x-finale">
+ <_comment>Finale notation (Enigma binary)</_comment>
+ <magic priority="50">
+ <match value="ENIGMA BINARY FILE" type="string" offset="0"/>
+ </magic>
+ </mime-type>
<mime-type type="application/x-font-bdf">
<glob pattern="*.bdf"/>
@@ -4328,6 +4752,14 @@
<glob pattern="*.spl"/>
</mime-type>
+ <mime-type type="application/x-glulx">
+ <_comment>Glulx game</_comment>
+ <magic priority="50">
+ <match value="Glul\x00\x02" type="string" offset="0"/>
+ <match value="Glul\x00\x03" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.ulx"/>
+ </mime-type>
<mime-type type="application/x-gnucash">
<glob pattern="*.gnucash" />
</mime-type>
@@ -4386,6 +4818,15 @@
<sub-class-of type="application/x-tika-msoffice"/>
<glob pattern="*.lyr"/>
</mime-type>
+ <mime-type type="application/x-haihaisoft-drm">
+ <_comment>Haihaisoft DRM-X protected file</_comment>
+ <magic priority="50">
+ <!-- the <XDRMHEADER> that follows names its license server at drm-x.com
-->
+ <match value="XDRM" type="string" offset="0">
+ <match value="HEAD" type="string" offset="10"/>
+ </match>
+ </magic>
+ </mime-type>
<mime-type type="application/x-hdf">
<_comment>Hierarchical Data Format File</_comment>
<magic priority="50">
@@ -4437,6 +4878,13 @@
<match value="0x535200040000000A000004" type="string" offset="0"/>
</magic>
</mime-type>
+
+ <mime-type type="application/x-imagedisk">
+ <_comment>ImageDisk floppy image</_comment>
+ <magic priority="50">
+ <match value="IMD [0-9]\\.[0-9]{2}: " type="regex" offset="0"/>
+ </magic>
+ </mime-type>
<mime-type type="application/x-internet-archive">
<acronym>ARC</acronym>
<_comment>ARC</_comment>
@@ -4517,6 +4965,18 @@
<match value="JigsawDownload template" offset="0"/>
</magic>
</mime-type>
+ <mime-type type="application/x-jw-cad">
+ <_comment>Jw_cad drawing</_comment>
+ <magic priority="50">
+ <match value="JwwData." type="string" offset="0"/>
+ <match value="JwsData." type="string" offset="0"/>
+ <!-- older JWC drawings and symbols -->
+ <match value="jw_cad(c)" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.jww"/>
+ <glob pattern="*.jwc"/>
+ <glob pattern="*.jws"/>
+ </mime-type>
<mime-type type="application/x-kdelnk">
<magic priority="50">
<match value="[KDE\ Desktop\ Entry]" type="string" offset="0"/>
@@ -4555,7 +5015,31 @@
<match value="-lz5-" type="string" offset="2"/>
</magic>
</mime-type>
+
+ <mime-type type="application/x-ligo-frame">
+ <_comment>LIGO/Virgo gravitational-wave frame file</_comment>
+ <magic priority="50">
+ <match value="IGWD\x00" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.gwf"/>
+ </mime-type>
+ <mime-type type="application/x-lrzip">
+ <_comment>Long Range ZIP</_comment>
+ <magic priority="50">
+ <match value="LRZI\x00" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.lrz"/>
+ </mime-type>
+ <mime-type type="application/x-lucene-index">
+ <_comment>Apache Lucene index file</_comment>
+ <magic priority="50">
+ <!-- CodecUtil header: magic 0x3FD76C17, then the codec name (length
byte + ASCII) -->
+ <match value="0x3FD76C17" type="string" offset="0">
+ <match value="(?s)\\x3f\\xd7\\x6c\\x17[\\x02-\\x40][A-Za-z]"
type="regex" offset="0"/>
+ </match>
+ </magic>
+ </mime-type>
<mime-type type="application/x-lz4">
<_comment>First match LZ4 Frame</_comment>
<_comment>Second match Legacy Frame</_comment>
@@ -4578,6 +5062,13 @@
<_comment>LZMA compressed archive</_comment>
<glob pattern="*.lzma"/>
</mime-type>
+ <mime-type type="application/x-lzx">
+ <_comment>Amiga LZX archive</_comment>
+ <magic priority="50">
+ <match value="LZX\x00" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.lzx"/>
+ </mime-type>
<mime-type type="application/x-mach-o">
<_comment>Mach-O</_comment>
<tika:link>https://www.nationalarchives.gov.uk/PRONOM/fmt/692</tika:link>
@@ -4781,6 +5272,14 @@
<sub-class-of type="application/x-bplist"/>
<glob pattern="*.memgraph"/>
</mime-type>
+ <mime-type type="application/x-mikrotik-npk">
+ <_comment>MikroTik RouterOS package</_comment>
+ <magic priority="50">
+ <match value="0x1EF1D0BA" type="string" offset="0"/>
+ <match value="0xFB0F10A1" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.npk"/>
+ </mime-type>
<mime-type type="application/x-mobipocket-ebook">
<acronym>MOBI</acronym>
@@ -4847,6 +5346,19 @@
<glob pattern="*.clp"/>
</mime-type>
+ <mime-type type="application/x-mpq">
+ <_comment>Blizzard MPQ archive</_comment>
+ <magic priority="50">
+ <!-- archive header with its v1-v4 header size, or the user-data header
that precedes one -->
+ <match value="MPQ\x1a\x20\x00\x00\x00" type="string" offset="0"/>
+ <match value="MPQ\x1a\x2c\x00\x00\x00" type="string" offset="0"/>
+ <match value="MPQ\x1a\x44\x00\x00\x00" type="string" offset="0"/>
+ <match value="MPQ\x1a\xd0\x00\x00\x00" type="string" offset="0"/>
+ <match value="MPQ\x1b" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.mpq"/>
+ </mime-type>
+
<mime-type type="application/x-msdownload">
<glob pattern="*.dll"/>
<glob pattern="*.com"/>
@@ -5029,6 +5541,42 @@
<match value="0x43444601" type="string" offset="0"/>
</magic>
</mime-type>
+
+ <mime-type type="application/x-nexrad-level3">
+ <_comment>NEXRAD Level III radar product</_comment>
+ <magic priority="50">
+ <!-- WMO abbreviated heading for radar data (SDUSii CCCC YYGGgg), then
the AWIPS id -->
+ <match value="SDUS" type="string" offset="0">
+ <match value="SDUS[0-9]{2} [A-Z]{4} [0-9]{6}(?:
[A-Z]{3})?\\r\\r\\n[A-Z0-9]{4,6}\\r\\r\\n" type="regex" offset="0"/>
+ </match>
+ </magic>
+ </mime-type>
+ <mime-type type="application/x-navitel-map">
+ <_comment>Navitel Navigator map</_comment>
+ <magic priority="50">
+ <!-- crawl files name their source "OpenStreetMap (converted by
KonTur)", a Navitel OSM map series -->
+ <match value="PMGC\x05\x00" type="string" offset="0">
+ <match value="0x8400000084000000" type="string" offset="20"/>
+ </match>
+ </magic>
+ </mime-type>
+ <mime-type type="application/x-nufx">
+ <_comment>NuFX (ShrinkIt) archive</_comment>
+ <magic priority="50">
+ <match value="0x4EF546E96CE5" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.shk"/>
+ </mime-type>
+ <mime-type type="application/x-nwc">
+ <_comment>NoteWorthy Composer song</_comment>
+ <magic priority="50">
+ <!-- compressed: zlib stream follows -->
+ <match value="[NWZ]\x00\x78" type="string" offset="0"/>
+ <match value="[NoteWorthy ArtWare]" type="string" offset="0"/>
+ <match value="[NoteWorthy Composer]" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.nwc"/>
+ </mime-type>
<mime-type type="application/x-parquet">
<glob pattern="*.parquet"/>
<magic priority="50">
@@ -5074,6 +5622,80 @@
<match value="0M3C" type="string" offset="8" />
</magic>
</mime-type>
+ <mime-type type="application/x-promethean-flipchart">
+ <_comment>Promethean ActivInspire flipchart</_comment>
+ <!-- " Bamboo " is the container; its zlib members carry the
prometheanworld.com/Inspire XML namespace -->
+ <magic priority="50">
+ <match value=" Bamboo " type="string" offset="0"/>
+ </magic>
+ </mime-type>
+ <mime-type type="application/x-ps3-package">
+ <_comment>PlayStation 3/PSP package</_comment>
+ <magic priority="50">
+ <!-- then package type: 0x80000001 retail, 0x00000001 debug -->
+ <match value="0x7F504B4700000001" mask="0xFFFFFFFF7FFFFFFF"
type="string" offset="0"/>
+ </magic>
+ </mime-type>
+ <mime-type type="application/x-ps3-theme">
+ <_comment>PlayStation 3 theme</_comment>
+ <magic priority="50">
+ <match value="P3TF\x00\x00" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.p3t"/>
+ </mime-type>
+ <mime-type type="application/x-python-bytecode">
+ <_comment>Python bytecode</_comment>
+ <magic priority="50">
+ <!-- 2.3-2.7 magic numbers, then CR LF -->
+ <match value="0x3BF20D0A" type="string" offset="0"/>
+ <match value="0x6DF20D0A" type="string" offset="0"/>
+ <match value="0xB3F20D0A" type="string" offset="0"/>
+ <match value="0xD1F20D0A" type="string" offset="0"/>
+ <match value="0x03F30D0A" type="string" offset="0"/>
+ <!-- 3.x: a little-endian magic (3.0 is 3131, 3.14 is 3627), then CR LF.
Text whose first line is
+ one character with CR CR LF endings looks the same, so also check
the rest of the header -->
+ <match value="0x0D0A" type="string" offset="2">
+ <!-- 3.0-3.4: the code object after the 8- or 12-byte header -->
+ <match value="0x0C" type="string" offset="1">
+ <match value="c" type="string" offset="8"/>
+ <match value="c" type="string" offset="12"/>
+ <match value="0xE3" type="string" offset="12"/>
+ </match>
+ <!-- 3.5-3.6: a code object at 12; 3.7+: PEP 552 flags 0-3 -->
+ <match value="0x0D" type="string" offset="1">
+ <match value="0xE3" type="string" offset="12"/>
+ <match value="0x00000000" mask="0xFCFFFFFF" type="string"
offset="4"/>
+ </match>
+ <match value="0x0E" type="string" offset="1">
+ <match value="0x00000000" mask="0xFCFFFFFF" type="string"
offset="4"/>
+ </match>
+ </match>
+ </magic>
+ <glob pattern="*.pyc"/>
+ <glob pattern="*.pyo"/>
+ </mime-type>
+ <mime-type type="application/x-python-pickle">
+ <_comment>Python pickle</_comment>
+ <magic priority="50">
+ <!-- protocol 2-3, then a dict/list/mark/global opcode; protocol 4-5
start with FRAME -->
+ <match value="0x80" type="string" offset="0">
+ <match value="(?s)\\x80[\\x02\\x03][}\\](c]" type="regex" offset="0"/>
+ <match value="(?s)\\x80[\\x04\\x05]\\x95" type="regex" offset="0"/>
+ </match>
+ </magic>
+ <glob pattern="*.pkl"/>
+ <glob pattern="*.pickle"/>
+ </mime-type>
+
+ <mime-type type="application/x-qemu-disk">
+ <_comment>QEMU copy-on-write disk image</_comment>
+ <magic priority="50">
+ <!-- qcow version 1, 2 or 3 -->
+ <match value="(?s)QFI\\xfb\\x00\\x00\\x00[\\x01-\\x03]" type="regex"
offset="0"/>
+ </magic>
+ <glob pattern="*.qcow"/>
+ <glob pattern="*.qcow2"/>
+ </mime-type>
<mime-type type="application/x-quattro-pro">
<_comment>
@@ -5357,6 +5979,22 @@
</magic>
<glob pattern="*.sz"/>
</mime-type>
+ <mime-type type="application/x-source-bsp">
+ <_comment>Valve Source engine map</_comment>
+ <magic priority="50">
+ <!-- then version 0x10-0x1f (Half-Life 2 .. CS:GO use 19-21) -->
+ <match value="VBSP" type="string" offset="0">
+ <match value="0x11" mask="0xF0" type="string" offset="4"/>
+ </match>
+ </magic>
+ </mime-type>
+ <mime-type type="application/x-spectrum-rzx">
+ <_comment>ZX Spectrum input recording (RZX)</_comment>
+ <magic priority="50">
+ <match value="RZX!\x00" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.rzx"/>
+ </mime-type>
<mime-type type="application/x-sfdu">
<_comment>Standard Formatted Data Units (SFDUs) data</_comment>
@@ -5409,14 +6047,6 @@
</magic>
<sub-class-of type="application/x-sqlite3"/>
</mime-type>
- <mime-type type="application/x-fossil-checkout">
- <magic priority="60">
- <match value="SQLite format 3\x00" type="string" offset="0">
- <match value="0x0f055112" type="string" offset="68"/>
- </match>
- </magic>
- <sub-class-of type="application/x-sqlite3"/>
- </mime-type>
<mime-type type="application/x-fossil-global-conf">
<magic priority="60">
<match value="SQLite format 3\x00" type="string" offset="0">
@@ -5457,6 +6087,16 @@
</magic>
<sub-class-of type="application/x-sqlite3"/>
</mime-type>
+ <mime-type type="application/x-mozlz4">
+ <_comment>Mozilla LZ4 (Firefox jsonlz4/mozlz4)</_comment>
+ <alias type="application/x-lz4+json"/>
+ <magic priority="50">
+ <match value="mozLz40\x00" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.jsonlz4"/>
+ <glob pattern="*.mozlz4"/>
+ <glob pattern="*.baklz4"/>
+ </mime-type>
<mime-type type="application/x-esri-spatially-enabled-db">
<magic priority="60">
<match value="SQLite format 3\x00" type="string" offset="0">
@@ -5481,6 +6121,26 @@
</magic>
<sub-class-of type="application/x-sqlite3"/>
</mime-type>
+
+ <mime-type type="application/vnd.squashfs">
+ <alias type="application/x-squashfs"/>
+ <_comment>SquashFS filesystem image</_comment>
+ <magic priority="50">
+ <!-- superblock magic, then s_major (2-4) at offset 28 in the same byte
order -->
+ <match value="hsqs" type="string" offset="0">
+ <match value="0x0400" type="string" offset="28"/>
+ <match value="0x0300" type="string" offset="28"/>
+ <match value="0x0200" type="string" offset="28"/>
+ </match>
+ <match value="sqsh" type="string" offset="0">
+ <match value="0x0004" type="string" offset="28"/>
+ <match value="0x0003" type="string" offset="28"/>
+ <match value="0x0002" type="string" offset="28"/>
+ </match>
+ </magic>
+ <glob pattern="*.sqsh"/>
+ <glob pattern="*.squashfs"/>
+ </mime-type>
<mime-type type="application/x-stata-do">
<_comment>Stata DTA Script</_comment>
<acronym>DO</acronym>
@@ -5536,6 +6196,9 @@
<mime-type type="application/x-stuffit">
<magic priority="50">
<match value="StuffIt" type="string" offset="0"/>
+ <match value="SIT!" type="string" offset="0">
+ <match value="rLau" type="string" offset="10"/>
+ </match>
</magic>
<glob pattern="*.sit"/>
</mime-type>
@@ -5610,6 +6273,32 @@
</magic>
<!-- do we want to use this for such a rare format? <glob
pattern="*.rpy"/> -->
</mime-type>
+ <mime-type type="application/x-trainz-cdp">
+ <_comment>Trainz content package</_comment>
+ <magic priority="50">
+ <match value="ACS$\x01\x00\x00\x00\x00\x00\x00\x00" type="string"
offset="0"/>
+ </magic>
+ <glob pattern="*.cdp"/>
+ </mime-type>
+
+ <mime-type type="application/x-ubi-image">
+ <_comment>UBI flash image</_comment>
+ <magic priority="50">
+ <!-- erase-count header, version 1 -->
+ <match value="UBI#\x01" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.ubi"/>
+ </mime-type>
+
+ <mime-type type="application/x-uboot-image">
+ <_comment>U-Boot legacy image</_comment>
+ <magic priority="50">
+ <!-- magic, then OS, architecture, image type and compression codes at
offset 28 -->
+ <match value="0x27051956" type="string" offset="0">
+ <match
value="(?s).{28}[\\x00-\\x18][\\x00-\\x1a][\\x00-\\x18][\\x00-\\x05]"
type="regex" offset="0"/>
+ </match>
+ </magic>
+ </mime-type>
<!-- =================================================================== -->
<!-- Microsoft Office binary file formats -->
@@ -5689,6 +6378,30 @@
</magic>
<glob pattern="*.uc2"/>
</mime-type>
+
+ <mime-type type="application/x-ucsc-2bit">
+ <_comment>UCSC 2bit sequence</_comment>
+ <magic priority="50">
+ <!-- 0x1A412743 in either byte order, then version 0 or 1 (64-bit
offsets) -->
+ <match value="0x4327411A00000000" mask="0xFFFFFFFFFEFFFFFF"
type="string" offset="0"/>
+ <match value="0x1A41274300000000" mask="0xFFFFFFFFFFFFFFFE"
type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.2bit"/>
+ </mime-type>
+ <mime-type type="application/x-unreal-uz">
+ <_comment>Unreal Tournament compressed package</_comment>
+ <magic priority="50">
+ <!-- UZ (1234) or UZ2 (5678), then the original file name: compact
length, name, NUL -->
+ <match value="0xD2040000" type="string" offset="0">
+ <match value="(?s)[\\x02-\\x3f][\\x20-\\x7e]{1,63}\\.u[a-z]{0,2}\\x00"
type="regex" offset="4"/>
+ </match>
+ <match value="0x2E160000" type="string" offset="0">
+ <match value="(?s)[\\x02-\\x3f][\\x20-\\x7e]{1,63}\\.u[a-z]{0,2}\\x00"
type="regex" offset="4"/>
+ </match>
+ </magic>
+ <glob pattern="*.uz"/>
+ <glob pattern="*.uz2"/>
+ </mime-type>
<mime-type type="application/x-ustar">
<glob pattern="*.ustar"/>
</mime-type>
@@ -5716,6 +6429,15 @@
<sub-class-of type="application/x-bplist"/>
<glob pattern="*.webarchive"/>
</mime-type>
+ <mime-type type="application/x-wfdb-annotation">
+ <_comment>PhysioNet WFDB annotation</_comment>
+ <magic priority="50">
+ <!-- NOTE annotation (type 22) carrying the "## time resolution" aux
string -->
+ <match value="0x0058" type="string" offset="0">
+ <match value="## time resolution: " type="string" offset="4"/>
+ </match>
+ </magic>
+ </mime-type>
<mime-type type="application/x-x509-cert">
<alias type="application/x-x509-ca-cert"/>
@@ -5732,6 +6454,13 @@
<mime-type type="application/x-x509-cert;format=der">
<sub-class-of type="application/x-x509-cert"/>
<glob pattern="*.der"/>
+ <magic priority="50">
+ <!-- Certificate { TBSCertificate { [0] version v1-v3, serialNumber
INTEGER ...
+ any signature algorithm (SM2, Ed25519, ...) -->
+ <match value="0x3080" mask="0xFFFC" type="string" offset="0">
+ <match
value="(?s)\\x30(?:\\x81.|\\x82..|\\x83...)\\x30(?:\\x81.|\\x82..|\\x83...)\\xa0\\x03\\x02\\x01[\\x00-\\x02]\\x02"
type="regex" offset="0"/>
+ </match>
+ </magic>
<magic priority="50">
<match value="0x3080" type="string"
mask="0xFFF8" offset="0">
@@ -5814,6 +6543,39 @@
<!-- <glob pattern="*.xmap"/> -->
</mime-type>
+ <mime-type type="application/x-vcdiff">
+ <_comment>VCDIFF delta (RFC 3284)</_comment>
+ <magic priority="50">
+ <!-- header bytes, version 0, then Hdr_Indicator with only its low 3
bits defined -->
+ <match value="0xD6C3C40000" mask="0xFFFFFFFFF8" type="string"
offset="0"/>
+ <!-- open-vcdiff's SDCH interleaved format puts 'S' in the version byte
-->
+ <match value="0xD6C3C45300" mask="0xFFFFFFFFF8" type="string"
offset="0"/>
+ </magic>
+ <glob pattern="*.vcdiff"/>
+ </mime-type>
+ <mime-type type="application/x-warcraft3-map">
+ <_comment>Warcraft III map</_comment>
+ <sub-class-of type="application/x-mpq"/>
+ <magic priority="50">
+ <match value="HM3W\x00\x00\x00\x00" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.w3m"/>
+ <glob pattern="*.w3x"/>
+ </mime-type>
+ <mime-type type="application/x-warcraft3-replay">
+ <_comment>Warcraft III replay</_comment>
+ <magic priority="50">
+ <match value="Warcraft III recorded game\x1a" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.w3g"/>
+ </mime-type>
+ <mime-type type="application/x-xdelta">
+ <_comment>Xdelta 1 patch</_comment>
+ <magic priority="50">
+ <match value="%XDZ00" type="string" offset="0"/>
+ </magic>
+ </mime-type>
+
<mime-type type="application/x-xz">
<glob pattern="*.xz"/>
<magic priority="50">
@@ -5833,6 +6595,13 @@
</magic>
<glob pattern="*.zoo"/>
</mime-type>
+ <mime-type type="application/x-zsync">
+ <_comment>zsync control file</_comment>
+ <magic priority="50">
+ <match value="zsync: " type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.zsync"/>
+ </mime-type>
<mime-type type="application/x400-bp"/>
<mime-type type="application/xcap-att+xml"/>
@@ -5883,6 +6652,12 @@
<magic priority="30">
<match value="<!--" type="string" offset="0"/>
</magic>
+ <!-- ties the text/html fragment rule (which sorts first) so a .xml name
can pick XML -->
+ <magic priority="15">
+ <match value="<" type="string" offset="0">
+ <match
value="(?i)</?(meta|script|p|br|table|font|center|small|img|a|b)[\\s>/]"
type="regex" offset="0"/>
+ </match>
+ </magic>
<glob pattern="*.xml"/>
<glob pattern="*.xsl"/>
<glob pattern="*.xsd"/>
@@ -6245,6 +7020,24 @@
<glob pattern="*.gslib"/>
<glob pattern="*.minigsf"/>
</mime-type>
+ <mime-type type="audio/x-rex2">
+ <_comment>Propellerhead ReCycle REX2 loop</_comment>
+ <magic priority="50">
+ <match value="CAT " type="string" offset="0">
+ <match value="REX2HEAD" type="string" offset="8"/>
+ </match>
+ </magic>
+ <glob pattern="*.rx2"/>
+ </mime-type>
+ <mime-type type="audio/x-s3m">
+ <_comment>Scream Tracker 3 module</_comment>
+ <magic priority="50">
+ <match value="0x1A10" type="string" offset="28">
+ <match value="SCRM" type="string" offset="44"/>
+ </match>
+ </magic>
+ <glob pattern="*.s3m"/>
+ </mime-type>
<mime-type type="audio/x-sap">
<_comment>Slight Atari Player</_comment>
<tika:link>https://asap.sourceforge.net/sap-format.html</tika:link>
@@ -6253,6 +7046,32 @@
</magic>
<glob pattern="*.sap"/>
</mime-type>
+ <mime-type type="audio/x-soundfont">
+ <_comment>SoundFont 2 bank</_comment>
+ <alias type="audio/x-sfbk"/>
+ <magic priority="50">
+ <match value="RIFF" type="string" offset="0">
+ <match value="sfbk" type="string" offset="8"/>
+ </match>
+ </magic>
+ <glob pattern="*.sf2"/>
+ </mime-type>
+
+ <mime-type type="audio/x-ym">
+ <_comment>YM Atari ST / Amstrad CPC chiptune</_comment>
+ <magic priority="50">
+ <match value="LeOnArD!" type="string" offset="4">
+ <match value="YM4!" type="string" offset="0"/>
+ <match value="YM5!" type="string" offset="0"/>
+ <match value="YM6!" type="string" offset="0"/>
+ </match>
+ <!-- older raw register dumps have no signature -->
+ <match value="YM2!" type="string" offset="0"/>
+ <match value="YM3!" type="string" offset="0"/>
+ <match value="YM3b" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.ym"/>
+ </mime-type>
<mime-type type="audio/x-oggflac">
<_comment>Ogg Packaged Free Lossless Audio Codec</_comment>
<alias type="audio/x-ogg-flac"/>
@@ -6309,9 +7128,19 @@
<mime-type type="audio/pcmu"/>
<mime-type type="audio/prs.sid">
+ <_comment>Commodore 64 SID tune</_comment>
<magic priority="50">
- <match value="PSID" type="string" offset="0"/>
+ <!-- PSID/RSID, then version 1-4 -->
+ <match value="PSID\x00\x01" type="string" offset="0"/>
+ <match value="PSID\x00\x02" type="string" offset="0"/>
+ <match value="PSID\x00\x03" type="string" offset="0"/>
+ <match value="PSID\x00\x04" type="string" offset="0"/>
+ <match value="RSID\x00\x02" type="string" offset="0"/>
+ <match value="RSID\x00\x03" type="string" offset="0"/>
+ <match value="RSID\x00\x04" type="string" offset="0"/>
</magic>
+ <glob pattern="*.sid"/>
+ <glob pattern="*.psid"/>
</mime-type>
<mime-type type="audio/qcelp"/>
@@ -6383,6 +7212,17 @@
<mime-type type="audio/vnd.sealedmedia.softseal.mpeg"/>
<mime-type type="audio/vnd.vmx.cvsd"/>
<mime-type type="audio/vorbis-config"/>
+ <mime-type type="audio/x-ape">
+ <_comment>Monkey's Audio</_comment>
+ <magic priority="50">
+ <!-- then format version 3.80-4.xx -->
+ <match value="MAC " type="string" offset="0">
+ <match value="0x0E" type="string" offset="5"/>
+ <match value="0x0F" type="string" offset="5"/>
+ </match>
+ </magic>
+ <glob pattern="*.ape"/>
+ </mime-type>
<mime-type type="audio/x-aac">
<alias type="audio/aac"/>
<glob pattern="*.aac"/>
@@ -6566,6 +7406,36 @@
<mime-type type="chemical/x-xyz">
<glob pattern="*.xyz"/>
</mime-type>
+
+ <mime-type type="font/woff">
+ <alias type="application/font-woff"/>
+ <alias type="application/x-font-woff"/>
+ <acronym>WOFF</acronym>
+ <_comment>Web Open Font Format</_comment>
+ <tika:link>https://www.w3.org/TR/WOFF/</tika:link>
+ <magic priority="50">
+ <match value="wOFF" type="string" offset="0">
+ <match value="0x00010000" type="string" offset="4"/>
+ <match value="OTTO" type="string" offset="4"/>
+ <match value="true" type="string" offset="4"/>
+ </match>
+ </magic>
+ <glob pattern="*.woff"/>
+ </mime-type>
+ <mime-type type="font/woff2">
+ <alias type="application/font-woff2"/>
+ <acronym>WOFF2</acronym>
+ <_comment>Web Open Font Format 2</_comment>
+ <tika:link>https://www.w3.org/TR/WOFF2/</tika:link>
+ <magic priority="50">
+ <match value="wOF2" type="string" offset="0">
+ <match value="0x00010000" type="string" offset="4"/>
+ <match value="OTTO" type="string" offset="4"/>
+ <match value="true" type="string" offset="4"/>
+ </match>
+ </magic>
+ <glob pattern="*.woff2"/>
+ </mime-type>
<mime-type type="image/x-3ds">
<_comment>3D Studio (V1)</_comment>
<magic priority="50">
@@ -6755,6 +7625,7 @@
</mime-type>
<mime-type type="image/heif-sequence">
<_comment>HEIF Sequence - High Efficiency Image Sequence</_comment>
+
<tika:link>https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format</tika:link>
<magic priority="60">
<match value="ftypmsf1" type="string" offset="4"/>
</magic>
@@ -6765,6 +7636,7 @@
<alias type="image/hevc"/>
<_comment>HEIF Image using HEVC Codec</_comment>
<acronym>HEIC</acronym>
+
<tika:link>https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format</tika:link>
<magic priority="60">
<match value="ftypheic" type="string" offset="4"/>
<match value="ftypheix" type="string" offset="4"/>
@@ -6776,6 +7648,7 @@
<alias type="image/hevc-sequence"/>
<_comment>HEIF Sequence using HEVC Codec</_comment>
<acronym>HEVC</acronym>
+
<tika:link>https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format</tika:link>
<magic priority="60">
<match value="ftyphevc" type="string" offset="4"/>
<match value="ftyphevx" type="string" offset="4"/>
@@ -7076,6 +7949,16 @@
</mime-type>
<mime-type type="image/vnd.mix"/>
+
+ <mime-type type="image/vnd.ms-dds">
+ <alias type="image/x-dds"/>
+ <_comment>DirectDraw Surface</_comment>
+ <magic priority="50">
+ <!-- magic, then the 124-byte header size -->
+ <match value="DDS \x7c\x00\x00\x00" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.dds"/>
+ </mime-type>
<mime-type type="image/vnd.ms-modi">
<glob pattern="*.mdi"/>
<_comment>Microsoft Document Imaging</_comment>
@@ -7136,38 +8019,6 @@
<glob pattern="*.webp"/>
</mime-type>
- <mime-type type="image/heic">
-
<tika:link>https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format</tika:link>
- <magic priority="50">
- <match value="ftypheic" type="string" offset="4"/>
- <match value="ftypheix" type="string" offset="4"/>
- </magic>
- <glob pattern="*.heic"/>
- </mime-type>
-
- <mime-type type="image/heif">
-
<tika:link>https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format</tika:link>
- <magic priority="50">
- <match value="ftypmif1" type="string" offset="4"/>
- </magic>
- <glob pattern="*.heif"/>
- </mime-type>
-
- <mime-type type="image/heif-sequence">
-
<tika:link>https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format</tika:link>
- <magic priority="50">
- <match value="ftypmsf1" type="string" offset="4"/>
- </magic>
- </mime-type>
-
- <mime-type type="image/heic-sequence">
-
<tika:link>https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format</tika:link>
- <magic priority="50">
- <match value="ftyphevc" type="string" offset="4"/>
- <match value="ftyphevx" type="string" offset="4"/>
- </magic>
- </mime-type>
-
<mime-type type="image/wmf">
<alias type="image/x-wmf"/>
<alias type="application/x-msmetafile"/>
@@ -7258,6 +8109,15 @@
</magic>
<glob pattern="*.jxl"/>
</mime-type>
+ <mime-type type="image/jxr">
+ <_comment>JPEG XR (HD Photo)</_comment>
+ <magic priority="50">
+ <match value="0x4949BC01" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.jxr"/>
+ <glob pattern="*.wdp"/>
+ <glob pattern="*.hdp"/>
+ </mime-type>
<mime-type type="image/x-niff">
<_comment>Navy Interchange File Format</_comment>
@@ -7266,6 +8126,14 @@
</magic>
</mime-type>
+ <mime-type type="image/x-paintshoppro">
+ <_comment>Paint Shop Pro image</_comment>
+ <magic priority="50">
+ <match value="Paint Shop Pro Image File\x0a\x1a" type="string"
offset="0"/>
+ </magic>
+ <glob pattern="*.pspimage"/>
+ </mime-type>
+
<mime-type type="image/x-pict">
<_comment>Apple Macintosh QuickDraw/PICT Format</_comment>
<magic priority="50">
@@ -7338,6 +8206,14 @@
</magic>
<glob pattern="*.ppm"/>
</mime-type>
+ <mime-type type="image/x-quake-sprite">
+ <_comment>Quake (v1) / Half-Life (v2) sprite</_comment>
+ <magic priority="50">
+ <match value="IDSP\x01\x00\x00\x00" type="string" offset="0"/>
+ <match value="IDSP\x02\x00\x00\x00" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.spr"/>
+ </mime-type>
<mime-type type="image/x-portable-arbitrarymap">
<sub-class-of type="image/x-portable-anymap"/>
<acronym>PAM</acronym>
@@ -7767,6 +8643,13 @@
<glob pattern="*.e57"/>
</mime-type>
<mime-type type="model/example"/>
+ <mime-type type="model/gltf-binary">
+ <_comment>glTF binary</_comment>
+ <magic priority="50">
+ <match value="glTF\x02\x00\x00\x00" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.glb"/>
+ </mime-type>
<mime-type type="model/iges">
<_comment>Initial Graphics Exchange Specification Format</_comment>
<glob pattern="*.igs"/>
@@ -8006,6 +8889,13 @@
<match value="(?i)<(html|head|body|title|div)[ >]" type="regex"
offset="0"/>
<match value="(?i)<h[123][ >]" type="regex" offset="0"/>
</magic>
+ <!-- fragments served as pages: meta-first, PHP warnings, partials; below
the weak
+ %PDF-/xhtml matches so a PDF or XHTML with a junk HTML prefix keeps
its type -->
+ <magic priority="15">
+ <match value="<" type="string" offset="0">
+ <match
value="(?i)</?(meta|script|p|br|table|font|center|small|img|a|b)[\\s>/]"
type="regex" offset="0"/>
+ </match>
+ </magic>
<!-- The magic priority needs to be lower than that of -->
<!-- files which contain HTML within them, eg mime emails -->
<magic priority="40">
@@ -8967,7 +9857,32 @@
<mime-type type="video/mp1s"/>
<mime-type type="video/mp2p"/>
- <mime-type type="video/mp2t"/>
+ <mime-type type="video/mp2t">
+ <magic priority="50">
+ <!-- sync byte on five consecutive 188-byte packets, or 192-byte M2TS
packets -->
+ <match value="0x47" type="string" offset="0">
+ <match value="0x47" type="string" offset="188">
+ <match value="0x47" type="string" offset="376">
+ <match value="0x47" type="string" offset="564">
+ <match value="0x47" type="string" offset="752"/>
+ </match>
+ </match>
+ </match>
+ </match>
+ <match value="0x47" type="string" offset="4">
+ <match value="0x47" type="string" offset="196">
+ <match value="0x47" type="string" offset="388">
+ <match value="0x47" type="string" offset="580">
+ <match value="0x47" type="string" offset="772"/>
+ </match>
+ </match>
+ </match>
+ </match>
+ </magic>
+ <!-- not *.ts (TypeScript) or *.mts (model/vnd.mts) -->
+ <glob pattern="*.m2ts"/>
+ <glob pattern="*.m2t"/>
+ </mime-type>
<mime-type type="video/mp4">
<magic priority="60">
@@ -9150,10 +10065,21 @@
</mime-type>
<mime-type type="video/x-flc">
+ <magic priority="50">
+ <!-- frame type, then 8-bit depth -->
+ <match value="0xAF12" type="little16" offset="4">
+ <match value="0x0008" type="little16" offset="12"/>
+ </match>
+ </magic>
<glob pattern="*.flc"/>
</mime-type>
<mime-type type="video/x-fli">
+ <magic priority="50">
+ <match value="0xAF11" type="little16" offset="4">
+ <match value="0x0008" type="little16" offset="12"/>
+ </match>
+ </magic>
<glob pattern="*.fli"/>
</mime-type>
@@ -9194,11 +10120,24 @@
<match value="0x3026b275" type="big32" offset="0" />
</magic>
</mime-type>
+ <mime-type type="application/x-ms-reader">
+ <_comment>Microsoft Reader eBook</_comment>
+ <magic priority="50">
+ <match value="ITOLITLS\x01\x00\x00\x00" type="string" offset="0"/>
+ </magic>
+ <glob pattern="*.lit"/>
+ </mime-type>
<mime-type type="application/x-ms-asx">
<_comment>Windows Media Metafile</_comment>
<glob pattern="*.asx"/>
<root-XML localName="asx"/>
<root-XML localName="ASX"/>
+ <!-- often no XML declaration -->
+ <magic priority="50">
+ <match value="<" type="string" offset="0">
+ <match value="(?i)<asx[\\s>]" type="regex" offset="0"/>
+ </match>
+ </magic>
<sub-class-of type="application/xml"/>
</mime-type>
<mime-type type="video/x-ms-wm">
@@ -9264,6 +10203,16 @@
<sub-class-of type="application/x-matroska"/>
<glob pattern="*.mka" />
</mime-type>
+ <mime-type type="audio/x-mfi">
+ <_comment>i-mode MFi melody</_comment>
+ <magic priority="50">
+ <!-- then lengths, types, track count and the first chunk name -->
+ <match value="melo" type="string" offset="0">
+ <match value="[a-z]{4}" type="regex" offset="13"/>
+ </match>
+ </magic>
+ <glob pattern="*.mld"/>
+ </mime-type>
<mime-type type="video/webm">
<sub-class-of type="application/x-matroska"/>
diff --git
a/tika-core/src/test/java/org/apache/tika/mime/MimeDetectionTest.java
b/tika-core/src/test/java/org/apache/tika/mime/MimeDetectionTest.java
index 7dd0178127..3c9d3a3e11 100644
--- a/tika-core/src/test/java/org/apache/tika/mime/MimeDetectionTest.java
+++ b/tika-core/src/test/java/org/apache/tika/mime/MimeDetectionTest.java
@@ -204,6 +204,216 @@ public class MimeDetectionTest {
assertEquals(ani, detect(acon, "pointer.cur"));
}
+ @Test
+ public void testWoffDetection() throws Exception {
+ assertDetected(MediaType.parse("font/woff"), header(0,
"wOFF\u0000\u0001\u0000\u0000"));
+ assertDetected(MediaType.parse("font/woff"), header(0, "wOFFOTTO"));
+ assertDetected(MediaType.parse("font/woff2"), header(0,
"wOF2\u0000\u0001\u0000\u0000"));
+ assertDetected(MediaType.OCTET_STREAM, header(0, "wOFFxxxx"));
+ }
+
+ @Test
+ public void testSquashfsDetection() throws Exception {
+ MediaType squashfs = MediaType.application("vnd.squashfs");
+ byte[] le = header(0, "hsqs");
+ le[28] = 4;
+ assertDetected(squashfs, le);
+ byte[] be = header(0, "sqsh");
+ be[29] = 3;
+ assertDetected(squashfs, be);
+ // no plausible major version
+ assertDetected(MediaType.OCTET_STREAM, header(0, "hsqs"));
+ }
+
+ @Test
+ public void testAdobeSwatchExchangeDetection() throws Exception {
+ assertDetected(MediaType.application("x-adobe-swatch-exchange"),
+ header(0, "ASEF\u0000\u0001\u0000\u0000"));
+ assertDetected(MediaType.OCTET_STREAM, header(0,
"ASEF\u0000\u0002\u0000\u0000"));
+ }
+
+ @Test
+ public void testPalmDocDetection() throws Exception {
+ assertDetected(MediaType.application("x-aportisdoc"), header(60,
"TEXtREAd"));
+ assertDetected(MediaType.application("x-mobipocket-ebook"), header(60,
"BOOKMOBI"));
+ }
+
+ @Test
+ public void testBinaryPgpSignatureDetection() throws Exception {
+ MediaType sig = MediaType.application("pgp-signature");
+ // v4, binary sig type, RSA, SHA-256; 2-byte length
+ assertDetected(sig, bytes(0x89, 0x01, 0x1c, 0x04, 0x00, 0x01, 0x08,
0x00));
+ // v4, text sig type, EdDSA; 1-byte length
+ assertDetected(sig, bytes(0x88, 0x75, 0x04, 0x01, 0x16, 0x08, 0x00));
+ // v3, hashed length 5
+ assertDetected(sig, bytes(0x88, 0x3f, 0x03, 0x05, 0x00, 0x3f, 0xc0));
+ // unknown public-key algorithm
+ assertDetected(MediaType.OCTET_STREAM, bytes(0x89, 0x01, 0x1c, 0x04,
0x00, 0x07, 0x08));
+ // new-format packets (RFC 9580 4.2): v6 Ed25519 SHA-512 with a 4-byte
hashed length; v4 RSA
+ assertDetected(sig, bytes(0xc2, 0x75, 0x06, 0x00, 0x1b, 0x0a, 0x00,
0x00, 0x00, 0x29));
+ assertDetected(sig, bytes(0xc2, 0xc0, 0x1c, 0x04, 0x00, 0x01, 0x08,
0x00));
+ // direct-key self-signature, not a detached one
+ assertDetected(MediaType.OCTET_STREAM, bytes(0xc2, 0x75, 0x06, 0x1f,
0x1b, 0x0a, 0x00));
+ assertDetected(MediaType.image("png"),
+ bytes(0x89, 'P', 'N', 'G', 0x0d, 0x0a, 0x1a, 0x0a, 0, 0, 0,
0x0d, 'I', 'H', 'D', 'R'));
+ }
+
+ @Test
+ public void testEmbeddedOpenTypeVersions() throws Exception {
+ MediaType eot = MediaType.application("vnd.ms-fontobject");
+ for (int[] version : new int[][]{{0, 0, 1, 0}, {1, 0, 2, 0}, {2, 0, 2,
0}}) {
+ byte[] b = header(34, "LP");
+ for (int i = 0; i < 4; i++) {
+ b[8 + i] = (byte) version[i];
+ }
+ assertDetected(eot, b);
+ }
+ }
+
+ @Test
+ public void testBitTorrentFirstKeys() throws Exception {
+ MediaType torrent = MediaType.application("x-bittorrent");
+ assertDetected(torrent,
"d8:announce35:udp://tracker".getBytes(US_ASCII));
+ assertDetected(torrent,
"d13:announce-listll35:udp".getBytes(US_ASCII));
+ assertDetected(torrent, "d7:comment5:hello4:infod".getBytes(US_ASCII));
+ assertDetected(torrent, "d4:infod6:lengthi1024e".getBytes(US_ASCII));
+ }
+
+ @Test
+ public void testMxfDetection() throws Exception {
+ assertDetected(MediaType.application("mxf"),
+ bytes(0x06, 0x0e, 0x2b, 0x34, 0x02, 0x05, 0x01, 0x01, 0x0d,
0x01, 0x02, 0x01,
+ 0x01, 0x02, 0x04, 0x00));
+ }
+
+ @Test
+ public void testJavaSerializationDetection() throws Exception {
+ MediaType ser = MediaType.application("java-serialized-object");
+ assertDetected(ser, bytes(0xac, 0xed, 0x00, 0x05, 0x73, 0x72, 0x00,
0x10));
+ assertDetected(ser, bytes(0xac, 0xed, 0x00, 0x05, 0x77, 0x04));
+ assertDetected(MediaType.OCTET_STREAM, bytes(0xac, 0xed, 0x53, 0xab,
0x49));
+ }
+
+ @Test
+ public void testHtmlFragmentDetection() throws Exception {
+ for (String s : new String[]{
+ "<meta http-equiv=\"Content-Type\" content=\"text/html\">",
+ "<br />\n<b>Warning</b>: filesize()",
+ "<p><a href='http://example.com'>x</a>",
+ "</b> <br>\n<font face=Arial>",
+ "<script type='text/javascript'>\nvar x = 1;",
+ "<SMALL>x</SMALL>"}) {
+ assertDetected(MediaType.TEXT_HTML, s.getBytes(US_ASCII));
+ }
+ assertDetected(MediaType.TEXT_PLAIN, "<abbr
title=x>".getBytes(US_ASCII));
+ assertDetected(MediaType.TEXT_PLAIN, "<bread> and
butter".getBytes(US_ASCII));
+ // injected analytics in front of a real document
+ assertDetected(MediaType.application("pdf"),
+ "<script>\n
(function(i,s,o,g,r,a,m){})\n</script>\n%PDF-1.4\n".getBytes(US_ASCII));
+ assertDetected(MediaType.application("xhtml+xml"),
+ ("<script src=\"urchin.js\"></script>\n" + "<!-- x
-->\n".repeat(12)
+ + "<html
xmlns=\"http://www.w3.org/1999/xhtml\";>").getBytes(US_ASCII));
+ // a .xml name still picks XML, except for roots text/html claims
(TIKA-377)
+ byte[] table = "<table name=\"t\"><row
id=\"1\"/></table>".getBytes(US_ASCII);
+ assertEquals(MediaType.APPLICATION_XML, detect(table, "test.xml"));
+ assertEquals(MediaType.TEXT_HTML, detect(table, "test.html"));
+ assertEquals(MediaType.TEXT_HTML, detect(table, null));
+ assertEquals(MediaType.TEXT_HTML,
detect("<p>x</p>".getBytes(US_ASCII), "test.xml"));
+ assertEquals(MediaType.TEXT_HTML,
+ detect("<br />\n<b>Warning</b>: x".getBytes(US_ASCII),
"test.php"));
+ }
+
+ @Test
+ public void testAsxWithoutXmlDeclaration() throws Exception {
+ MediaType asx = MediaType.application("x-ms-asx");
+ assertDetected(asx, "<ASX version =
\"3.0\">\n<Entry>".getBytes(US_ASCII));
+ assertDetected(asx, "<Asx Version=\"3.0\"><Entry>".getBytes(US_ASCII));
+ }
+
+ @Test
+ public void testMpegTransportStream() throws Exception {
+ MediaType ts = MediaType.video("mp2t");
+ assertDetected(ts, packets(0, 188, 5));
+ assertDetected(ts, packets(4, 192, 5));
+ assertDetected(MediaType.OCTET_STREAM, packets(0, 188, 4));
+ }
+
+ @Test
+ public void testMarcWithBlankBibliographicLevel() throws Exception {
+ assertDetected(MediaType.application("marc"),
+ "01252na a2200325 4500001001300000".getBytes(US_ASCII));
+ }
+
+ @Test
+ public void testDerCrlAndCertificateAnySignatureAlgorithm() throws
Exception {
+ // v2 CRL signed with SM2 (OID 1.2.156.10197.1.501), long and short
inner lengths
+ assertDetected(MediaType.application("pkix-crl"), bytes(0x30, 0x82,
0x01, 0x17, 0x30, 0x81, 0xbe,
+ 0x02, 0x01, 0x01, 0x30, 0x0a, 0x06, 0x08, 0x2a, 0x81, 0x1c,
0xcf, 0x55, 0x01, 0x83, 0x75));
+ assertDetected(MediaType.application("pkix-crl"), bytes(0x30, 0x81,
0xa0, 0x30, 0x48,
+ 0x02, 0x01, 0x01, 0x30, 0x05, 0x06, 0x03, 0x2b, 0x65, 0x70));
+ // v1 CRL (no version field) from openssl ca -gencrl,
ecdsa-with-SHA256, issuer CN=T
+ byte[] crlV1 = new byte[64];
+ byte[] crlHead = {0x30, (byte) 0x81, (byte) 0x91, 0x30, 0x38, 0x30,
0x0a, 0x06, 0x08, 0x2a,
+ (byte) 0x86, 0x48, (byte) 0xce, 0x3d, 0x04, 0x03, 0x02, 0x30,
0x0c, 0x31, 0x0a, 0x30, 0x08,
+ 0x06, 0x03, 0x55, 0x04, 0x03, 0x0c, 0x01, 0x54, 0x17, 0x0d};
+ System.arraycopy(crlHead, 0, crlV1, 0, crlHead.length);
+ System.arraycopy("261105180134Z".getBytes(US_ASCII), 0, crlV1,
crlHead.length, 13);
+ assertDetected(MediaType.application("pkix-crl"), crlV1);
+ // v3 certificate, then serialNumber
+ assertDetected(MediaType.parse("application/x-x509-cert;format=der"),
bytes(0x30, 0x82, 0x0a, 0x5d,
+ 0x30, 0x82, 0x0a, 0x0c, 0xa0, 0x03, 0x02, 0x01, 0x02, 0x02,
0x11, 0x00));
+ // v1 certificate with serial 1 opens like a v2 CRL; its validity
tells them apart
+ byte[] v1 = new byte[128];
+ byte[] head = {0x30, (byte) 0x82, 0x02, 0x7c, 0x30, (byte) 0x81,
(byte) 0xe5, 0x02, 0x01, 0x01,
+ 0x30, 0x0d, 0x06, 0x09, 0x2a, (byte) 0x86, 0x48, (byte) 0x86,
(byte) 0xf7, 0x0d, 0x01, 0x01,
+ 0x0b, 0x05, 0x00, 0x30, 0x0b, 0x31, 0x09, 0x30, 0x07, 0x06,
0x03, 0x55, 0x04, 0x03, 0x0c, 0x00,
+ 0x30, 0x1e, 0x17, 0x0d};
+ System.arraycopy(head, 0, v1, 0, head.length);
+
System.arraycopy("250101000000Z\u0017\r260101000000Z".getBytes(US_ASCII), 0,
v1, head.length, 28);
+ assertDetected(MediaType.parse("application/x-x509-cert;format=der"),
v1);
+ // RFC 3161 reply: granted, "Operation Okay", then the signedData token
+ byte[] tsr = bytes(0x30, 0x82, 0x0a, 0xb5, 0x30, 0x15, 0x02, 0x01,
0x00, 0x30, 0x10, 0x0c, 0x0e);
+ System.arraycopy("Operation Okay".getBytes(US_ASCII), 0, tsr, 13, 14);
+ byte[] token = {0x30, (byte) 0x82, 0x0a, (byte) 0x9a, 0x06, 0x09,
0x2a, (byte) 0x86, 0x48, (byte) 0x86,
+ (byte) 0xf7, 0x0d, 0x01, 0x07, 0x02};
+ System.arraycopy(token, 0, tsr, 27, token.length);
+ assertDetected(MediaType.application("timestamp-reply"), tsr);
+ // PKCS#10 request: version 0, then a Name, not an AlgorithmIdentifier
+ assertDetected(MediaType.OCTET_STREAM, bytes(0x30, 0x82, 0x01, 0x0a,
0x30, 0x81, 0xb1,
+ 0x02, 0x01, 0x00, 0x30, 0x46, 0x31, 0x0b));
+ }
+
+ private static byte[] packets(int first, int size, int count) {
+ byte[] b = new byte[1024];
+ for (int i = 0; i < count; i++) {
+ b[first + i * size] = 0x47;
+ }
+ return b;
+ }
+
+ @Test
+ public void testSubRipLineEndings() throws Exception {
+ MediaType srt = MediaType.application("x-subrip");
+ for (String eol : new String[]{"\n", "\r", "\r\n"}) {
+ assertDetected(srt, ("1" + eol + "00:00:01,000 --> 00:00:02,000" +
eol + "Hi").getBytes(US_ASCII));
+ }
+ }
+
+ private static byte[] header(int offset, String ascii) {
+ byte[] header = new byte[128];
+ byte[] value = ascii.getBytes(US_ASCII);
+ System.arraycopy(value, 0, header, offset, value.length);
+ return header;
+ }
+
+ private static byte[] bytes(int... values) {
+ byte[] bytes = new byte[64];
+ for (int i = 0; i < values.length; i++) {
+ bytes[i] = (byte) values[i];
+ }
+ return bytes;
+ }
+
private static byte[] tgaHeader(int idLength, int colourMapType, int
imageType,
int colourMapEntrySize, int pixelDepth) {
byte[] header = new byte[64];
diff --git
a/tika-core/src/test/java/org/apache/tika/mime/MimeTypesLintTest.java
b/tika-core/src/test/java/org/apache/tika/mime/MimeTypesLintTest.java
new file mode 100644
index 0000000000..43e225bec2
--- /dev/null
+++ b/tika-core/src/test/java/org/apache/tika/mime/MimeTypesLintTest.java
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.mime;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.ByteArrayInputStream;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+
+import org.junit.jupiter.api.Test;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+import org.w3c.dom.NodeList;
+
+import org.apache.tika.utils.XMLReaderUtils;
+
+/**
+ * Rules in tika-mimetypes.xml that load fine but can never match as written.
+ * Checks the raw XML: once decoded, a mistyped value is indistinguishable
from an intended one.
+ */
+public class MimeTypesLintTest {
+
+ @Test
+ public void testDefaultMimeTypes() throws Exception {
+ try (InputStream is =
MimeTypesReader.class.getResourceAsStream("tika-mimetypes.xml")) {
+ List<String> problems = lint(is);
+ assertTrue(problems.isEmpty(), String.join("\n", problems));
+ }
+ }
+
+ @Test
+ public void testLintCatchesEachProblem() throws Exception {
+ String xml = "<mime-info>" +
+ "<mime-type type=\"application/x-a\"><magic>" +
+ "<match value=\"0x310D0A13030\" type=\"string\"
offset=\"0\"/>" +
+ "<match value=\"0x31zz\" type=\"string\" offset=\"0\"/>" +
+ "<match value=\"(?i)<html[\\s>]\" type=\"regex\"
offset=\"0\"/>" +
+ "<match value=\"a\\db\" type=\"string\" offset=\"0\"/>" +
+ "<match value=\"0x0102\" mask=\"0xFF\" type=\"string\"
offset=\"0\"/>" +
+ "<match value=\"x\" type=\"string\" offset=\"8:4\"/>" +
+ "<match value=\"x\" type=\"string\" offset=\"0:65536\"/>" +
+ "<match value=\"\\xd2\\x04\" type=\"regex\" offset=\"0\"/>" +
+ "</magic><sub-class-of
type=\"application/x-undefined\"/></mime-type>" +
+ "<mime-type type=\"application/x-b\">" +
+ "<alias type=\"application/x-b-alias\"/><magic>" +
+ "<match value=\"(?i)<html[\\\\s>]\" type=\"regex\"
offset=\"0\"/>" +
+ "<match value=\"\\\\xd2\\x04\" type=\"regex\" offset=\"0\"/>" +
+ "<match value=\"\\x00\\n\\\\a\\101\" type=\"string\"
offset=\"0\"/>" +
+ "</magic><sub-class-of type=\"application/x-a\"/></mime-type>"
+
+ "<mime-type type=\"application/x-c\"><sub-class-of
type=\"application/x-b-alias\"/>" +
+ "</mime-type></mime-info>";
+ List<String> problems = lint(new
ByteArrayInputStream(xml.getBytes(UTF_8)));
+ assertEquals(9, problems.size(), String.join("\n", problems));
+ for (String p : problems) {
+ assertTrue(p.startsWith("application/x-a "), p);
+ }
+ }
+
+ static List<String> lint(InputStream is) throws Exception {
+ Document doc = XMLReaderUtils.buildDOM(is);
+ NodeList types = doc.getElementsByTagName("mime-type");
+ Set<String> defined = new HashSet<>();
+ for (int i = 0; i < types.getLength(); i++) {
+ Element type = (Element) types.item(i);
+ defined.add(type.getAttribute("type").toLowerCase(Locale.ROOT));
+ for (Element alias : children(type, "alias")) {
+
defined.add(alias.getAttribute("type").toLowerCase(Locale.ROOT));
+ }
+ }
+ List<String> problems = new ArrayList<>();
+ for (int i = 0; i < types.getLength(); i++) {
+ Element type = (Element) types.item(i);
+ String name = type.getAttribute("type");
+ for (Element parent : children(type, "sub-class-of")) {
+ String p = parent.getAttribute("type");
+ if (!defined.contains(p.toLowerCase(Locale.ROOT))) {
+ problems.add(name + " sub-class-of undefined type " + p);
+ }
+ }
+ NodeList matches = type.getElementsByTagName("match");
+ for (int j = 0; j < matches.getLength(); j++) {
+ lintMatch(name, (Element) matches.item(j), problems);
+ }
+ }
+ return problems;
+ }
+
+ private static void lintMatch(String name, Element match, List<String>
problems) {
+ String kind = match.hasAttribute("type") ? match.getAttribute("type")
: "string";
+ String value = match.getAttribute("value");
+ String mask = match.getAttribute("mask");
+ String where = name + " match '" + value + "': ";
+ boolean bytewise = kind.startsWith("string") || kind.equals("regex");
+ if (value.startsWith("0x")) {
+ String hex = value.substring(2);
+ if (!hex.matches("[0-9A-Fa-f]*")) {
+ problems.add(where + "non-hex digit");
+ } else if (bytewise && hex.length() % 2 != 0) {
+ // decoding takes digit pairs and drops the last one
+ problems.add(where + "odd number of hex digits");
+ }
+ if (mask.startsWith("0x") && mask.length() != value.length()) {
+ problems.add(where + "mask and value differ in length");
+ }
+ } else {
+ for (int k = 0; k + 1 < value.length(); k++) {
+ if (value.charAt(k) != '\\') {
+ continue;
+ }
+ char c = value.charAt(++k);
+ if (c == 'x' && kind.equals("regex") && k + 2 < value.length()
+ && Integer.parseInt(value.substring(k + 1, k + 3), 16)
>= 0x80) {
+ // the pattern is compiled from UTF-8, so a raw high byte
becomes U+FFFD
+ problems.add(where + "\\x" + value.substring(k + 1, k + 3)
+ " in a regex needs \\\\x");
+ }
+ if (c == '\\' || c == 'x' || c == 'n' || c == 'r' || c == 't'
|| Character.isDigit(c)) {
+ continue;
+ }
+ // the decoder keeps only c: "\s" is "s", "\." is "."; regex
escapes need "\\"
+ if (kind.equals("regex") || Character.isLetter(c)) {
+ problems.add(where + "escape \\" + c + " decodes to a bare
'" + c + "'");
+ }
+ }
+ }
+ String offset = match.getAttribute("offset");
+ if (!offset.isEmpty()) {
+ String[] range = offset.split(":");
+ int start = Integer.parseInt(range[0].trim());
+ int end = range.length > 1 ? Integer.parseInt(range[1].trim()) :
start;
+ if (end < start) {
+ problems.add(where + "offset range ends before it starts");
+ } else if (end >= 64 * 1024) {
+ problems.add(where + "offset beyond the bytes MimeTypes
reads");
+ }
+ }
+ }
+
+ private static List<Element> children(Element parent, String tag) {
+ List<Element> out = new ArrayList<>();
+ NodeList nodes = parent.getElementsByTagName(tag);
+ for (int i = 0; i < nodes.getLength(); i++) {
+ out.add((Element) nodes.item(i));
+ }
+ return out;
+ }
+}
diff --git
a/tika-core/src/test/java/org/apache/tika/mime/MimeTypesReaderTest.java
b/tika-core/src/test/java/org/apache/tika/mime/MimeTypesReaderTest.java
index 6dd4d5d624..cee8d0497a 100644
--- a/tika-core/src/test/java/org/apache/tika/mime/MimeTypesReaderTest.java
+++ b/tika-core/src/test/java/org/apache/tika/mime/MimeTypesReaderTest.java
@@ -426,6 +426,41 @@ public class MimeTypesReaderTest {
assertThrows(IllegalArgumentException.class, () ->
MimeTypes.getDefaultMimeTypes(new CustomClassLoader()));
}
+ @Test
+ public void testDuplicateDefinitionFailsLoad() {
+ String dup = "<mime-info><mime-type type=\"application/x-dup\"><glob
pattern=\"*.dup\"/>"
+ + "</mime-type><mime-type
type=\"application/x-dup\"/></mime-info>";
+ MimeTypeException e = assertThrows(MimeTypeException.class,
+ () -> MimeTypesFactory.create(stream(dup)));
+ assertTrue(e.getCause().getMessage().contains("application/x-dup"),
e.getCause().getMessage());
+
+ // a second definition through an alias is the same type
+ String viaAlias = "<mime-info><mime-type type=\"application/x-dup\">"
+ + "<alias type=\"application/x-dup-alias\"/></mime-type>"
+ + "<mime-type type=\"application/x-dup-alias\"/></mime-info>";
+ assertThrows(MimeTypeException.class, () ->
MimeTypesFactory.create(stream(viaAlias)));
+ }
+
+ @Test
+ public void testLaterFileMayExtendType() throws Exception {
+ String core = "<mime-info><mime-type type=\"application/x-dup\"><glob
pattern=\"*.dup\"/>"
+ + "</mime-type></mime-info>";
+ String custom = "<mime-info><mime-type
type=\"application/x-dup\"><glob pattern=\"*.dup2\"/>"
+ + "</mime-type></mime-info>";
+ MimeTypes types = MimeTypesFactory.create(stream(core),
stream(custom));
+ assertEquals(List.of(".dup", ".dup2"),
types.forName("application/x-dup").getExtensions());
+ }
+
+ @Test
+ public void testDefaultMimeTypesHaveNoDuplicates() throws Exception {
+ // fails the load, so reaching here is the assertion
+ assertNotNull(MimeTypesFactory.create("tika-mimetypes.xml"));
+ }
+
+ private static ByteArrayInputStream stream(String xml) {
+ return new ByteArrayInputStream(xml.getBytes(StandardCharsets.UTF_8));
+ }
+
private static class CustomClassLoader extends ClassLoader {
}
}
diff --git
a/tika-core/src/test/java/org/apache/tika/mime/NicheFormatDetectionTest.java
b/tika-core/src/test/java/org/apache/tika/mime/NicheFormatDetectionTest.java
new file mode 100644
index 0000000000..cac379a631
--- /dev/null
+++ b/tika-core/src/test/java/org/apache/tika/mime/NicheFormatDetectionTest.java
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.mime;
+
+import static java.nio.charset.StandardCharsets.ISO_8859_1;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.IOException;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.tika.io.TikaInputStream;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.parser.ParseContext;
+
+/**
+ * Formats found as application/octet-stream in Common Crawl; headers are
synthesized
+ * from the shapes seen there.
+ */
+public class NicheFormatDetectionTest {
+
+ private static final MimeTypes MIME_TYPES =
MimeTypes.getDefaultMimeTypes();
+
+ @Test
+ public void testFec() throws Exception {
+ MediaType fec = MediaType.application("x-fec");
+ assertDetected(fec, "HDR\u001cFEC\u001c8.3\u001cCMDI CRIMSON
FILER\u001c5.0\u001c\u001c\n");
+ assertTrue(MIME_TYPES.getMediaTypeRegistry().isSpecializationOf(fec,
MediaType.TEXT_PLAIN));
+ }
+
+ @Test
+ public void testGarminGdb() throws Exception {
+ assertDetected(MediaType.application("x-garmin-gdb"),
+ "MsRcf\u0000\u0002\u0000\u0000\u0000Dm\u0000");
+ }
+
+ @Test
+ public void testGettextCatalog() throws Exception {
+ MediaType mo = MediaType.application("x-gettext-translation");
+ assertDetected(mo,
"Þ\u0012\u0004\u0095\u0000\u0000\u0000\u0000\u008e\u0005\u0000\u0000");
+ assertDetected(mo,
"\u0095\u0004\u0012Þ\u0000\u0000\u0000\u0000\u0000\u0000\u0005\u008e");
+ // minor revision 1: msgfmt output with system-dependent strings
+ assertDetected(mo, "Þ\u0012\u0004\u0095\u0001\u0000\u0000\u0000");
+ assertDetected(mo, "\u0095\u0004\u0012Þ\u0000\u0000\u0000\u0001");
+ // unknown revision
+ assertDetected(MediaType.OCTET_STREAM,
"Þ\u0012\u0004\u0095\u0000\u0007\u0000\u0000");
+ }
+
+ @Test
+ public void testMpqAndWarcraft() throws Exception {
+ MediaType mpq = MediaType.application("x-mpq");
+ assertDetected(mpq, "MPQ\u001a \u0000\u0000\u0000");
+ assertDetected(mpq, "MPQ\u001b\u0000\u0002\u0000\u0000");
+ assertDetected(MediaType.OCTET_STREAM,
"MPQ\u001a\u0099\u0000\u0000\u0000");
+ assertDetected(MediaType.application("x-warcraft3-map"),
"HM3W\u0000\u0000\u0000\u0000Map");
+ assertDetected(MediaType.application("x-warcraft3-replay"),
+ "Warcraft III recorded game\u001a\u0000");
+ }
+
+ @Test
+ public void testBinaryDiffs() throws Exception {
+ MediaType vcdiff = MediaType.application("x-vcdiff");
+ assertDetected(vcdiff, "ÖÃÄ\u0000\u0005");
+ assertDetected(vcdiff, "ÖÃÄ\u0000\u0000");
+ // open-vcdiff SDCH interleaved format
+ assertDetected(vcdiff, "ÖÃÄS\u0000");
+ // undefined Hdr_Indicator bits
+ assertDetected(MediaType.OCTET_STREAM, "ÖÃÄ\u0000\u0010");
+ assertDetected(MediaType.application("x-xdelta"),
"%XDZ004%\u0000\u0000");
+ }
+
+ @Test
+ public void testSid() throws Exception {
+ MediaType sid = MediaType.audio("prs.sid");
+ assertDetected(sid, "RSID\u0000\u0002\u0000|");
+ assertDetected(sid, "PSID\u0000\u0001\u0000v");
+ assertDetected(MediaType.OCTET_STREAM, "PSID\u0000\t\u0000v");
+ }
+
+ @Test
+ public void testPaintShopPro() throws Exception {
+ assertDetected(MediaType.image("x-paintshoppro"),
+ "Paint Shop Pro Image
File\n\u001a\u0000\u0000\u0000\u0000\u0000");
+ }
+
+ @Test
+ public void testElastoManiaLevel() throws Exception {
+ MediaType lev = MediaType.application("x-elasto-mania-level");
+ assertDetected(lev, "POT14\u0000\u0000");
+ assertDetected(lev, "POT06\u0000\u0000");
+ }
+
+ @Test
+ public void testBinaryPgpKeys() throws Exception {
+ MediaType keys = MediaType.application("pgp-keys");
+ // public key, 2-byte length, RSA
+ assertDetected(keys, "\u0099\u0001\r\u0004ABCD\u0001");
+ // secret key, 1-byte length, EdDSA
+ assertDetected(keys, "\u0094X\u0004ABCD\u0016");
+ assertDetected(MediaType.OCTET_STREAM,
"\u0099\u0001\r\u0004ABCD\u0007");
+ // new-format v6 Ed25519 public and secret keys, as in RFC 9580 A.3
and A.4
+ assertDetected(keys, "Æ*\u0006c\u0087\u007fã\u001b\u0000\u0000\u0000
");
+ assertDetected(keys, "ÅK\u0006c\u0087\u007fã\u001b\u0000\u0000\u0000
");
+ // new-format v4 RSA public key, 2-byte length
+ assertDetected(keys, "ÆÁ\r\u0004ABCD\u0001");
+ assertDetected(MediaType.OCTET_STREAM, "Æ*\u0006c\u0087\u007fã\u0007");
+ }
+
+ @Test
+ public void testRetroArchivesAndGames() throws Exception {
+ assertDetected(MediaType.application("x-stuffit"),
"SIT!\u0000\u0003\u0000$abrLau");
+ assertDetected(MediaType.application("vnd.symbian.install"), "z\u001a
\u0010\u0000\u0000\u0000\u0000");
+ assertDetected(MediaType.application("x-ms-reader"),
"ITOLITLS\u0001\u0000\u0000\u0000");
+ assertDetected(MediaType.application("x-lrzip"), "LRZI\u0000\u0006");
+ assertDetected(MediaType.application("x-glulx"),
"Glul\u0000\u0003\u0001\u0000");
+ assertDetected(MediaType.video("x-flc"),
"abcd\u0012¯\u0016\u0000ABCD\b\u0000");
+ assertDetected(MediaType.video("x-fli"),
"abcd\u0011¯\u0016\u0000ABCD\b\u0000");
+ }
+
+ @Test
+ public void testAudioModulesAndApe() throws Exception {
+ assertDetected(MediaType.audio("x-ape"), "MAC \u0096\u000f4\u0000");
+ assertDetected(MediaType.audio("x-s3m"), "unknown" +
"\u0000".repeat(21) + "\u001a\u0010"
+ + "\u0000".repeat(14) + "SCRM");
+ }
+
+ @Test
+ public void testMusicAndAudioFormats() throws Exception {
+ assertDetected(MediaType.application("x-nwc"), "[NWZ]\u0000xÚ");
+ assertDetected(MediaType.application("x-nwc"), "[NoteWorthy
ArtWare]\u0000");
+ assertDetected(MediaType.application("x-finale"), "ENIGMA BINARY
FILE\u0000");
+ assertDetected(MediaType.audio("x-rex2"), "CAT
\u0000\t\u0000\u0000REX2HEAD");
+ assertDetected(MediaType.audio("x-mfi"),
"melo\u0000\u0000\u0011#\u0000]\u0001\u0001\u0004vers");
+ assertDetected(MediaType.OCTET_STREAM,
"melody\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000");
+ }
+
+ @Test
+ public void testGameAndConsoleFormats() throws Exception {
+ assertDetected(MediaType.application("x-gbx"), "GBX\u0006\u0000BUCR");
+ assertDetected(MediaType.application("x-gbx"), "GBX\u0003\u0000BUC");
+ assertDetected(MediaType.OCTET_STREAM, "GBX\t\u0000BUCR");
+ assertDetected(MediaType.application("x-ps3-package"),
"\u007fPKG\u0080\u0000\u0000\u0001");
+ assertDetected(MediaType.application("x-ps3-package"),
"\u007fPKG\u0000\u0000\u0000\u0001");
+ assertDetected(MediaType.application("x-ps3-theme"),
"P3TF\u0000\u0000\u0001\u0010");
+ assertDetected(MediaType.application("x-trainz-cdp"),
"ACS$\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000");
+ assertDetected(MediaType.application("x-spectrum-rzx"),
"RZX!\u0000\f");
+ assertDetected(MediaType.image("x-quake-sprite"),
"IDSP\u0002\u0000\u0000\u0000");
+ }
+
+ @Test
+ public void testArchivesAndPackages() throws Exception {
+ assertDetected(MediaType.application("x-nufx"), "NõFél姳");
+ assertDetected(MediaType.application("x-lzx"),
"LZX\u0000\f\u0000\n\u0004");
+ assertDetected(MediaType.application("x-mikrotik-npk"), "\u001eñк");
+ assertDetected(MediaType.application("x-clamav-diff"),
"ClamAV-Diff:119:526:");
+ assertDetected(MediaType.application("x-clamav-database"),
"ClamAV-VDB:01 Jan 2020");
+ }
+
+ @Test
+ public void testCasioFxInterface() throws Exception {
+ StringBuilder fxi = new StringBuilder("20");
+ for (char c : "\u001fFX-INTERFACE - YELLOW COMPUTING".toCharArray()) {
+ fxi.append((char) (c ^ 0xe7));
+ }
+ assertDetected(MediaType.application("x-casio-fxi"), fxi.toString());
+ }
+
+ @Test
+ public void testModelAndCadFormats() throws Exception {
+ assertDetected(MediaType.parse("model/gltf-binary"),
"glTF\u0002\u0000\u0000\u0000ð@\u001c\u0000");
+ assertDetected(MediaType.application("x-blender"), "BLENDER-v300REND");
+ assertDetected(MediaType.application("x-blender"), "BLENDER_v244REND");
+ // 5.0 header layout
+ assertDetected(MediaType.application("x-blender"),
"BLENDER17-01v0500REND");
+ assertDetected(MediaType.application("x-fbx"), "Kaydara FBX Binary
\u0000\u001a\u0000");
+ assertDetected(MediaType.application("x-jw-cad"),
"JwwData.¼\u0002\u0000\u0000");
+ assertDetected(MediaType.application("x-fastcad"), "FCW (FastCAD for
Windows) 6.20.0\r\n\u001a");
+ assertDetected(MediaType.application("x-source-bsp"),
"VBSP\u0014\u0000\u0000\u0000");
+ assertDetected(MediaType.OCTET_STREAM, "VBSP\u0002\u0000\u0000\u0000");
+ }
+
+ @Test
+ public void testGameAudioAndDataFormats() throws Exception {
+ assertDetected(MediaType.application("x-unreal-uz"),
+ "Ò\u0004\u0000\u0000\u0011FCommandoReg.utx\u0000¡É");
+ assertDetected(MediaType.audio("x-soundfont"),
"RIFF\u0000\u0010\u0000\u0000sfbkLIST");
+ assertDetected(MediaType.application("x-zsync"), "zsync:
0.6.2\nFilename: x.tcz\n");
+ assertDetected(MediaType.application("x-wfdb-annotation"),
+ "\u0000X\u0017ü## time resolution: 250\u0000\u0000");
+ }
+
+ @Test
+ public void testPrinterJobs() throws Exception {
+ MediaType pcl = MediaType.application("vnd.hp-pcl");
+ MediaType pclxl = MediaType.application("vnd.hp-pclxl");
+ assertDetected(pclxl, "\u001b%-12345X@PJL COMMENT x\r\n@PJL ENTER
LANGUAGE = PCLXL\r\n");
+ assertDetected(pclxl, ") HP-PCL XL;3;0;Comment");
+ assertDetected(pcl, "\u001b%-12345X@PJL JOB\r\n@PJL ENTER
LANGUAGE=PCL\r\n\u001bE");
+ assertDetected(pcl, "\u001b%-12345X@PJL ENTER
LANGUAGE=PCL\u001bE\u001b&l0O");
+ // driver headers can put the command past 1K
+ String longJob = "\u001b%-12345X" + "@PJL COMMENT
padding\r\n".repeat(80) + "@PJL ENTER LANGUAGE=PCLXL\r\n";
+ try (TikaInputStream tis =
TikaInputStream.get(longJob.getBytes(ISO_8859_1))) {
+ assertEquals(pclxl, MIME_TYPES.detect(tis, new Metadata(), new
ParseContext()));
+ }
+ assertDetected(pcl, "\u001b%-12345X\u001bE\u001b&u600D");
+ assertDetected(pcl, "\u001bE\u001b&l0O\u001b&l0H");
+ assertDetected(MediaType.application("postscript"),
+ "\u001b%-12345X@PJL\r\n@PJL ENTER LANGUAGE =
POSTSCRIPT\r\n%!PS");
+ }
+
+ @Test
+ public void testAfpDumpsAndMetafiles() throws Exception {
+ assertDetected(MediaType.application("vnd.ibm.modcap"),
"Z\u0000\u0010\u0000\u0000");
+ assertDetected(MediaType.application("x-dmp"),
"MDMP\u0093§\u0000\u0000");
+ assertDetected(MediaType.application("x-windows-kernel-dump"),
"PAGEDU64DU64");
+ assertDetected(MediaType.image("x-svm"), "VCLMTF\u0001\u0000");
+
assertDetected(MediaType.parse("application/vnd.lotus-1-2-3;version=97+9.x"),
+ "\u0000\u0000\u001a\u0000\u0005\u0010\u0004\u0000");
+ }
+
+ @Test
+ public void testIdentifiedCrawlClusters() throws Exception {
+ assertDetected(MediaType.application("x-promethean-flipchart"),
+ " Bamboo
\u0007\u0000\u0006\u0000\u0000\u00001\u0000.\u00000\u0000");
+ StringBuilder casio = new StringBuilder();
+ for (char c : "USBPower1\u0000\u0010\u0000\u0010\u0000".toCharArray())
{
+ casio.append((char) (~c & 0xff));
+ }
+ assertDetected(MediaType.application("x-casio-g1m"), casio.toString());
+ assertDetected(MediaType.application("x-haihaisoft-drm"),
+
"XDRM\u0000\u0000\u0000\n\u0000\u0003HEAD\u0000\u0000\u0000Ñ\u0000\u0003<XDRMHEADER>");
+ assertDetected(MediaType.application("x-navitel-map"),
+
"PMGC\u0005\u0000Ò\u000f\u0004\u0000\f\u0000\u0000\u0000\u0000\u0000"
+ +
"\u0084Sÿ\u0018\u0084\u0000\u0000\u0000\u0084\u0000\u0000\u0000");
+ }
+
+ @Test
+ public void testDeveloperArtifacts() throws Exception {
+ assertDetected(MediaType.application("x-gcov-data"),
"adcg*504qc\u0000\u0000");
+ assertDetected(MediaType.application("x-gcov-notes"),
"oncg*204\u0000\u0000");
+ assertDetected(MediaType.application("x-python-bytecode"),
"\u0003ó\r\nSc\u0000\u0000");
+ assertDetected(MediaType.application("x-python-bytecode"),
"U\r\r\n\u0000\u0000\u0000\u0000");
+ // CPython magic numbers: 3.0 is 3131, 3.14 is 3627, 3.15 starts at
3650
+ // 3.0: mtime, then the code object; 3.6: mtime and source size, then
a FLAG_REF code object
+ assertDetected(MediaType.application("x-python-bytecode"),
";\f\r\nABCDc\u0000\u0000");
+ assertDetected(MediaType.application("x-python-bytecode"),
"3\r\r\nABCD\u0010\u0000\u0000\u0000ã");
+ // text with a one-character first line and CR CR LF endings
+ assertDetected(MediaType.OCTET_STREAM, " \r\r\n \r\r\n");
+ assertDetected(MediaType.application("x-python-bytecode"),
"+\u000e\r\n\u0000\u0000\u0000\u0000");
+ assertDetected(MediaType.application("x-python-bytecode"),
"B\u000e\r\n\u0000\u0000\u0000\u0000");
+ assertDetected(MediaType.application("x-python-pickle"),
"\u0080\u0002}q\u0000(U\tcollector");
+ assertDetected(MediaType.application("x-python-pickle"),
"\u0080\u0004\u0095\u0010\u0000\u0000\u0000");
+ assertDetected(MediaType.application("x-lucene-index"),
"?×l\u0017\u0013Lucene62SegmentInfo");
+ assertDetected(MediaType.application("x-git-pack"),
"PACK\u0000\u0000\u0000\u0002\u0000\u0000\u0000\u0003");
+ assertDetected(MediaType.OCTET_STREAM,
"PACK\u0000\u0000\u0000\u0001\u0000\u0000\u0002-");
+ }
+
+ @Test
+ public void testSystemDataFormats() throws Exception {
+ assertDetected(MediaType.application("tzif"),
"TZif2\u0000\u0000\u0000\u0000\u0000");
+ assertDetected(MediaType.application("x-mozlz4"),
"mozLz40\u0000]\u0000\u0000\u0000{\"type\"");
+ assertDetected(MediaType.image("jxr"), "II¼\u0001 \u0000\u0000\u0000");
+ assertDetected(MediaType.application("x-intel-aml"),
+ "SSDTÈ\u0000\u0000\u0000\u0001\u008eBOCHS BXPCSSDT");
+ }
+
+ @Test
+ public void testScienceFormats() throws Exception {
+ assertDetected(MediaType.application("x-nexrad-level3"),
+ "SDUS63 KSGF 280002\r\r\nDODSGF\r\r\n\u0000®L");
+ assertDetected(MediaType.OCTET_STREAM, "SDUS63 KSGF
280002\r\nDODSGF\r\n");
+ assertDetected(MediaType.application("x-bigbed"),
"ëò\u0089\u0087\u0004\u0000\b\u0000");
+ assertDetected(MediaType.application("x-bigwig"),
"&ü\u008f\u0088\u0004\u0000\n\u0000");
+ assertDetected(MediaType.OCTET_STREAM,
"&ü\u008f\u0088\t\u0000\n\u0000");
+ assertDetected(MediaType.application("x-ucsc-2bit"),
"C'A\u001a\u0000\u0000\u0000\u0000î");
+ assertDetected(MediaType.application("x-ucsc-2bit"),
"C'A\u001a\u0001\u0000\u0000\u0000s");
+ assertDetected(MediaType.application("x-bam"),
"BAM\u0001j\u0006\u0000\u0000@HD\tVN:1.0");
+ assertDetected(MediaType.application("x-bam-index"),
"BAI\u0001\u0016\u0000\u0000\u0000");
+ assertDetected(MediaType.application("x-osm-pbf"),
"\u0000\u0000\u0000\u000e\n\tOSMHeader\u0018");
+ assertDetected(MediaType.application("x-ligo-frame"),
"IGWD\u0000\b\u0001\u0002");
+ }
+
+ @Test
+ public void testDiskAndFirmwareImages() throws Exception {
+ assertDetected(MediaType.application("x-qemu-disk"),
"QFIû\u0000\u0000\u0000\u0003");
+ assertDetected(MediaType.OCTET_STREAM, "QFIû\u0000\u0000\u0000\u0007");
+ assertDetected(MediaType.application("x-imagedisk"), "IMD 1.18:
12/07/2013 8:49:58\r\n\u001a");
+ assertDetected(MediaType.application("x-mame-chd"),
"MComprHD\u0000\u0000\u0000|\u0000\u0000\u0000\u0005");
+ assertDetected(MediaType.application("x-device-tree-blob"),
+
"Ð\rþí\u0000µCè\u0000\u0000\u00008\u0000µ?ü\u0000\u0000\u0000(\u0000\u0000\u0000\u0011");
+ assertDetected(MediaType.OCTET_STREAM,
+
"Ð\rþí\u0000µCè\u0000\u0000\u00008\u0000µ?ü\u0000\u0000\u0000(\u0000\u0000\u0000\u0002");
+ StringBuilder uimage = new StringBuilder("'\u0005\u0019V");
+
uimage.append("\u0000".repeat(24)).append("\u0005\u0002\u0002\u0003DD-WRT");
+ assertDetected(MediaType.application("x-uboot-image"),
uimage.toString());
+ assertDetected(MediaType.application("x-ubi-image"),
"UBI#\u0001\u0000\u0000\u0000");
+ assertDetected(MediaType.application("x-amiga-executable"),
+
"\u0000\u0000\u0003ó\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0002\u0000\u0000\u0000\u0000");
+ }
+
+ @Test
+ public void testMediaFormats() throws Exception {
+ assertDetected(MediaType.image("vnd.ms-dds"), "DDS
|\u0000\u0000\u0000\u0007\u0010\n\u0000");
+ assertDetected(MediaType.audio("x-ym"),
"YM6!LeOnArD!\u0000\u0000\u001b§");
+ assertDetected(MediaType.audio("x-ym"), "YM3!\u0000\u0000\u0001@");
+ assertDetected(MediaType.application("x-vst-preset"),
"CcnK\u0000\u0000\u0000dFxCk\u0000\u0000\u0000\u0001");
+ assertDetected(MediaType.OCTET_STREAM, "CcnK\u0000\u0000\u0000dABCD");
+ assertDetected(MediaType.application("x-jw-cad"), "jw_cad(c)data....");
+ }
+
+ @Test
+ public void testEstablishedNamesAreAliases() throws Exception {
+ String[][] pairs = {
+ {"application/x-palmdoc", "application/x-aportisdoc"},
+ {"application/x-acpi-table", "application/x-intel-aml"},
+ {"audio/x-sfbk", "audio/x-soundfont"},
+ {"application/x-ms-dmp", "application/x-windows-kernel-dump"},
+ {"application/x-lz4+json", "application/x-mozlz4"},
+ {"application/x-git", "application/x-git-pack"},
+ {"application/x-clamav", "application/x-clamav-database"},
+ {"image/x-dds", "image/vnd.ms-dds"}};
+ for (String[] pair : pairs) {
+ assertEquals(MediaType.parse(pair[1]),
MIME_TYPES.forName(pair[0]).getType(), pair[0]);
+ }
+ }
+
+ @Test
+ public void testSharedExtensionsAreNotClaimed() throws Exception {
+ // .atr Atari disks, .bsp Quake/GoldSrc maps, .lev other games,
.flipchart Wacom
+ String[][] cases = {
+ {"x.atr", "application/x-wfdb-annotation"},
+ {"x.bsp", "application/x-source-bsp"},
+ {"x.lev", "application/x-elasto-mania-level"},
+ {"x.flipchart", "application/x-promethean-flipchart"}};
+ for (String[] c : cases) {
+ assertNotEquals(MediaType.parse(c[1]),
MIME_TYPES.getMimeType(c[0]).getType(), c[0]);
+ }
+ }
+
+ private static void assertDetected(MediaType expected, String latin1)
throws IOException {
+ byte[] bytes = new byte[64];
+ byte[] header = latin1.getBytes(ISO_8859_1);
+ System.arraycopy(header, 0, bytes, 0, header.length);
+ try (TikaInputStream tis = TikaInputStream.get(bytes)) {
+ assertEquals(expected, MIME_TYPES.detect(tis, new Metadata(), new
ParseContext()));
+ }
+ }
+}
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/parser/ParsingReaderTest.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/parser/ParsingReaderTest.java
index 81b159d431..7f5ec8dcdf 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/parser/ParsingReaderTest.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/java/org/apache/tika/parser/ParsingReaderTest.java
@@ -56,7 +56,7 @@ public class ParsingReaderTest extends TikaTest {
@Test
public void testXML() throws Exception {
- String data = "<p>test <span>content</span></p>";
+ String data = "<doc>test <span>content</span></doc>";
InputStream stream = new ByteArrayInputStream(data.getBytes(UTF_8));
Reader reader = new ParsingReader(stream, "test.xml");
assertEquals(' ', (char) reader.read());
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-cad-module/pom.xml
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-cad-module/pom.xml
index 6df15881e4..3fa98cf50b 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-cad-module/pom.xml
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-cad-module/pom.xml
@@ -54,6 +54,15 @@
<build>
<plugins>
+ <plugin>
+ <groupId>org.apache.rat</groupId>
+ <artifactId>apache-rat-plugin</artifactId>
+ <configuration>
+ <inputExcludes>
+ <inputExclude>src/test/resources/test-documents/*</inputExclude>
+ </inputExcludes>
+ </configuration>
+ </plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
