[
https://issues.apache.org/jira/browse/TIKA-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18099732#comment-18099732
]
Tim Allison edited comment on TIKA-4794 at 7/28/26 12:39 PM:
-------------------------------------------------------------
Looking at this a bit more... How's this?
The key part is that we'll add a legacy key metadata mapper that should help
people migrate. Where the underlying patterns have changed, we can't do
anything... wordperfect->quattropro. But we can document known limitations. We
can assess the success and find areas for improvement on our 1.2m corpus.
Another important proposal [~dpol1] and [~krickert] is that we'll automatically
generate a json table of closed set keys and open set prefixes. This should
play very well with the grpc work?
{noformat}
PR1 — metadata-key-tk-finalization (tk: consolidation)
- Fold X-TIKA: + scattered natives (tika:chunks, imagereader:, signature:,
rendering:, tika_pg:, hasSignature/embeddedResourceType) into one kebab tk:
namespace
- resourceName finished → tk:resource-name; Content-Type-* variants →
tk:content-type-*; WordPerfect/QuattroPro prefix collision split
- Reserved-key guard reserves both tk: and legacy X-TIKA: (drops untrusted
String writes; filters/deserialize run trusted)
- Human migration table shipped (metadata-changes-4x.adoc)
- Status: swept; tika-core + tika-serialization green; full reactor build not
re-run; not committed; awaiting review PR2 — metadata-key-format-kebab (= PR1
+ format-namespace kebab)
- Kebab Tika-coined keys inside format namespaces (pdf, zip, database, pst,
dwg, access-permission, wordperfect, quattropro, msoffice,
extended-properties), ~78 keys;
standard/file/schema names left verbatim
- Status: tika-core green; built on PR1
- Open (found this session):
- rtf_meta: namespace missed entirely — prefix + suffixes still underscored
→ fix to rtf-meta: + kebab
- digest keys emit enum .name() (SHA3_256) → change DigestDef.metadataKey()
to getJavaName() → tk:digest:SHA-256/SHA3-256 (also SHA256→SHA-256); bounded
sweep (goldens + any reader
of the key) PR3 — metadata-key-registry (new module tika-metadata-schema)
- Generated + CI-gated metadata-keys.json (634 closed keys incl. 27 digest
cross-product); dependency-free SchemaGenerator classpath scan;
MetadataSchemaTest diff-gate
- Based on main; independent of PR1/PR2
- Status: committed PR4 — metadata-key-passthrough (open-namespace
discoverability + lint) — stacked on PR3
- PassthroughPrefix makes open/passthrough namespaces auto-discoverable; 6
core holders + 22 parser files; metadata-open-namespaces.json generated +
gated; MetadataCoverageTest
backstop (no key-declaring module escapes the scan)
- Status: committed/pushed (head "property prefix")
- Staged, not yet committed on top: MetadataKeyValidator (registry-driven
CLOSED/OPEN/TEMPLATE/UNKNOWN lint) + metadata-string-keys.json (bare-String
closed keys:
Content-Type/Content-*/Message-*/tika:chunks) + MetadataStringKeysTest gate +
README Cross-cutting / newly identified (target branch: format-kebab)
- Rule: no underscores in Tika-coined 4.x keys; external-standard names
verbatim. Bundle allowlist currently empty → rule = "zero underscores in the
standard registry"
- Add no-underscore lint to the registry gates (would have caught rtf_meta:)
- 3.x→4.x migration bridge:
- LegacyKeyMigrationFilter — MetadataFilter (runs trusted, so it can write
reserved tk: keys); excludeUnmapped=false; one-line enable
- Not a flat map: flat table (~131 enumerated) + tk:digest: prefix rule
(exact) + tk:exception:/tk:warn: prefix+suffix (snake↔kebab) rule
- Migration table must be generated from a ground-truth registry diff (main
vs format-kebab). Sweep scripts proven unreliable — missed the quattropro
prefix split, embedded_parser,
and rtf_meta:
- Table gated: every v4 valid via MetadataKeyValidator
- Corpus drift-gate: run MetadataKeyValidator over real parser output, assert
no UNKNOWN; needs a module above parsers (e.g. tika-parsers-integration-tests)
to avoid a reactor cycle
- Blast radius note: majority of keys unchanged (dc:, xmp*, exif:, tiff:,
Content-* verbatim); only Tika-native + Tika-coined format keys move
Recommended sequence
1. Rebase registry module (PR3) onto format-kebab
2. Run SchemaGenerator → authoritative 4.x key set
3. Add no-underscore lint → flags rtf_meta: + digest
4. Fix rtf_meta:→rtf-meta:; DigestDef → getJavaName()
5. Generate migration table from clean 3.x↔4.x diff
6. Build LegacyKeyMigrationFilter (table + rules)
7. Corpus drift-gate
{noformat}
was (Author: [email protected]):
Looking at this a bit more... How's this?
The key part is that we'll add a legacy key metadata mapper that should help
people migrate. Where the underlying patterns have changed, we can't do
anything... wordperfect->quattropro. But we can document known limitations. We
can assess the success and find areas for improvement on our 1.2m corpus.
{noformat}
PR1 — metadata-key-tk-finalization (tk: consolidation)
- Fold X-TIKA: + scattered natives (tika:chunks, imagereader:, signature:,
rendering:, tika_pg:, hasSignature/embeddedResourceType) into one kebab tk:
namespace
- resourceName finished → tk:resource-name; Content-Type-* variants →
tk:content-type-*; WordPerfect/QuattroPro prefix collision split
- Reserved-key guard reserves both tk: and legacy X-TIKA: (drops untrusted
String writes; filters/deserialize run trusted)
- Human migration table shipped (metadata-changes-4x.adoc)
- Status: swept; tika-core + tika-serialization green; full reactor build not
re-run; not committed; awaiting review PR2 — metadata-key-format-kebab (= PR1
+ format-namespace kebab)
- Kebab Tika-coined keys inside format namespaces (pdf, zip, database, pst,
dwg, access-permission, wordperfect, quattropro, msoffice,
extended-properties), ~78 keys;
standard/file/schema names left verbatim
- Status: tika-core green; built on PR1
- Open (found this session):
- rtf_meta: namespace missed entirely — prefix + suffixes still underscored
→ fix to rtf-meta: + kebab
- digest keys emit enum .name() (SHA3_256) → change DigestDef.metadataKey()
to getJavaName() → tk:digest:SHA-256/SHA3-256 (also SHA256→SHA-256); bounded
sweep (goldens + any reader
of the key) PR3 — metadata-key-registry (new module tika-metadata-schema)
- Generated + CI-gated metadata-keys.json (634 closed keys incl. 27 digest
cross-product); dependency-free SchemaGenerator classpath scan;
MetadataSchemaTest diff-gate
- Based on main; independent of PR1/PR2
- Status: committed PR4 — metadata-key-passthrough (open-namespace
discoverability + lint) — stacked on PR3
- PassthroughPrefix makes open/passthrough namespaces auto-discoverable; 6
core holders + 22 parser files; metadata-open-namespaces.json generated +
gated; MetadataCoverageTest
backstop (no key-declaring module escapes the scan)
- Status: committed/pushed (head "property prefix")
- Staged, not yet committed on top: MetadataKeyValidator (registry-driven
CLOSED/OPEN/TEMPLATE/UNKNOWN lint) + metadata-string-keys.json (bare-String
closed keys:
Content-Type/Content-*/Message-*/tika:chunks) + MetadataStringKeysTest gate +
README Cross-cutting / newly identified (target branch: format-kebab)
- Rule: no underscores in Tika-coined 4.x keys; external-standard names
verbatim. Bundle allowlist currently empty → rule = "zero underscores in the
standard registry"
- Add no-underscore lint to the registry gates (would have caught rtf_meta:)
- 3.x→4.x migration bridge:
- LegacyKeyMigrationFilter — MetadataFilter (runs trusted, so it can write
reserved tk: keys); excludeUnmapped=false; one-line enable
- Not a flat map: flat table (~131 enumerated) + tk:digest: prefix rule
(exact) + tk:exception:/tk:warn: prefix+suffix (snake↔kebab) rule
- Migration table must be generated from a ground-truth registry diff (main
vs format-kebab). Sweep scripts proven unreliable — missed the quattropro
prefix split, embedded_parser,
and rtf_meta:
- Table gated: every v4 valid via MetadataKeyValidator
- Corpus drift-gate: run MetadataKeyValidator over real parser output, assert
no UNKNOWN; needs a module above parsers (e.g. tika-parsers-integration-tests)
to avoid a reactor cycle
- Blast radius note: majority of keys unchanged (dc:, xmp*, exif:, tiff:,
Content-* verbatim); only Tika-native + Tika-coined format keys move
Recommended sequence
1. Rebase registry module (PR3) onto format-kebab
2. Run SchemaGenerator → authoritative 4.x key set
3. Add no-underscore lint → flags rtf_meta: + digest
4. Fix rtf_meta:→rtf-meta:; DigestDef → getJavaName()
5. Generate migration table from clean 3.x↔4.x diff
6. Build LegacyKeyMigrationFilter (table + rules)
7. Corpus drift-gate
{noformat}
> Normalize/simplify metadata keys in 4.x
> ---------------------------------------
>
> Key: TIKA-4794
> URL: https://issues.apache.org/jira/browse/TIKA-4794
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Major
>
> *Background*
> Tika's metadata keys are inconsistent, and 4.0.0 is the last free break until
> 5.0 (whatever ships is frozen for the 4.x line):
> * Tika's own keys are scattered across ~8 prefixes (X-TIKA:, tika:,
> tika_pg:, rendering:, signature:, imagereader:, bare
> hasSignature/embeddedResourceType), with 3+ casings inside X-TIKA: alone
> (snake, camel, Title-Hyphen, UPPER sub-namespaces).
> * resourceName shipped half-migrated (bare → X-TIKA:resourceName) — the
> worst state.
> * The metadata object is also a control plane (media type drives routing,
> resourceName drives detection, X-TIKA:* carries provenance/digests), but the
> boundary is convention only: a crafted file can inject or overwrite Tika's
> own keys.
> * Format namespaces mix casings; QuattroPro reuses WordPerfect's prefix
> (collision); there's no rule for "is this key Tika's or the file's".
> Options
> # Leave as-is. Ship the inconsistency + half-migrated resourceName +
> convention-only boundary; frozen until 5.x
> # tk:. Consolidate every Tika-asserted key under one kebab-cased tk:
> prefix; finish resourceName → tk:resource-name; add a reserved-namespace
> trust boundary. Format namespaces untouched.
> # tk: + kebab. Everything in (2), plus apply the casing rule inside the
> format namespaces.
> *Guiding principles (2 and 3)*
> 1. Who asserted the key? Tika-computed → owned, normalized, trusted.
> File/standard-authored → verbatim (the spelling/capitalization is the interop
> contract).
> 2. Match enforcement to stakes. Control-plane / Tika-native keys →
> reserved, writable only via a Property; cosmetic drift → convention.
> 3. Casing follows key-name authorship, not the namespace (the prefix is
> just the domain). Tika coined the name → kebab; a standard/file authored it →
> verbatim.
> 4. Clean 3→4 break, documented with a migration table; no auto-remap
> machinery.
> *2 — tk: (rationale + consistency)*
> One prefix, one casing for everything Tika asserts, replacing the 8
> scattered prefixes and their mixed casings; resourceName finished. tk: (and
> legacy X-TIKA: ) become reserved: a String write to them is dropped unless
> trusted, so a scraped file can't forge Tika's
> routing/provenance/digest/content keys or inject fake tk:chunks embeddings —
> enforced once in tika-core, not per-parser discipline. Consistency: any Tika
> key is now predictable (tk: + kebab), and the trust boundary is mechanism
> rather than convention.
> *3 — tk: + kebab (rationale + consistency)*
> Extends "Tika-coined → kebab" into the format namespaces (e.g.
> pdf:hasMarkedContent → pdf:has-marked-content, zip:compressedSize →
> zip:compressed-size, access_permission: → access-permission: ). Names from a
> standard stay verbatim (dc:, xmp:, EXIF/tiff:, MAPI, email headers, ECMA-376
> OOXML props, PDF/A/X sub-namespaces) — re-casing them would break
> round-trip/interop. Also fixes the QuattroPro/WordPerfect prefix collision.
> Consistency: a Tika-authored key is kebab wherever it lives (under tk: or a
> format prefix); a standard/file key keeps its source spelling — one rule
> answers casing everywhere.
>
> If we did steps 2 or 3, we'd probably want to release a 4.0.0-beta-2.
>
> Steps 2 or 3 would be fairly major changes for downstream users, but it would
> allow us to clean up years of metadata key name bloat.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
