Kristian Rickert created OPENNLP-1897:
-----------------------------------------
Summary: Term vector layer: aggregated (term, frequency, offsets)
roll-up over the token layer for index consumers
Key: OPENNLP-1897
URL: https://issues.apache.org/jira/browse/OPENNLP-1897
Project: OpenNLP
Issue Type: New Feature
Reporter: Kristian Rickert
Assignee: Kristian Rickert
h3. Abstract
Consumers building search indexes need term vectors: for each distinct
normalized term in a document, its frequency and the offsets of every
occurrence. OpenNLP today produces everything required - token spans from the
tokenizer, normalization through the normalizer layers (rungs!), and offset
fidelity back to the original text through {{Alignment}} - but the roll-up
itself is left to every consumer to re-implement: group tokens by normalized
form, count, collect spans, map through {{Alignment}}.
h3. Design
Three additions, touching no existing type, in the feature's own package:
# A {{payload}} record: term string, occurrence count, and the occurrence
spans. Spans always point into the original text; any normalization applied
maps back through Alignment, per the container's existing rule.
# A {{LayerKey}} for the term-vector layer.
# A {{TermVectorAnnotator}} that requires the token layer (and optionally a
normalization layer), and provides the term-vector layer.
*No new dependencies*. Whether occurrences carry full per-occurrence spans or
only the count is an annotator option, so scoring-only consumers don't pay for
offsets they never read.
h3. Coexistence with the task-level API
Nothing changes. The annotator reads layers produced by existing annotators
over existing task interfaces.
h3. Acceptance criteria
# The layer is added without modifying the container's package, proving the
additive property holds for an aggregation-shaped layer (OPENNLP-1888 validated
graph-shaped; this validates roll-up-shaped).
# Tests cover offset fidelity through a normalization-shifted case (whitespace
collapse, eszett case-fold expansion), proving spans land on the original text.
# Scoring-only mode (counts without spans) and full-offset mode both covered.
h3. Guardrail note
The annotator aggregates; it does not analyze. Term identity (what
normalization, if any, defines "the same term") is delegated to the layers it
reads, so the term-vector layer never grows its own language behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
