[
https://issues.apache.org/jira/browse/OPENNLP-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18099314#comment-18099314
]
Kristian Rickert commented on OPENNLP-1897:
-------------------------------------------
Requires DocumentShape to be completed
> Term vector layer: aggregated (term, frequency, offsets) roll-up over the
> token layer for index consumers
> ---------------------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1897
> URL: https://issues.apache.org/jira/browse/OPENNLP-1897
> Project: OpenNLP
> Issue Type: New Feature
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
>
> h3. Abstract
> Consumers building search indexes need term vectors: for each distinct
> normalized term in a document, its frequency and the offsets of every
> occurrence. OpenNLP today produces everything required - token spans from the
> tokenizer, normalization through the normalizer layers (rungs!), and offset
> fidelity back to the original text through {{Alignment}} - but the roll-up
> itself is left to every consumer to re-implement: group tokens by normalized
> form, count, collect spans, map through {{Alignment}}.
> h3. Design
> Three additions, touching no existing type, in the feature's own package:
> # A {{payload}} record: term string, occurrence count, and the occurrence
> spans. Spans always point into the original text; any normalization applied
> maps back through Alignment, per the container's existing rule.
> # A {{LayerKey}} for the term-vector layer.
> # A {{TermVectorAnnotator}} that requires the token layer (and optionally a
> normalization layer), and provides the term-vector layer.
> *No new dependencies*. Whether occurrences carry full per-occurrence spans or
> only the count is an annotator option, so scoring-only consumers don't pay
> for offsets they never read.
> h3. Coexistence with the task-level API
> Nothing changes. The annotator reads layers produced by existing annotators
> over existing task interfaces.
> h3. Acceptance criteria
> # The layer is added without modifying the container's package, proving the
> additive property holds for an aggregation-shaped layer (OPENNLP-1888
> validated graph-shaped; this validates roll-up-shaped).
> # Tests cover offset fidelity through a normalization-shifted case
> (whitespace collapse, eszett case-fold expansion), proving spans land on the
> original text.
> # Scoring-only mode (counts without spans) and full-offset mode both covered.
> h3. Guardrail note
> The annotator aggregates; it does not analyze. Term identity (what
> normalization, if any, defines "the same term") is delegated to the layers it
> reads, so the term-vector layer never grows its own language behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
