Kristian Rickert created OPENNLP-1917:
-----------------------------------------
Summary: Add ONNX-backed syntactic chunking with reproducible
English model training
Key: OPENNLP-1917
URL: https://issues.apache.org/jira/browse/OPENNLP-1917
Project: OpenNLP
Issue Type: Improvement
Components: language model
Reporter: Kristian Rickert
Assignee: Kristian Rickert
h3. Background
OpenNLP provides the classic {{ChunkerME}} API and implementation, but the
current Apache model distribution does not include a recently trained syntactic
chunker model. The remaining downloadable English model is a legacy
SourceForge-era artifact and should not be the default model for a modern
OpenNLP demonstration.
The {{opennlp-dl}} module already provides shared ONNX Runtime, BERT
tokenization, subword alignment, CPU/GPU selection, and lifecycle foundations
for other NLP tasks.
h3. Proposed work
Add an ONNX-backed syntactic chunker while preserving the existing OpenNLP
chunker API.
* Implement {{ChunkerDL}} behind the existing chunker contract.
* Define and document the supported ONNX input, output, vocabulary, label,
and model-manifest contract.
* Preserve exact correspondence between caller-supplied tokens and returned
chunk labels.
* Decode standard BIO chunk labels such as {{B-NP}}, {{I-NP}}, {{B-VP}}, and
{{B-PP}}.
* Validate tensor names, shapes, label cardinality, maximum input length, and
malformed model output.
* Support bounded, thread-safe inference and deterministic native-resource
cleanup.
* Add a reproducible English chunker training and ONNX export pipeline.
* Pin corpus revisions, preprocessing configuration, tool versions, and
artifact checksums.
* Produce held-out precision, recall, and F1 evaluation reports.
* Document model provenance and licensing before publishing an artifact
through {{apache/opennlp-models}}.
* Add usage and training documentation with executable or mirror-tested
examples.
The runtime implementation and the model artifact may be reviewed as separate
pull requests under this JIRA issue.
h3. Acceptance criteria
* {{ChunkerDL}} passes the existing OpenNLP chunker contract tests.
* Tests cover Unicode text, subword splits, unknown tokens, empty input, long
input, malformed outputs, model-loading failures, concurrency, and cleanup.
* Training and evaluation are reproducible from documented, checksum-pinned
inputs.
* Evaluation includes a clearly identified held-out corpus and published
precision, recall, and F1.
* The model manifest records the label set, vocabulary identity,
preprocessing rules, corpus provenance, artifact hash, and OpenNLP
compatibility.
* No test, build, or demonstration requires a SourceForge-era model.
* The resulting model can be loaded by the OpenNLP gRPC service without
changing the public chunk annotation contract.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
