[
https://issues.apache.org/jira/browse/OPENNLP-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Wiesner resolved OPENNLP-1885.
-------------------------------------
Resolution: Fixed
> Introduce pure-Java SentencePiece inference with exact original-text spans
> ---------------------------------------------------------------------------
>
> Key: OPENNLP-1885
> URL: https://issues.apache.org/jira/browse/OPENNLP-1885
> Project: OpenNLP
> Issue Type: New Feature
> Components: Sentence Detector
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0-M6, 3.0.0
>
> Time Spent: 6h
> Remaining Estimate: 0h
>
> Adds a new opennlp-extensions/opennlp-subword module implementing
> SentencePiece model inference purely in Java: the model file reader
> (hand-written protobuf wire parsing, no new dependency), the model-embedded
> normalizer (precompiled character map over the double-array trie format,
> whitespace collapsing and escaping, word-boundary marker), unigram best-path
> segmentation, BPE agenda merging, byte fallback, and user-defined symbol
> handling. The public contract is SubwordTokenizer/SubwordPiece; every piece
> reports the exact UTF-16 span of the caller's original text it came from, and
> the model normalizer is also exposed as an OffsetAwareNormalizer producing
> AlignedText.
> Parity with the reference implementation is asserted, not assumed. Five tiny
> trained models are bundled with fixtures generated by the sentencepiece
> Python package (unigram, unigram with byte fallback, BPE, identity
> normalization, whitespace-as-suffix), checked piece for piece, id for id,
> span for span, plus each model's embedded self-test samples. An opt-in test
> (-Dopennlp.subword.eval.dir) runs the same assertions against real downloaded
> models; t5-small (32k vocabulary) and albert-base-v2 (30k) pass exactly,
> including mixed scripts, emoji ZWJ sequences, BOM, and CRLF inputs. Fixture
> regeneration scripts live in the test resources.
> Measured single-thread throughput on the t5-small vocabulary is about 2.8M
> pieces/s; the native reference is about 1.6x faster. The value here is zero
> native dependencies, one shareable thread-safe instance, and exact
> original-text offsets. A non-breaking performance follow-up with identified
> wins is planned separately.
> JIRA to follow; title gets the key once filed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
