[
https://issues.apache.org/jira/browse/OPENNLP-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111303#comment-18111303
]
ASF GitHub Bot commented on OPENNLP-1885:
-----------------------------------------
krickert commented on PR #178:
URL: https://github.com/apache/opennlp-addons/pull/178#issuecomment-5532737091
So this is our "canary" PR that would go into our "addons" - of course we'll
need to setup the infra for the whole addons, but this would be the impl for
the OPENNLP-1885. This will unlock a lot of the embedding add-ons that are
being proposed.
> Pure-Java SentencePiece inference with exact original-text spans
> -----------------------------------------------------------------
>
> Key: OPENNLP-1885
> URL: https://issues.apache.org/jira/browse/OPENNLP-1885
> Project: OpenNLP
> Issue Type: New Feature
> Components: Sentence Detector
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0-M6, 3.0.0
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Adds a new opennlp-extensions/opennlp-subword module implementing
> SentencePiece model inference purely in Java: the model file reader
> (hand-written protobuf wire parsing, no new dependency), the model-embedded
> normalizer (precompiled character map over the double-array trie format,
> whitespace collapsing and escaping, word-boundary marker), unigram best-path
> segmentation, BPE agenda merging, byte fallback, and user-defined symbol
> handling. The public contract is SubwordTokenizer/SubwordPiece; every piece
> reports the exact UTF-16 span of the caller's original text it came from, and
> the model normalizer is also exposed as an OffsetAwareNormalizer producing
> AlignedText.
> Parity with the reference implementation is asserted, not assumed. Five tiny
> trained models are bundled with fixtures generated by the sentencepiece
> Python package (unigram, unigram with byte fallback, BPE, identity
> normalization, whitespace-as-suffix), checked piece for piece, id for id,
> span for span, plus each model's embedded self-test samples. An opt-in test
> (-Dopennlp.subword.eval.dir) runs the same assertions against real downloaded
> models; t5-small (32k vocabulary) and albert-base-v2 (30k) pass exactly,
> including mixed scripts, emoji ZWJ sequences, BOM, and CRLF inputs. Fixture
> regeneration scripts live in the test resources.
> Measured single-thread throughput on the t5-small vocabulary is about 2.8M
> pieces/s; the native reference is about 1.6x faster. The value here is zero
> native dependencies, one shareable thread-safe instance, and exact
> original-text offsets. A non-breaking performance follow-up with identified
> wins is planned separately.
> JIRA to follow; title gets the key once filed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
