[
https://issues.apache.org/jira/browse/OPENNLP-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristian Rickert updated OPENNLP-1942:
--------------------------------------
Description:
The shared deep-learning encoder in {{AbstractDL}}, used by every opennlp-dl
component, had no tests that run without an ONNX model. Its javadoc also
misstated the token type array.
* Adds {{SharedDlEncodingTest}}: verifies the complete
{{tokens}}/{{ids}}/{{mask}}/{{types}} arrays over a plain vocabulary through a
session-free {{AbstractDL}} stub, rejects null input, and checks that repeated
and concurrent calls return independent arrays.
* Corrects the {{Tokens#types}} javadoc: the values are segment ids, all 0 for
a single input segment, not "typically all 1".
* Links the Unicode SpecialCasing.txt data file from the {{WordpieceEncoder}}
case-mapping javadoc.
No runtime behavior changes; the production diff is javadoc only.
PR: https://github.com/apache/opennlp/pull/1288
was:
Adds a new opennlp-extensions/opennlp-subword module implementing SentencePiece
model inference purely in Java: the model file reader (hand-written protobuf
wire parsing, no new dependency), the model-embedded normalizer (precompiled
character map over the double-array trie format, whitespace collapsing and
escaping, word-boundary marker), unigram best-path segmentation, BPE agenda
merging, byte fallback, and user-defined symbol handling. The public contract
is SubwordTokenizer/SubwordPiece; every piece reports the exact UTF-16 span of
the caller's original text it came from, and the model normalizer is also
exposed as an OffsetAwareNormalizer producing AlignedText.
Parity with the reference implementation is asserted, not assumed. Five tiny
trained models are bundled with fixtures generated by the sentencepiece Python
package (unigram, unigram with byte fallback, BPE, identity normalization,
whitespace-as-suffix), checked piece for piece, id for id, span for span, plus
each model's embedded self-test samples. An opt-in test
(-Dopennlp.subword.eval.dir) runs the same assertions against real downloaded
models; t5-small (32k vocabulary) and albert-base-v2 (30k) pass exactly,
including mixed scripts, emoji ZWJ sequences, BOM, and CRLF inputs. Fixture
regeneration scripts live in the test resources.
Measured single-thread throughput on the t5-small vocabulary is about 2.8M
pieces/s; the native reference is about 1.6x faster. The value here is zero
native dependencies, one shareable thread-safe instance, and exact
original-text offsets. A non-breaking performance follow-up with identified
wins is planned separately.
JIRA to follow; title gets the key once filed.
> Test the shared deep-learning encoder and fix the token type javadoc
> --------------------------------------------------------------------
>
> Key: OPENNLP-1942
> URL: https://issues.apache.org/jira/browse/OPENNLP-1942
> Project: OpenNLP
> Issue Type: New Feature
> Components: Sentence Detector
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0-M6, 3.0.0
>
>
> The shared deep-learning encoder in {{AbstractDL}}, used by every opennlp-dl
> component, had no tests that run without an ONNX model. Its javadoc also
> misstated the token type array.
> * Adds {{SharedDlEncodingTest}}: verifies the complete
> {{tokens}}/{{ids}}/{{mask}}/{{types}} arrays over a plain vocabulary through
> a session-free {{AbstractDL}} stub, rejects null input, and checks that
> repeated and concurrent calls return independent arrays.
> * Corrects the {{Tokens#types}} javadoc: the values are segment ids, all 0
> for a single input segment, not "typically all 1".
> * Links the Unicode SpecialCasing.txt data file from the {{WordpieceEncoder}}
> case-mapping javadoc.
> No runtime behavior changes; the production diff is javadoc only.
> PR: https://github.com/apache/opennlp/pull/1288
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
