Kristian Rickert created OPENNLP-1942:
-----------------------------------------
Summary: Test the shared deep-learning encoder and fix the token
type javadoc
Key: OPENNLP-1942
URL: https://issues.apache.org/jira/browse/OPENNLP-1942
Project: OpenNLP
Issue Type: New Feature
Components: Sentence Detector
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Fix For: 3.0.0-M6, 3.0.0
Adds a new opennlp-extensions/opennlp-subword module implementing SentencePiece
model inference purely in Java: the model file reader (hand-written protobuf
wire parsing, no new dependency), the model-embedded normalizer (precompiled
character map over the double-array trie format, whitespace collapsing and
escaping, word-boundary marker), unigram best-path segmentation, BPE agenda
merging, byte fallback, and user-defined symbol handling. The public contract
is SubwordTokenizer/SubwordPiece; every piece reports the exact UTF-16 span of
the caller's original text it came from, and the model normalizer is also
exposed as an OffsetAwareNormalizer producing AlignedText.
Parity with the reference implementation is asserted, not assumed. Five tiny
trained models are bundled with fixtures generated by the sentencepiece Python
package (unigram, unigram with byte fallback, BPE, identity normalization,
whitespace-as-suffix), checked piece for piece, id for id, span for span, plus
each model's embedded self-test samples. An opt-in test
(-Dopennlp.subword.eval.dir) runs the same assertions against real downloaded
models; t5-small (32k vocabulary) and albert-base-v2 (30k) pass exactly,
including mixed scripts, emoji ZWJ sequences, BOM, and CRLF inputs. Fixture
regeneration scripts live in the test resources.
Measured single-thread throughput on the t5-small vocabulary is about 2.8M
pieces/s; the native reference is about 1.6x faster. The value here is zero
native dependencies, one shareable thread-safe instance, and exact
original-text offsets. A non-breaking performance follow-up with identified
wins is planned separately.
JIRA to follow; title gets the key once filed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
