Kristian Rickert created OPENNLP-1988:
-----------------------------------------
Summary: Add support for Hugging Face tokenizer.json files whose
tokenizer configuration can be represented by OpenNLP's WordPiece encoder.
Key: OPENNLP-1988
URL: https://issues.apache.org/jira/browse/OPENNLP-1988
Project: OpenNLP
Issue Type: Improvement
Reporter: Kristian Rickert
Assignee: Kristian Rickert
This work is separated from OPENNLP-1931, which replaces regex-based vocabulary
and configuration parsing. That PR will support vocab.txt and flat vocab.json
files; tokenizer.json support belongs in this follow-up.
Reading model.vocab alone is insufficient. AbstractDL constructs a
WordpieceEncoder, but tokenizer.json also represents BPE and Unigram models.
Accepting a BPE vocabulary and applying WordPiece can produce incorrect model
inputs without reporting an error. Even WordPiece files can specify settings
that differ from the encoder's defaults.
Scope
- Require model.type to be WordPiece and model.vocab to be an object mapping
tokens to non-negative integer IDs. Preserve the supplied IDs.
- Reject BPE, Unigram, missing or unknown model types, and unsupported layouts
with an error identifying the unsupported setting.
- Define the supported WordPiece configuration, including
continuing_subword_prefix, unk_token, max_input_chars_per_word, normalization,
pre-tokenization, and special-token processing. Honor supported settings and
reject incompatible settings instead of silently using encoder defaults.
- Define how tokenizer settings interact with explicit InferenceOptions,
particularly lowerCase. Document defaults and conflict handling.
- Define handling of added_tokens, including tokens absent from model.vocab,
conflicting IDs, and special-token matching. Reject unsupported behavior
rather than silently dropping entries that affect tokenization.
- Reuse the JSON scanner from OPENNLP-1931. Report malformed or unsupported
resource files as InvalidFormatException at the file-loading boundary.
- Keep existing vocab.txt and flat vocab.json behavior unchanged. Document
the supported tokenizer.json subset in the manual.
Validation
- Add fixtures for supported WordPiece configurations and rejection cases for
BPE, Unigram, missing model types, invalid IDs, incompatible settings, and
unsupported added_tokens behavior.
- Check exact token pieces and IDs against reference outputs for representative
cased and uncased WordPiece models. Cover continuation pieces, unknown tokens,
Unicode normalization, and special tokens; vocabulary-map equality alone is
insufficient.
- Verify errors through the public DL file-loading boundary and retain coverage
for existing vocabulary formats.
- Run the affected DL tests and DocumentCategorizerDLEval with a supported model
before merge.
Out of scope: implementing BPE or Unigram tokenizers, or claiming support for
arbitrary Hugging Face tokenizer pipelines.
References
OPENNLP-1931: https://issues.apache.org/jira/browse/OPENNLP-1931
PR #1277: https://github.com/apache/opennlp/pull/1277
Review requesting this split:
https://github.com/apache/opennlp/pull/1277#issuecomment-5743853571
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
