[
https://issues.apache.org/jira/browse/OPENNLP-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121755#comment-18121755
]
ASF GitHub Bot commented on OPENNLP-1988:
-----------------------------------------
krickert commented on PR #1319:
URL: https://github.com/apache/opennlp/pull/1319#issuecomment-5942190504
This was a ticket for helping follow-ups stacked tix with remaining regex
removal. It can hold off.
> Add support for Hugging Face tokenizer.json files whose tokenizer
> configuration can be represented by OpenNLP's WordPiece encoder.
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1988
> URL: https://issues.apache.org/jira/browse/OPENNLP-1988
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This work is separated from OPENNLP-1931, which replaces regex-based
> vocabulary
> and configuration parsing. That PR will support vocab.txt and flat vocab.json
> files; tokenizer.json support belongs in this follow-up.
> Reading model.vocab alone is insufficient. AbstractDL constructs a
> WordpieceEncoder, but tokenizer.json also represents BPE and Unigram models.
> Accepting a BPE vocabulary and applying WordPiece can produce incorrect model
> inputs without reporting an error. Even WordPiece files can specify settings
> that differ from the encoder's defaults.
> Scope
> - Require model.type to be WordPiece and model.vocab to be an object mapping
> tokens to non-negative integer IDs. Preserve the supplied IDs.
> - Reject BPE, Unigram, missing or unknown model types, and unsupported layouts
> with an error identifying the unsupported setting.
> - Define the supported WordPiece configuration, including
> continuing_subword_prefix, unk_token, max_input_chars_per_word,
> normalization,
> pre-tokenization, and special-token processing. Honor supported settings and
> reject incompatible settings instead of silently using encoder defaults.
> - Define how tokenizer settings interact with explicit InferenceOptions,
> particularly lowerCase. Document defaults and conflict handling.
> - Define handling of added_tokens, including tokens absent from model.vocab,
> conflicting IDs, and special-token matching. Reject unsupported behavior
> rather than silently dropping entries that affect tokenization.
> - Reuse the JSON scanner from OPENNLP-1931. Report malformed or unsupported
> resource files as InvalidFormatException at the file-loading boundary.
> - Keep existing vocab.txt and flat vocab.json behavior unchanged. Document
> the supported tokenizer.json subset in the manual.
> Validation
> - Add fixtures for supported WordPiece configurations and rejection cases for
> BPE, Unigram, missing model types, invalid IDs, incompatible settings, and
> unsupported added_tokens behavior.
> - Check exact token pieces and IDs against reference outputs for
> representative
> cased and uncased WordPiece models. Cover continuation pieces, unknown
> tokens,
> Unicode normalization, and special tokens; vocabulary-map equality alone is
> insufficient.
> - Verify errors through the public DL file-loading boundary and retain
> coverage
> for existing vocabulary formats.
> - Run the affected DL tests and DocumentCategorizerDLEval with a supported
> model
> before merge.
> Out of scope: implementing BPE or Unigram tokenizers, or claiming support for
> arbitrary Hugging Face tokenizer pipelines.
> References
> OPENNLP-1931: https://issues.apache.org/jira/browse/OPENNLP-1931
> PR #1277: https://github.com/apache/opennlp/pull/1277
> Review requesting this split:
> https://github.com/apache/opennlp/pull/1277#issuecomment-5743853571
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
