[
https://issues.apache.org/jira/browse/OPENNLP-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18123698#comment-18123698
]
ASF GitHub Bot commented on OPENNLP-1988:
-----------------------------------------
krickert commented on code in PR #1319:
URL: https://github.com/apache/opennlp/pull/1319#discussion_r4193611998
##########
opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java:
##########
@@ -96,11 +115,14 @@ protected record TextChunk(String text, int start, int
end) {
* @param sessionOptions The session options (e.g. CUDA execution provider);
build with
* {@link #sessionOptions(InferenceOptions)} when honoring {@link
InferenceOptions}.
* @param lowerCase {@code true} for uncased models (lower casing and accent
stripping
- * during tokenization), {@code false} for cased models.
+ * during tokenization), {@code false} for cased models. A {@code
tokenizer.json} that
+ * sets {@code normalizer.lowercase} must agree with it.
*
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if the {@code model} or {@code vocabulary}
cannot be read.
- * @throws InvalidFormatException Thrown if a JSON {@code vocabulary} is
malformed.
+ * @throws InvalidFormatException Thrown if a JSON {@code vocabulary} is
malformed, has an
+ * unsupported layout, or sets {@code normalizer.lowercase} to the other
value than
+ * {@code lowerCase}.
*/
protected AbstractDL(final File model, final File vocabulary,
Review Comment:
I converted all the ones I added from before. The others predate the PR, I
can piggy back that though - give a thumbs up or on ok and I'm on it.
> Add support for Hugging Face tokenizer.json files whose tokenizer
> configuration can be represented by OpenNLP's WordPiece encoder.
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1988
> URL: https://issues.apache.org/jira/browse/OPENNLP-1988
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This work is separated from OPENNLP-1931, which replaces regex-based
> vocabulary
> and configuration parsing. That PR will support vocab.txt and flat vocab.json
> files; tokenizer.json support belongs in this follow-up.
> Reading model.vocab alone is insufficient. AbstractDL constructs a
> WordpieceEncoder, but tokenizer.json also represents BPE and Unigram models.
> Accepting a BPE vocabulary and applying WordPiece can produce incorrect model
> inputs without reporting an error. Even WordPiece files can specify settings
> that differ from the encoder's defaults.
> Scope
> - Require model.type to be WordPiece and model.vocab to be an object mapping
> tokens to non-negative integer IDs. Preserve the supplied IDs.
> - Reject BPE, Unigram, missing or unknown model types, and unsupported layouts
> with an error identifying the unsupported setting.
> - Define the supported WordPiece configuration, including
> continuing_subword_prefix, unk_token, max_input_chars_per_word,
> normalization,
> pre-tokenization, and special-token processing. Honor supported settings and
> reject incompatible settings instead of silently using encoder defaults.
> - Define how tokenizer settings interact with explicit InferenceOptions,
> particularly lowerCase. Document defaults and conflict handling.
> - Define handling of added_tokens, including tokens absent from model.vocab,
> conflicting IDs, and special-token matching. Reject unsupported behavior
> rather than silently dropping entries that affect tokenization.
> - Reuse the JSON scanner from OPENNLP-1931. Report malformed or unsupported
> resource files as InvalidFormatException at the file-loading boundary.
> - Keep existing vocab.txt and flat vocab.json behavior unchanged. Document
> the supported tokenizer.json subset in the manual.
> Validation
> - Add fixtures for supported WordPiece configurations and rejection cases for
> BPE, Unigram, missing model types, invalid IDs, incompatible settings, and
> unsupported added_tokens behavior.
> - Check exact token pieces and IDs against reference outputs for
> representative
> cased and uncased WordPiece models. Cover continuation pieces, unknown
> tokens,
> Unicode normalization, and special tokens; vocabulary-map equality alone is
> insufficient.
> - Verify errors through the public DL file-loading boundary and retain
> coverage
> for existing vocabulary formats.
> - Run the affected DL tests and DocumentCategorizerDLEval with a supported
> model
> before merge.
> Out of scope: implementing BPE or Unigram tokenizers, or claiming support for
> arbitrary Hugging Face tokenizer pipelines.
> References
> OPENNLP-1931: https://issues.apache.org/jira/browse/OPENNLP-1931
> PR #1277: https://github.com/apache/opennlp/pull/1277
> Review requesting this split:
> https://github.com/apache/opennlp/pull/1277#issuecomment-5743853571
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
