Kristian Rickert created OPENNLP-1931:
-----------------------------------------
Summary: RegEx removal part 3: JSON vocabulary and id2label scrape
in opennlp-dl
Key: OPENNLP-1931
URL: https://issues.apache.org/jira/browse/OPENNLP-1931
Project: OpenNLP
Issue Type: Improvement
Components: Deep Learning
Reporter: Kristian Rickert
Assignee: Kristian Rickert
{{AbstractDL.loadJsonVocab}} and {{DocumentCategorizerConfig.fromJson}} located
string keys and values with regular expressions. Both now use a small cursor
helper, {{opennlp.dl.JsonScan}} (public, marked internal), that finds string
literals honoring backslash escapes, skips ASCII whitespace, and reads digit
runs.
Behavior is unchanged, including the quirks of the old patterns, which the
tests pin: a value with a decimal fraction yields its integer prefix, a
negative id is skipped, the id2label object is cut at the first closing brace
even inside a nested value, and a key spanning a newline is accepted while a
value with a newline is skipped.
Verification: the old patterns and the new scan were compared over the real
vocabulary and configuration resources, 57 hand-written edge cases, and 400,000
generated inputs with zero differences. {{JsonScanTest}} adds 71 cases,
{{LoadVocabTest}} 27, {{DocumentCategorizerConfigTest}} 24.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
