Richard Zowalla created OPENNLP-1953:
----------------------------------------
Summary: AbstractDL.loadVocabFile misdetects a JSON vocabulary
that starts with a BOM
Key: OPENNLP-1953
URL: https://issues.apache.org/jira/browse/OPENNLP-1953
Project: OpenNLP
Issue Type: Bug
Reporter: Richard Zowalla
{{AbstractDL.loadVocabFile}} \(lines 185\-194\) detects JSON with
{{content.trim\(\).startsWith\("{"\)}}. {{Files.readString}} keeps U\+FEFF, and
{{trim\(\)}} doesn't remove it, so a vocab.json saved with a BOM falls through
to the plain\-text path. Each JSON line then becomes a token, and the ids are
silently wrong: no error, just wrong predictions.
Verified: {{"{\"a\":1}".trim\(\).startsWith\("{"\)}} is {{false}}.
_Fix:_ strip a leading U\+FEFF before detection \(and in
{{DocumentCategorizerConfig}} too\), and add a test with a BOM\-prefixed
vocab.json.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
