Richard Zowalla created OPENNLP-1947:
----------------------------------------
Summary: TokenizerME changes keepNewLines on the shared
WhitespaceTokenizer.INSTANCE
Key: OPENNLP-1947
URL: https://issues.apache.org/jira/browse/OPENNLP-1947
Project: OpenNLP
Issue Type: Bug
Reporter: Richard Zowalla
{code:java}
// TokenizerME.java:195\-196
WhitespaceTokenizer whitespaceTokenizer = WhitespaceTokenizer.INSTANCE;
whitespaceTokenizer.setKeepNewLines\(keepNewLines\);
{code}
{{TokenizerME}} sets {{keepNewLines}} on the process\-wide singleton on every
{{tokenizePos}} call.
_Consequences:_
\* After any {{TokenizerME}} with {{keepNewLines=true}} has run, every other
user of {{WhitespaceTokenizer.INSTANCE}} gets {{\n}} and {{\r}} back as tokens.
About 25 production callers are affected, e.g. {{NameSample.parse}},
{{POSSample.parse}}, {{TokenSample}}, {{DocumentSampleStream}},
{{SentimentSampleStream}}, the brat and OntoNotes readers, and the CLI tools.
\* {{TokenizerME}} is annotated {{@ThreadSafe}}, but two instances with
different {{keepNewLines}} settings race on the same field.
_Repro:_ run a {{TokenizerME}} with {{setKeepNewLines\(true\)}}, then
{{WhitespaceTokenizer.INSTANCE.tokenize\("a\nb"\)}} returns {{\[a, \n, b\]}}.
_Fix:_ use a local {{WhitespaceTokenizer}} in {{TokenizerME}}, or pass
{{keepNewLines}} as a parameter. Longer term, make {{INSTANCE}} immutable
\(deprecate {{setKeepNewLines}} on it\).
The open regex\-removal PRs add more callers of {{INSTANCE}}
\(OPENNLP\-1928/1929/1930/1933\), so this should land first.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
