[
https://issues.apache.org/jira/browse/OPENNLP-1947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Wiesner updated OPENNLP-1947:
------------------------------------
Fix Version/s: 2.5.13
3.0.0
> Stop TokenizerME from changing the shared WhitespaceTokenizer
> -------------------------------------------------------------
>
> Key: OPENNLP-1947
> URL: https://issues.apache.org/jira/browse/OPENNLP-1947
> Project: OpenNLP
> Issue Type: Bug
> Affects Versions: 2.5.12, 3.0.0-M6
> Reporter: Richard Zowalla
> Assignee: Kristian Rickert
> Priority: Minor
> Fix For: 2.5.13, 3.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {code:java}
> // TokenizerME.java:195\-196
> WhitespaceTokenizer whitespaceTokenizer = WhitespaceTokenizer.INSTANCE;
> whitespaceTokenizer.setKeepNewLines\(keepNewLines\);
> {code}
> {{TokenizerME}} sets {{keepNewLines}} on the process\-wide singleton on every
> {{tokenizePos}} call.
> _Consequences:_
> \* After any {{TokenizerME}} with {{keepNewLines=true}} has run, every other
> user of {{WhitespaceTokenizer.INSTANCE}} gets {{\n}} and {{\r}} back as
> tokens. About 25 production callers are affected, e.g. {{NameSample.parse}},
> {{POSSample.parse}}, {{TokenSample}}, {{DocumentSampleStream}},
> {{SentimentSampleStream}}, the brat and OntoNotes readers, and the CLI tools.
> \* {{TokenizerME}} is annotated {{@ThreadSafe}}, but two instances with
> different {{keepNewLines}} settings race on the same field.
> _Repro:_ run a {{TokenizerME}} with {{setKeepNewLines\(true\)}}, then
> {{WhitespaceTokenizer.INSTANCE.tokenize\("a\nb"\)}} returns {{\[a, \n, b\]}}.
> _Fix:_ use a local {{WhitespaceTokenizer}} in {{TokenizerME}}, or pass
> {{keepNewLines}} as a parameter. Longer term, make {{INSTANCE}} immutable
> \(deprecate {{setKeepNewLines}} on it\).
> The open regex\-removal PRs add more callers of {{INSTANCE}}
> \(OPENNLP\-1928/1929/1930/1933\), so this should land first.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
