Kristian Rickert created OPENNLP-1929:
-----------------------------------------
Summary: BasicContextGenerator splits on the separator as a
regular expression
Key: OPENNLP-1929
URL: https://issues.apache.org/jira/browse/OPENNLP-1929
Project: OpenNLP
Issue Type: Bug
Components: Machine Learning
Reporter: Kristian Rickert
Assignee: Kristian Rickert
{{BasicContextGenerator(String sep)}} documents its argument as a "separator
character", but {{getContext}} calls {{String.split(separator)}}, which
compiles the argument as a regular expression on every call.
Consequences for a caller who follows the Javadoc:
* {{"|"}}, {{"."}}, {{"+"}}, {{"*"}}, {{"?"}} split on every character or on
nothing.
* {{"("}} or {{"["}} throw {{PatternSyntaxException}} at prediction time, not
at construction.
* An empty separator splits the input into single characters.
* If an application takes the separator from configuration, a crafted pattern
is re-evaluated on every context string, which is a backtracking exposure.
No code in the repository calls the separator constructor; it arrived with the
maxent modularization (OPENNLP-1740) and has no test.
Fix: split on the literal separator with a cursor scan, keeping
{{String.split}} trailing-empty semantics; reject an empty separator with
{{IllegalArgumentException}} at construction; state in the Javadoc that the
separator is literal. Tests: default space, a metacharacter separator, a
multi-character separator, leading and trailing separators, empty and null
separator.
Compatibility: external code that intentionally passes a pattern such as
{{\s+}} relied on undocumented behavior; 3.0 is the release to correct it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
