Richard Zowalla created OPENNLP-1975:
----------------------------------------
Summary: RegEx removal part 9: Correct the output of the regex
sites in TokenSampleStream, ParserTool, ConlluStream, DownloadUtil and
FeatureGeneratorUtil
Key: OPENNLP-1975
URL: https://issues.apache.org/jira/browse/OPENNLP-1975
Project: OpenNLP
Issue Type: Bug
Reporter: Richard Zowalla
Assignee: Kristian Rickert
Fix For: 3.0.0
Split out of OPENNLP\-1928 \(PR #1275\). Depends on OPENNLP\-1928 \(RegEx
removal part 1\).
Replaces the regular expressions of these sites with scans that follow the
intended format instead of copying the regex quirks. The output changes:*
{{TokenSampleStream}} \(English\): tokens are split on Unicode White\_Space; a
leading space no longer yields an empty first token; a line without a token
resets the open\-quote state; a token with a letter or digit of any script is a
word.
* {{ParserTool}}: brackets that follow each other are each separated
\({{a\)\)b}} gives {{a \) \) b}}\).
* {{ConlluStream}}: a malformed multiword token id \({{3\-}}, {{\-4}},
{{3\-\-4}}, {{1\-2\-3}}, {{4\-3}}\) fails with an {{InvalidFormatException}}
naming the id.
* {{DownloadUtil}}: a blank checksum file is an {{IOException}} saying so.
* {{FeatureGeneratorUtil}}: the {{cp}} class requires exactly one capital and
one period.
Before merge: tokenizer eval build that reads through {{TokenSampleStream}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
