[
https://issues.apache.org/jira/browse/OPENNLP-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121303#comment-18121303
]
ASF GitHub Bot commented on OPENNLP-1975:
-----------------------------------------
rzo1 commented on PR #1314:
URL: https://github.com/apache/opennlp/pull/1314#issuecomment-5917700340
@krickert This is your change, I only ported it to a branch on
apache/opennlp (because of the org-fork thing) so we can both push to it. Could
you please re-review it?
I rebased it onto current main. There were no conflicts and no content
changes, and the tests for the touched classes pass. Your fixes from Sep 20 are
all in. If anything is missing or you'd like to change something, feel free to
push directly to the branch.
> RegEx removal part 9: Correct the output of the regex sites in
> TokenSampleStream, ParserTool, ConlluStream, DownloadUtil and
> FeatureGeneratorUtil
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1975
> URL: https://issues.apache.org/jira/browse/OPENNLP-1975
> Project: OpenNLP
> Issue Type: Bug
> Reporter: Richard Zowalla
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Split out of OPENNLP\-1928 \(PR #1275\). Depends on OPENNLP\-1928 \(RegEx
> removal part 1\).
> Replaces the regular expressions of these sites with scans that follow the
> intended format instead of copying the regex quirks. The output changes:*
> {{TokenSampleStream}} \(English\): tokens are split on Unicode White\_Space;
> a leading space no longer yields an empty first token; a line without a token
> resets the open\-quote state; a token with a letter or digit of any script is
> a word.
> * {{ParserTool}}: brackets that follow each other are each separated
> \({{a\)\)b}} gives {{a \) \) b}}\).
> * {{ConlluStream}}: a malformed multiword token id \({{3\-}}, {{\-4}},
> {{3\-\-4}}, {{1\-2\-3}}, {{4\-3}}\) fails with an {{InvalidFormatException}}
> naming the id.
> * {{DownloadUtil}}: a blank checksum file is an {{IOException}} saying so.
> * {{FeatureGeneratorUtil}}: the {{cp}} class requires exactly one capital and
> one period.
> Before merge: tokenizer eval build that reads through {{TokenSampleStream}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
