[
https://issues.apache.org/jira/browse/OPENNLP-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121800#comment-18121800
]
ASF GitHub Bot commented on OPENNLP-1975:
-----------------------------------------
rzo1 opened a new pull request, #1351:
URL: https://github.com/apache/opennlp/pull/1351
Follow-up to #1314, which was merged before @mawiesne's review comments were
addressed.
## Changes
- **`TokenSampleStream`**
- The checked symbols are now constants; the two chains of `equals` calls
are now `SPACED_PUNCTUATION` and `ATTACHING_TO_NEXT` sets.
- Dropped `(!lastToken.equals("(") || !lastToken.equals("{"))`, which is
always true. Output is unchanged.
- Javadoc wording: "Tests" is now "Checks".
- `containsLetterOrDigit` is now `static`.
- **`DownloadUtil`**
- A blank checksum file throws `IllegalArgumentException`, and a checksum
mismatch throws `IllegalStateException`. Both used to throw `IOException`.
- Javadoc is updated on `verifyChecksum`, `validateModel`,
`validateCachedModel`, `existsModel` and both `downloadModel` overloads.
- **`ConlluStream`**: `parseContractionRange` and `extractTextLang` don't
use instance state, so they are now `static`.
- **`ConlluStreamTest`**: `sent1.getWordLines()` is extracted into a local
and checked for null before indexing. The `parseContractionRange` tests call
the static method directly.
## Note for reviewers
`DownloadUtil.downloadModel(...)` is public. Callers that caught
`IOException` to handle checksum failures will now get an unchecked exception
instead, which shows up through the `*ME(String language)` constructors.
## Tests
`TokenSampleStreamTest`, `DownloadUtilCacheIntegrityTest` and
`ConlluStreamTest` pass, and checkstyle is clean.
> RegEx removal part 9: Correct the output of the regex sites in
> TokenSampleStream, ParserTool, ConlluStream, DownloadUtil and
> FeatureGeneratorUtil
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1975
> URL: https://issues.apache.org/jira/browse/OPENNLP-1975
> Project: OpenNLP
> Issue Type: Bug
> Reporter: Richard Zowalla
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Split out of OPENNLP\-1928 \(PR #1275\). Depends on OPENNLP\-1928 \(RegEx
> removal part 1\).
> Replaces the regular expressions of these sites with scans that follow the
> intended format instead of copying the regex quirks. The output changes:*
> {{TokenSampleStream}} \(English\): tokens are split on Unicode White\_Space;
> a leading space no longer yields an empty first token; a line without a token
> resets the open\-quote state; a token with a letter or digit of any script is
> a word.
> * {{ParserTool}}: brackets that follow each other are each separated
> \({{a\)\)b}} gives {{a \) \) b}}\).
> * {{ConlluStream}}: a malformed multiword token id \({{3\-}}, {{\-4}},
> {{3\-\-4}}, {{1\-2\-3}}, {{4\-3}}\) fails with an {{InvalidFormatException}}
> naming the id.
> * {{DownloadUtil}}: a blank checksum file is an {{IOException}} saying so.
> * {{FeatureGeneratorUtil}}: the {{cp}} class requires exactly one capital and
> one period.
> Before merge: tokenizer eval build that reads through {{TokenSampleStream}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
