[
https://issues.apache.org/jira/browse/OPENNLP-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121644#comment-18121644
]
ASF GitHub Bot commented on OPENNLP-1975:
-----------------------------------------
krickert commented on code in PR #1314:
URL: https://github.com/apache/opennlp/pull/1314#discussion_r4158377147
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/featuregen/FeatureGeneratorUtil.java:
##########
@@ -110,4 +107,21 @@ else if (pattern.isInitialCapitalLetter()) {
return (feat);
}
+
+ /**
+ * Tests for one uppercase code point followed by a period, and nothing
else. Any script
+ * counts, including the supplementary planes.
+ *
+ * @param token The token.
+ * @return {@code true} for exactly that shape.
+ */
+ private static boolean isCapPeriod(String token) {
+ if (token.isEmpty()) {
+ return false;
+ }
+ final int first = token.codePointAt(0);
+ final int periodOffset = Character.charCount(first);
+ return Character.isUpperCase(first) && periodOffset + 1 == token.length()
Review Comment:
Pushed the switch.
In the default mode a single capital of any script followed by a period is
`cp`. Under `-Dopennlp.compat.mode=LEGACY` the old set applies, `A` to `Z` plus
`Ä` `Ö` `Ü`, including the one trailing line terminator the old `$` accepted,
so a model trained before 3.0.0 gets what it was trained with.
I had a test written first, then applied change; the legacy rows pin `É.`,
`Ω.`, `Я.` as `ic` and the line-terminator cases.
For the models in the eval suite nothing moves, since their initials are
ASCII or umlauts. The only tokens that change class are initials from other
scripts, like `Я.`, and only in the default mode.
This is the same approach as the emoji normalizer in #1313: we made Unicode
is default for 3.0, and the switch is what keeps old models whole.
An ASCII-only rule here would be the one exception to that, which is why I
pushed back.
> RegEx removal part 9: Correct the output of the regex sites in
> TokenSampleStream, ParserTool, ConlluStream, DownloadUtil and
> FeatureGeneratorUtil
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1975
> URL: https://issues.apache.org/jira/browse/OPENNLP-1975
> Project: OpenNLP
> Issue Type: Bug
> Reporter: Richard Zowalla
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Split out of OPENNLP\-1928 \(PR #1275\). Depends on OPENNLP\-1928 \(RegEx
> removal part 1\).
> Replaces the regular expressions of these sites with scans that follow the
> intended format instead of copying the regex quirks. The output changes:*
> {{TokenSampleStream}} \(English\): tokens are split on Unicode White\_Space;
> a leading space no longer yields an empty first token; a line without a token
> resets the open\-quote state; a token with a letter or digit of any script is
> a word.
> * {{ParserTool}}: brackets that follow each other are each separated
> \({{a\)\)b}} gives {{a \) \) b}}\).
> * {{ConlluStream}}: a malformed multiword token id \({{3\-}}, {{\-4}},
> {{3\-\-4}}, {{1\-2\-3}}, {{4\-3}}\) fails with an {{InvalidFormatException}}
> naming the id.
> * {{DownloadUtil}}: a blank checksum file is an {{IOException}} saying so.
> * {{FeatureGeneratorUtil}}: the {{cp}} class requires exactly one capital and
> one period.
> Before merge: tokenizer eval build that reads through {{TokenSampleStream}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
