[
https://issues.apache.org/jira/browse/OPENNLP-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Zowalla closed OPENNLP-1898.
------------------------------------
Resolution: Fixed
> Address defects missed in OPENNLP-1876
> --------------------------------------
>
> Key: OPENNLP-1898
> URL: https://issues.apache.org/jira/browse/OPENNLP-1898
> Project: OpenNLP
> Issue Type: Bug
> Affects Versions: 3.0.0-M4
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0-M5
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h2. Description
> The review of OPENNLP-1876 ("Replace regex with cursor scans in the legacy
> CharSequenceNormalizers", PR #1151) raised follow-up notes on javadoc, test
> pattern
> handling, and the adversarial URL test. Those notes were addressed in a
> commit written on
> 2026-07-16 that was never pushed, and PR #1151 was merged later the same day
> without it.
> The work is therefore missing from {{{}main{}}}.
> The change is small and self-contained: 6 files, 68 insertions, 41 deletions.
> h2. What is missing from main
> h3. 1. Undocumented exceptions on public spellcheck API
> {{SpellCheckingCharSequenceNormalizer.normalize}} throws
> {{IllegalArgumentException}} when its input is {{null}} and
> {{IllegalStateException}} when no {{SpellChecker}} is attached, and two
> setters reject out-of-range values. None of it is documented. The follow-up
> adds the {{@throws}} tags so the contract is visible to callers, which is the
> convention the rest of the normalizer family already follows.
> h3. 2. Lost explanation on SpellCorrectingTokenStream
> The class javadoc no longer explains that the stream is token-count
> preserving _in contrast to_ {{SpellCorrectingObjectStream}} in compound mode,
> which is the property a caller relies on to keep tags and spans aligned with
> corrected tokens. The follow-up restores that contrast and the
> cross-reference.
> h3. 3. Four characterization tests recompile their reference patterns on a
> hot path
> The {{{}Number{}}}, {{{}Shrink{}}}, {{SocialMedia}} and {{Url}}
> characterization tests each call {{Pattern.compile}} inside the test body for
> the former regexes they differentially test against. These are the randomized
> tests that run 5000 generated inputs per case, so the patterns are recompiled
> per invocation. The follow-up hoists them into named {{static final}}
> constants, which also names what each pattern is instead of leaving a bare
> literal inside the assertion:
> - {{FORMER_DIGIT_REGEX}}
> - {{{}FORMER_SPACE_REGEX{}}}, {{FORMER_REPEATED_CHAR_REGEX}}
> - {{{}FORMER_HASH_USER_REGEX{}}}, {{{}FORMER_RT_REGEX{}}},
> {{FORMER_FACE_REGEX}}
> - {{{}FORMER_URL_REGEX{}}}, {{FORMER_MAIL_REGEX}}
> h3. 4. The adversarial URL test hides which input failed
> {{weirdUrlsMatchTheFormerRegexExactly}} drives a hard-coded {{String[]}} of
> adversarial URL shapes (userinfo, ports, percent escapes, IPv6 brackets,
> backslashes, IDN and punycode hosts) through a single {{{}@Test{}}}, so a
> regression reports only that the test failed, not which shape broke. The
> follow-up converts it to a {{@ParameterizedTest}} with
> {{{}@MethodSource{}}}, so the failing input is named in the report.
> h2. Verification
> Branch {{OPENNLP-1876-followup}} cherry-picks the original commit onto
> current {{main }}({{{}a8642301f{}}}); it applies without conflict. On that
> branch:
> - {{./mvnw -pl opennlp-extensions/opennlp-spellcheck,
> opennlpcore/opennlp-runtime -am test-compile}} succeeds
> - characterization suite: {*}55 tests, 0 failures, 0 errors{*}, including
> {{SocialMediaCharSequenceNormalizerCharacterizationTest}} (9) and
> {{UrlCharSequenceNormalizerCharacterizationTest}} (32)
> h2. Files touched
> {noformat}
> opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/normalizer/SpellCheckingCharSequenceNormalizer.java
> opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/stream/SpellCorrectingTokenStream.java
> opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/NumberCharSequenceNormalizerCharacterizationTest.java
> opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/ShrinkCharSequenceNormalizerCharacterizationTest.java
> opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/SocialMediaCharSequence-NormalizerCharacterizationTest.java
> opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/UrlCharSequenceNormalizerCharacterizationTest.java
> {noformat}
> h2. Related
> OPENNLP-1876 / PR #1151, merged 2026-07-16, which this completes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
