Kristian Rickert created OPENNLP-1898:
-----------------------------------------
Summary: Address defects missed in OPENNLP-1876
Key: OPENNLP-1898
URL: https://issues.apache.org/jira/browse/OPENNLP-1898
Project: OpenNLP
Issue Type: Bug
Affects Versions: 3.0.0-M4
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Fix For: 3.0.0-M5
h2. Description
The review of OPENNLP-1876 ("Replace regex with cursor scans in the legacy
CharSequenceNormalizers", PR #1151) raised follow-up notes on javadoc, test
pattern
handling, and the adversarial URL test. Those notes were addressed in a commit
written on
2026-07-16 that was never pushed, and PR #1151 was merged later the same day
without it.
The work is therefore missing from {{{}main{}}}.
The change is small and self-contained: 6 files, 68 insertions, 41 deletions.
h2. What is missing from main
h3. 1. Undocumented exceptions on public spellcheck API
{{SpellCheckingCharSequenceNormalizer.normalize}} throws
{{IllegalArgumentException}} when its input is {{null}} and
{{IllegalStateException}} when no {{SpellChecker}} is attached, and two setters
reject out-of-range values. None of it is documented. The follow-up adds the
{{@throws}} tags so the contract is visible to callers, which is the convention
the rest of the normalizer family already follows.
h3. 2. Lost explanation on SpellCorrectingTokenStream
The class javadoc no longer explains that the stream is token-count preserving
_in contrast to_ {{SpellCorrectingObjectStream}} in compound mode, which is the
property a caller relies on to keep tags and spans aligned with corrected
tokens. The follow-up restores that contrast and the cross-reference.
h3. 3. Four characterization tests recompile their reference patterns on a hot
path
The {{{}Number{}}}, {{{}Shrink{}}}, {{SocialMedia}} and {{Url}}
characterization tests each call {{Pattern.compile}} inside the test body for
the former regexes they differentially test against. These are the randomized
tests that run 5000 generated inputs per case, so the patterns are recompiled
per invocation. The follow-up hoists them into named {{static final}}
constants, which also names what each pattern is instead of leaving a bare
literal inside the assertion:
- {{FORMER_DIGIT_REGEX}}
- {{{}FORMER_SPACE_REGEX{}}}, {{FORMER_REPEATED_CHAR_REGEX}}
- {{{}FORMER_HASH_USER_REGEX{}}}, {{{}FORMER_RT_REGEX{}}},
{{FORMER_FACE_REGEX}}
- {{{}FORMER_URL_REGEX{}}}, {{FORMER_MAIL_REGEX}}
h3. 4. The adversarial URL test hides which input failed
{{weirdUrlsMatchTheFormerRegexExactly}} drives a hard-coded {{String[]}} of
adversarial URL shapes (userinfo, ports, percent escapes, IPv6 brackets,
backslashes, IDN and punycode hosts) through a single {{{}@Test{}}}, so a
regression reports only that the test failed, not which shape broke. The
follow-up converts it to a {{@ParameterizedTest}} with
{{{}@MethodSource{}}}, so the failing input is named in the report.
h2. Verification
Branch {{OPENNLP-1876-followup}} cherry-picks the original commit onto current
{{main }}({{{}a8642301f{}}}); it applies without conflict. On that branch:
- {{./mvnw -pl opennlp-extensions/opennlp-spellcheck,
opennlpcore/opennlp-runtime -am test-compile}} succeeds
- characterization suite: {*}55 tests, 0 failures, 0 errors{*}, including
{{SocialMediaCharSequenceNormalizerCharacterizationTest}} (9) and
{{UrlCharSequenceNormalizerCharacterizationTest}} (32)
h2. Files touched
{noformat}
opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/normalizer/SpellCheckingCharSequenceNormalizer.java
opennlp-extensions/opennlp-spellcheck/src/main/java/opennlp/spellcheck/stream/SpellCorrectingTokenStream.java
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/NumberCharSequenceNormalizerCharacterizationTest.java
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/ShrinkCharSequenceNormalizerCharacterizationTest.java
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/SocialMediaCharSequence-NormalizerCharacterizationTest.java
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/normalizer/UrlCharSequenceNormalizerCharacterizationTest.java
{noformat}
h2. Related
OPENNLP-1876 / PR #1151, merged 2026-07-16, which this completes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
