krickert commented on PR #1151: URL: https://github.com/apache/opennlp/pull/1151#issuecomment-4940182158
Thanks. Addressed in 55b8326a and 7d1a1ccf: - SpellCorrectingTokenStream split: the walk was rewritten to an indexOf scan that is output-identical to split(line, -1) on the quoted literal, and tests now cover leading, trailing, duplicate, and multi-character delimiters, all asserting the empty edge tokens survive the round trip. - JDK 21 boundary semantics: the word-boundary comment and the characterization suite now state that the pinned behavior is the JDK 21+ engine's and is tied to the project's Java baseline. - Null contract: kept IllegalArgumentException on the four rewritten normalizers and lifted the contract to the CharSequenceNormalizer interface javadoc, which now documents IAE-on-null and notes that implementations predating the contract are aligned as they are migrated. Aligning the untouched legacy siblings rides the tracked consistency work rather than this PR. - No-match copying: fixed. Every cursor pass in Twitter, Url, and Shrink allocates its builder only on the first actual change and returns the input instance otherwise, restoring the former replaceAll behavior on no-match String input; identity tests assert the uncopied return for each normalizer. Output is unchanged and the randomized differential suites still pass. - isNumberLike reject side: the classification is package-private and tested directly with the reject-side pins (5%%, +%, 1,2a, a lone sign, percent-only, marks-only) plus a 20k-token generated differential against the former regex. - Duplication: the ASCII whitespace set and case helpers are one shared package-private class used by both scans, and the four characterization suites share one input-generation and escape helper. - Self-referential javadoc openers: fixed in Shrink, Twitter, and Url, matching the Number fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
