[
https://issues.apache.org/jira/browse/OPENNLP-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristian Rickert updated OPENNLP-1933:
--------------------------------------
Description:
Replace regex operations in Leipzig filename checks, Morfologik suffix
handling, and frequency-dictionary column parsing with character scans.
Preserve Unicode dictionary terms and supported integer-count representations,
removing the proposed ASCII-only count restriction. Keep space/tab column
separators and existing negative-count and overflow checks. Add focused
regression tests and document the supported format, referencing SymSpell's
dictionary documentation:
https://github.com/wolfgarbe/SymSpell#dictionary-file-format
was:
||Class||Call||Replacement||
|{{SimpleEventStreamBuilder}}, {{RealValueFileEventStream}},
{{RealBasicEventStream}}|{{split("\\s+")}}|{{StringUtil.splitOnAsciiWhitespace}}
from part 1|
|{{LeipzigLanguageSampleStream}}|{{matches("[a-z]+")}} on the file name
prefix|ASCII lowercase scan; the length 3 becomes a shared constant|
|{{MascNamedEntityParser}}, {{MascPennTagParser}}, {{MascWordParser}}|six
{{replaceFirst}} calls removing id prefixes|package-private
{{MascIdentifiers.removeFirst}}, first occurrence only|
|{{MorfologikDictionaryBuilder}}|anchored {{replaceAll}} of the metadata
suffix|{{endsWith}} and {{substring}}|
|{{FrequencyDictionaryLoader}}|{{[\t ]+}} column split|tab-and-space scan with
{{String.split}} semantics|
Each new helper is tested against the old call as an oracle, and each parser
has a test through its public surface. Single-character literal splits
elsewhere (comma, tab, space, semicolon) take the JDK fast path and compile
nothing, so they are not touched.
Not changed here: {{SpellCheckingCharSequenceNormalizer.URL_LIKE}} is a
multi-scheme URL and email heuristic and goes on the exempt list;
{{BasicContextGenerator}} is handled in its own bug ticket.
> RegEx removal part 5a: Remove regex from Leipzig filenames and dictionary
> loading
> ----------------------------------------------------------------------------------
>
> Key: OPENNLP-1933
> URL: https://issues.apache.org/jira/browse/OPENNLP-1933
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 2.5.13, 3.0.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Replace regex operations in Leipzig filename checks, Morfologik suffix
> handling, and frequency-dictionary column parsing with character scans.
> Preserve Unicode dictionary terms and supported integer-count
> representations, removing the proposed ASCII-only count restriction. Keep
> space/tab column separators and existing negative-count and overflow checks.
> Add focused regression tests and document the supported format, referencing
> SymSpell's dictionary documentation:
> https://github.com/wolfgarbe/SymSpell#dictionary-file-format
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
