Kristian Rickert created OPENNLP-1926:
-----------------------------------------
Summary: Replace incidental RegEx usage in production code with
cursor and code-point loops
Key: OPENNLP-1926
URL: https://issues.apache.org/jira/browse/OPENNLP-1926
Project: OpenNLP
Issue Type: Epic
Reporter: Kristian Rickert
Assignee: Kristian Rickert
OpenNLP's processing path is RegEx-free by policy; a genuine RegEx step (for
example {{RegexNameFinder}}) is the documented exception (OPENNLP-1925). An
audit of main found the following production classes touching
{{java.util.regex}}. This epic covers replacing every incidental usage with
cursor or code-point loops. Classes whose job IS pattern matching are exempt
and listed here only for completeness.
h3. In scope: incidental usage to replace
|| Class || Module || Regex usage || Effort ||
| {{DefaultLemmatizerContextGenerator}} | opennlp-runtime | {{[A-Z]}} /
{{[0-9]}} presence checks | trivial: code-point loop |
| {{DefaultPOSContextGenerator}} | opennlp-runtime | {{[A-Z]}} / {{[0-9]}}
presence checks | trivial: code-point loop |
| {{FeatureGeneratorUtil}} | opennlp-runtime | {{^[A-ZÄÖÜ]\.$}} | trivial:
length-2 check |
| {{TokenPatternFeatureGenerator}} | opennlp-runtime | {{[^a-zA-Z]}} strip |
trivial: code-point loop |
| {{BioCodec}} | opennlp-runtime | {{(.+)-\w+}} outcome split | trivial:
lastIndexOf('-') |
| {{NameFinderME}} | opennlp-runtime | {{(.+)-\w+}} outcome split | trivial:
lastIndexOf('-') |
| {{BrownCluster}} | opennlp-runtime | split on tab | trivial: cursor split |
| {{TokenSampleStream}} | opennlp-runtime | {{split("\\s+")}} | trivial: cursor
split |
| {{ConlluStream}} | opennlp-formats | {{split("-")}}, {{text_([a-z]{2,3})}} |
trivial: startsWith / substring |
| {{EmojiCharSequenceNormalizer}} | opennlp-runtime | surrogate-range replace |
trivial: code-point loop |
| {{ParserTool}} | opennlp-cli | parenthesis spacing | moderate: single cursor
scan |
| {{DownloadUtil}} | opennlp-runtime | {{split("\\s")}}, HTML link scrape |
moderate: cursor + tag scan |
| {{AbstractDL}} | opennlp-dl | ad-hoc JSON field scrape | moderate: cursor
parse |
| {{DocumentCategorizerConfig}} | opennlp-dl | ad-hoc JSON map scrape |
moderate: cursor parse |
| {{ADNameSampleStream}}, {{ADPOSSampleStream}}, {{ADSentenceSampleStream}},
{{ADSentenceStream}} | opennlp-formats | AD corpus markup parsing | moderate:
cursor parse |
| {{AbstractClassPathModelFinder}}, {{DirectoryModelFinder}},
{{SimpleClassPathModelFinder}} | opennlp-model-resolver | wildcard-to-regex
glob matching | moderate: hand-rolled glob |
h3. Exempt: genuine RegEx steps (no change)
|| Class || Module || Why exempt ||
| {{RegexNameFinder}}, {{RegexNameFinderFactory}} | opennlp-runtime | the class
IS a regex step (OPENNLP-1925 note added) |
| {{AncoraSpanishHeadRules}} | opennlp-runtime | head rules are regex strings
defined by the treebank spec |
| {{Parse}} | opennlp-api | Penn Treebank markup grammar |
| {{NameSample}} | opennlp-api | name-sample markup grammar |
Still determining:
|| Class || Module || Investigating ||
| {{TokenizerFactory}}, {{TokenizerME}}, {{TokSpanEventStream}},
{{lang.Factory}}, {{lang.en.TokenSampleStream}} | opennlp-runtime |
alphanumeric pattern comes from model training parameters, part of the API
contract |
h3. Rules for replacement
* Behavior must be identical; existing tests stay green, and any predicate
rewrite gets pinning tests for both the accept and reject side.
* Replacements use cursor and code-point loops only; case mapping goes through
{{StringUtil}}.
* One commit per class or cohesive group so each fix can be cherry-picked onto
its own ticket branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
