[
https://issues.apache.org/jira/browse/OPENNLP-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116701#comment-18116701
]
Richard Zowalla commented on OPENNLP-1928:
------------------------------------------
Split up: this issue now covers only the rewrites whose output is unchanged
\(PR #1311\). CompatibilityMode moved to OPENNLP\-1973 \(#1312\), the
EmojiCharSequenceNormalizer fix to OPENNLP\-1974 \(#1313\), the output
corrections of TokenSampleStream, ParserTool, ConlluStream, DownloadUtil and
FeatureGeneratorUtil to OPENNLP\-1975 \(#1314\), and the AD reader changes to
OPENNLP\-1930 \(#1276\).
> RegEx removal part 1: Deterministic text scans replace trivial regular
> expressions
> -----------------------------------------------------------------------------------
>
> Key: OPENNLP-1928
> URL: https://issues.apache.org/jira/browse/OPENNLP-1928
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 2.5.13, 3.0.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Regex isn't compatible with the correct whitespace characters and punctuation
> marks. Every language's parser slightly changes. This ticket focuses on all
> the trivial spots we found regex:
>
> ||Site||Pattern removed||
> |{{DefaultPOSContextGenerator}},
> {{DefaultLemmatizerContextGenerator}}|{{[A-Z]}}, {{[0-9]}} capital and digit
> detection|
> |{{FeatureGeneratorUtil}}|{{^[A-ZÄÖÜ]\.$}} capital-period check|
> |{{TokenPatternFeatureGenerator}}|{{[^a-zA-Z]}} non-letter check|
> |{{BioCodec}}, {{NameFinderME}}|{{(.+)-\w+}} typed-outcome split|
> |{{BrownCluster}}|tab split|
> |{{TokenSampleStream}}|{{\s+}} split and {{[A-Za-z0-9]}} check|
> |{{EmojiCharSequenceNormalizer}}|surrogate-range emoji pattern|
> |{{ConlluStream}}|token-id split on {{-}} and {{text_([a-z]\{2,3\})}}
> language comment|
> |{{ParserTool}}|the two untokenized-bracket spacing patterns|
> |{{DownloadUtil}}|checksum-line split and the {{<a href>}} link pattern|
> |{{ADNameSampleStream}}|tag, whitespace, underscore, hyphen, and alphanumeric
> patterns|
> |{{ADPOSSampleStream}}, {{ADSentenceSampleStream}},
> {{ADSentenceStream}}|whitespace, metadata, and punctuation patterns|
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
