[
https://issues.apache.org/jira/browse/OPENNLP-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Zowalla updated OPENNLP-1928:
-------------------------------------
Fix Version/s: 3.0.0
2.5.13
> RegEx removal part 1: Deterministic text scans replace trivial regular
> expressions
> -----------------------------------------------------------------------------------
>
> Key: OPENNLP-1928
> URL: https://issues.apache.org/jira/browse/OPENNLP-1928
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 2.5.13, 3.0.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Regex isn't compatible with the correct whitespace characters and punctuation
> marks. Every language's parser slightly changes. This ticket focuses on all
> the trivial spots we found regex:
>
> ||Site||Pattern removed||
> |{{DefaultPOSContextGenerator}},
> {{DefaultLemmatizerContextGenerator}}|{{[A-Z]}}, {{[0-9]}} capital and digit
> detection|
> |{{FeatureGeneratorUtil}}|{{^[A-ZÄÖÜ]\.$}} capital-period check|
> |{{TokenPatternFeatureGenerator}}|{{[^a-zA-Z]}} non-letter check|
> |{{BioCodec}}, {{NameFinderME}}|{{(.+)-\w+}} typed-outcome split|
> |{{BrownCluster}}|tab split|
> |{{TokenSampleStream}}|{{\s+}} split and {{[A-Za-z0-9]}} check|
> |{{EmojiCharSequenceNormalizer}}|surrogate-range emoji pattern|
> |{{ConlluStream}}|token-id split on {{-}} and {{text_([a-z]\{2,3\})}}
> language comment|
> |{{ParserTool}}|the two untokenized-bracket spacing patterns|
> |{{DownloadUtil}}|checksum-line split and the {{<a href>}} link pattern|
> |{{ADNameSampleStream}}|tag, whitespace, underscore, hyphen, and alphanumeric
> patterns|
> |{{ADPOSSampleStream}}, {{ADSentenceSampleStream}},
> {{ADSentenceStream}}|whitespace, metadata, and punctuation patterns|
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
