[
https://issues.apache.org/jira/browse/OPENNLP-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristian Rickert updated OPENNLP-1928:
--------------------------------------
Description:
Regex isn't compatible with the correct whitespace characters and punctuation
marks. Every language's parser slightly changes. This ticket focuses on all
the trivial spots we found regex:
||Site||Pattern removed||
|{{DefaultPOSContextGenerator}},
{{DefaultLemmatizerContextGenerator}}|{{[A-Z]}}, {{[0-9]}} capital and digit
detection|
|{{FeatureGeneratorUtil}}|{{^[A-ZÄÖÜ]\.$}} capital-period check|
|{{TokenPatternFeatureGenerator}}|{{[^a-zA-Z]}} non-letter check|
|{{BioCodec}}, {{NameFinderME}}|{{(.+)-\w+}} typed-outcome split|
|{{BrownCluster}}|tab split|
|{{TokenSampleStream}}|{{\s+}} split and {{[A-Za-z0-9]}} check|
|{{EmojiCharSequenceNormalizer}}|surrogate-range emoji pattern|
|{{ConlluStream}}|token-id split on {{-}} and {{text_([a-z]\{2,3\})}} language
comment|
|{{ParserTool}}|the two untokenized-bracket spacing patterns|
|{{DownloadUtil}}|checksum-line split and the {{<a href>}} link pattern|
|{{ADNameSampleStream}}|tag, whitespace, underscore, hyphen, and alphanumeric
patterns|
|{{ADPOSSampleStream}}, {{ADSentenceSampleStream}},
{{ADSentenceStream}}|whitespace, metadata, and punctuation patterns|
was:
Regex isn't compatible with the correct whitespace characters and punctuation
marks. Every language's parser slightly changes. This ticket focuses on all
the trivial spots we found regex:
> RegEx removal part 1: trival batch
> -----------------------------------
>
> Key: OPENNLP-1928
> URL: https://issues.apache.org/jira/browse/OPENNLP-1928
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Regex isn't compatible with the correct whitespace characters and punctuation
> marks. Every language's parser slightly changes. This ticket focuses on all
> the trivial spots we found regex:
>
> ||Site||Pattern removed||
> |{{DefaultPOSContextGenerator}},
> {{DefaultLemmatizerContextGenerator}}|{{[A-Z]}}, {{[0-9]}} capital and digit
> detection|
> |{{FeatureGeneratorUtil}}|{{^[A-ZÄÖÜ]\.$}} capital-period check|
> |{{TokenPatternFeatureGenerator}}|{{[^a-zA-Z]}} non-letter check|
> |{{BioCodec}}, {{NameFinderME}}|{{(.+)-\w+}} typed-outcome split|
> |{{BrownCluster}}|tab split|
> |{{TokenSampleStream}}|{{\s+}} split and {{[A-Za-z0-9]}} check|
> |{{EmojiCharSequenceNormalizer}}|surrogate-range emoji pattern|
> |{{ConlluStream}}|token-id split on {{-}} and {{text_([a-z]\{2,3\})}}
> language comment|
> |{{ParserTool}}|the two untokenized-bracket spacing patterns|
> |{{DownloadUtil}}|checksum-line split and the {{<a href>}} link pattern|
> |{{ADNameSampleStream}}|tag, whitespace, underscore, hyphen, and alphanumeric
> patterns|
> |{{ADPOSSampleStream}}, {{ADSentenceSampleStream}},
> {{ADSentenceStream}}|whitespace, metadata, and punctuation patterns|
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
