[
https://issues.apache.org/jira/browse/OPENNLP-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristian Rickert updated OPENNLP-1930:
--------------------------------------
Summary: RegEx removal part 2: Parse Arvores Deitadas markup with cursor
scans (was: RegEx removal part 2: Arvores Deitadas markup parsing)
> RegEx removal part 2: Parse Arvores Deitadas markup with cursor scans
> ----------------------------------------------------------------------
>
> Key: OPENNLP-1930
> URL: https://issues.apache.org/jira/browse/OPENNLP-1930
> Project: OpenNLP
> Issue Type: Improvement
> Components: Formats
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 2.5.13, 3.0.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Replaces the remaining regular expressions in the AD corpus readers with
> cursor parsers, keeping the exact accept set of each pattern.
> ||Class||Patterns removed||
> |{{ADSentenceStream.SentenceParser}}|node, leaf, and bizarre-leaf line
> patterns; the {{\w.{*}?[\.<>].{*}}} lexeme check|
> |{{ADSentenceStream}}|the nine sentence, title, box, paragraph, and text tag
> patterns matched per line|
> |{{ADNameSampleStream}}|the three corpus-specific metadata patterns (text id,
> paragraph, source)|
> A package-private {{ADMetadata}} holds the metadata scan shared by
> {{ADNameSampleStream}} and {{{}ADSentenceSampleStream{}}}; the copy from part
> 1 in {{ADSentenceSampleStream}} is removed, and the line-terminator check the
> regexes had (a dot does not cross a line terminator) is restored there.
> Verification: a side-by-side harness ran the original patterns and the new
> parsers over the {{ad.sample}} corpus, hand-written edge lines, random
> mutations, and random strings, about 911,000 lines for the tree parser,
> 300,000 for the tag checks, and 500,000 for the metadata, with zero
> differences. {{ADSentenceStreamTest}} grows from 36 to 83 cases,
> {{ADNameSampleStreamTest}} from 58 to 64, and {{ADMetadataTest}} adds 61.
> Out of scope: {{PortugueseContractionUtility.split("_")}} is a
> single-character literal split and compiles no pattern.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
