Kristian Rickert created OPENNLP-1930:
-----------------------------------------
Summary: RegEx removal part 2: Arvores Deitadas markup parsing
Key: OPENNLP-1930
URL: https://issues.apache.org/jira/browse/OPENNLP-1930
Project: OpenNLP
Issue Type: Improvement
Components: Formats
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Replaces the remaining regular expressions in the AD corpus readers with cursor
parsers, keeping the exact accept set of each pattern.
||Class||Patterns removed||
|{{ADSentenceStream.SentenceParser}}|node, leaf, and bizarre-leaf line
patterns; the {{\w.{*}?[\.<>].{*}}} lexeme check|
|{{ADSentenceStream}}|the nine sentence, title, box, paragraph, and text tag
patterns matched per line|
|{{ADNameSampleStream}}|the three corpus-specific metadata patterns (text id,
paragraph, source)|
A package-private {{ADMetadata}} holds the metadata scan shared by
{{ADNameSampleStream}} and {{{}ADSentenceSampleStream{}}}; the copy from part 1
in {{ADSentenceSampleStream}} is removed, and the line-terminator check the
regexes had (a dot does not cross a line terminator) is restored there.
Verification: a side-by-side harness ran the original patterns and the new
parsers over the {{ad.sample}} corpus, hand-written edge lines, random
mutations, and random strings, about 911,000 lines for the tree parser, 300,000
for the tag checks, and 500,000 for the metadata, with zero differences.
{{ADSentenceStreamTest}} grows from 36 to 83 cases, {{ADNameSampleStreamTest}}
from 58 to 64, and {{ADMetadataTest}} adds 61.
Out of scope: {{PortugueseContractionUtility.split("_")}} is a single-character
literal split and compiles no pattern.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
