[
https://issues.apache.org/jira/browse/OPENNLP-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristian Rickert updated OPENNLP-1930:
--------------------------------------
Description:
Implement cursor-based character scanning for the Arvores Deitadas (AD) corpus
readers using {{StringUtil}} helpers, removing regular expression parsing in
the format reader.
h2. Details
* Implement cursor scans in {{ADSentenceStream.SentenceParser}} for node, leaf,
and fallback line parsing.
* Match markup tags ({{<s>}}, {{<p>}}, {{<t>}}, {{<caixa>}}, {{<ext>}}) by
exact tag name, and terminate quoted lemmas at the first matching quote at a
field boundary.
* Add package-private {{ADMetadata}} to extract text ID, paragraph ID, and
source metadata for {{ADNameSampleStream}} and {{ADSentenceSampleStream}}.
* Update {{ADNameSampleStream}} and {{ADPOSSampleStream}} to use cursor
scanning for multiword underscore tokens, contractions, and POS tag joining.
* Close the detokenizer dictionary input stream in
{{DetokenizerSampleStreamFactory}} after loading.
* Document AD format parsing contracts in
{{opennlp-docs/src/docbkx/corpora.xml}}.
h2. Validation
* Unit tests pass in {{ADSentenceStreamTest}}, {{ADMetadataTest}},
{{ADNameSampleStreamTest}}, {{ADPOSSampleStreamTest}}, and
{{ADSentenceSampleStreamTest}}.
* Verified training data hashes and Portuguese tokenizer 10-fold
cross-validation in {{ArvoresDeitadasEval}} across Bosque and FlorestaVirgem
corpora.
was:
Replaces the remaining regular expressions in the AD corpus readers with cursor
parsers, keeping the exact accept set of each pattern.
||Class||Patterns removed||
|{{ADSentenceStream.SentenceParser}}|node, leaf, and bizarre-leaf line
patterns; the {{\w.{*}?[\.<>].{*}}} lexeme check|
|{{ADSentenceStream}}|the nine sentence, title, box, paragraph, and text tag
patterns matched per line|
|{{ADNameSampleStream}}|the three corpus-specific metadata patterns (text id,
paragraph, source)|
A package-private {{ADMetadata}} holds the metadata scan shared by
{{ADNameSampleStream}} and {{{}ADSentenceSampleStream{}}}; the copy from part 1
in {{ADSentenceSampleStream}} is removed, and the line-terminator check the
regexes had (a dot does not cross a line terminator) is restored there.
Verification: a side-by-side harness ran the original patterns and the new
parsers over the {{ad.sample}} corpus, hand-written edge lines, random
mutations, and random strings, about 911,000 lines for the tree parser, 300,000
for the tag checks, and 500,000 for the metadata, with zero differences.
{{ADSentenceStreamTest}} grows from 36 to 83 cases, {{ADNameSampleStreamTest}}
from 58 to 64, and {{ADMetadataTest}} adds 61.
Out of scope: {{PortugueseContractionUtility.split("_")}} is a single-character
literal split and compiles no pattern.
> RegEx removal part 2: Parse Arvores Deitadas markup with cursor scans
> ----------------------------------------------------------------------
>
> Key: OPENNLP-1930
> URL: https://issues.apache.org/jira/browse/OPENNLP-1930
> Project: OpenNLP
> Issue Type: Improvement
> Components: Formats
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 2.5.13, 3.0.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Implement cursor-based character scanning for the Arvores Deitadas (AD)
> corpus readers using {{StringUtil}} helpers, removing regular expression
> parsing in the format reader.
> h2. Details
> * Implement cursor scans in {{ADSentenceStream.SentenceParser}} for node,
> leaf, and fallback line parsing.
> * Match markup tags ({{<s>}}, {{<p>}}, {{<t>}}, {{<caixa>}}, {{<ext>}}) by
> exact tag name, and terminate quoted lemmas at the first matching quote at a
> field boundary.
> * Add package-private {{ADMetadata}} to extract text ID, paragraph ID, and
> source metadata for {{ADNameSampleStream}} and {{ADSentenceSampleStream}}.
> * Update {{ADNameSampleStream}} and {{ADPOSSampleStream}} to use cursor
> scanning for multiword underscore tokens, contractions, and POS tag joining.
> * Close the detokenizer dictionary input stream in
> {{DetokenizerSampleStreamFactory}} after loading.
> * Document AD format parsing contracts in
> {{opennlp-docs/src/docbkx/corpora.xml}}.
> h2. Validation
> * Unit tests pass in {{ADSentenceStreamTest}}, {{ADMetadataTest}},
> {{ADNameSampleStreamTest}}, {{ADPOSSampleStreamTest}}, and
> {{ADSentenceSampleStreamTest}}.
> * Verified training data hashes and Portuguese tokenizer 10-fold
> cross-validation in {{ArvoresDeitadasEval}} across Bosque and FlorestaVirgem
> corpora.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
