[
https://issues.apache.org/jira/browse/OPENNLP-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Zowalla updated OPENNLP-1930:
-------------------------------------
Fix Version/s: (was: 2.5.13)
> RegEx removal part 2: Parse Arvores Deitadas markup with cursor scans
> ----------------------------------------------------------------------
>
> Key: OPENNLP-1930
> URL: https://issues.apache.org/jira/browse/OPENNLP-1930
> Project: OpenNLP
> Issue Type: Improvement
> Components: Formats
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Implement cursor-based character scanning for the Arvores Deitadas (AD)
> corpus readers using {{StringUtil}} helpers, removing regular expression
> parsing in the format reader.
> h2. Details
> * Implement cursor scans in {{ADSentenceStream.SentenceParser}} for node,
> leaf, and fallback line parsing.
> * Match markup tags ({{<s>}}, {{<p>}}, {{<t>}}, {{<caixa>}}, {{<ext>}}) by
> exact tag name, and terminate quoted lemmas at the first matching quote at a
> field boundary.
> * Add package-private {{ADMetadata}} to extract text ID, paragraph ID, and
> source metadata for {{ADNameSampleStream}} and {{ADSentenceSampleStream}}.
> * Update {{ADNameSampleStream}} and {{ADPOSSampleStream}} to use cursor
> scanning for multiword underscore tokens, contractions, and POS tag joining.
> * Close the detokenizer dictionary input stream in
> {{DetokenizerSampleStreamFactory}} after loading.
> * Document AD format parsing contracts in
> {{opennlp-docs/src/docbkx/corpora.xml}}.
> h2. Validation
> * Unit tests pass in {{ADSentenceStreamTest}}, {{ADMetadataTest}},
> {{ADNameSampleStreamTest}}, {{ADPOSSampleStreamTest}}, and
> {{ADSentenceSampleStreamTest}}.
> * Verified training data hashes and Portuguese tokenizer 10-fold
> cross-validation in {{ArvoresDeitadasEval}} across Bosque and FlorestaVirgem
> corpora.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
