Kristian Rickert created OPENNLP-1984:
-----------------------------------------
Summary: Preserve hyphenated words by default in Arvores Deitadas
readers and detokenize them losslessly
Key: OPENNLP-1984
URL: https://issues.apache.org/jira/browse/OPENNLP-1984
Project: OpenNLP
Issue Type: Task
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Configure the Arvores Deitadas (AD) corpus readers to preserve hyphenated words
as single tokens by default, and provide lossless detokenization for
reconstructed samples.
h2. Details
* Set \{{splitHyphenatedTokens}} to default to \{{false}} in
\{{ADNameSampleStreamFactory}}. The \{{-splitHyphenatedTokens}} CLI option
remains available for splitting when needed.
* Add package-private \{{ADDetokenizer}} wrapping the configured detokenizer in
\{{ADTokenSampleStreamFactory}}. This handles hyphen joining for clitics (such
as \{{ofereceu-me}}) and ensures combining accents stay attached to base
characters across precomposed and decomposed forms.
* Update CLI and corpora documentation in \{{opennlp-docs}} covering the
\{{-splitHyphenatedTokens}} option and clitic detokenization rules.
h2. Compatibility
Training models with \{{-format ad}} without specifying
\{{-splitHyphenatedTokens}} retains hyphenated compounds as single tokens.
Callers requiring split hyphens can pass \{{-splitHyphenatedTokens true}}.
h2. Validation
* Verified via \{{ADHyphenatedTokenTest}}, covering hyphen preservation,
explicit splitting, clitics, and combining marks.
* 10-fold cross-validation in \{{ArvoresDeitadasEval}}
(\{{evalPortugueseTokenizerDefaultNaiveBayes}}) passes on FlorestaVirgem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
