[
https://issues.apache.org/jira/browse/OPENNLP-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Wiesner updated OPENNLP-1886:
------------------------------------
Fix Version/s: 3.0.0-M6
> UniNE light and minimal stemmer tiers with vocabulary parity fixtures
> ---------------------------------------------------------------------
>
> Key: OPENNLP-1886
> URL: https://issues.apache.org/jira/browse/OPENNLP-1886
> Project: OpenNLP
> Issue Type: New Feature
> Components: Sentence Detector
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0-M6, 3.0.0
>
>
> STACKED on [#1163|https://github.com/apache/opennlp/pull/1163]
> (stemmer-factory): this branch builds on that PR's StemmerFactory contract,
> and since that head lives on a fork it cannot be this PR's base ref. Until
> [#1163|https://github.com/apache/opennlp/pull/1163] merges, the diff here
> shows its commits too; only the last commit
> ([{{592380e}}|https://github.com/apache/opennlp/commit/592380e351ce0aa9d3f558ed10fbfecc95d85d99])
> is this PR. After [#1163|https://github.com/apache/opennlp/pull/1163] lands,
> the branch rebases onto main and the diff collapses to this change alone.
> Adds sixteen stemmers in a new opennlp.tools.stemmer.light package: light and
> minimal tiers for German, French, Spanish, Norwegian (Bokmaal and Nynorsk
> varieties), and Swedish, plus light stemmers for Finnish, Hungarian, Italian,
> Portuguese, and Russian, and the English minimal stemmer. These fill the tier
> between no stemming and the aggressive Snowball algorithms: they remove
> plural and inflectional endings with far less conflation. Adapted from Apache
> Lucene's analysis-common module; the UniNE algorithms carry Jacques Savoy's
> BSD notice in each source file and in the distribution LICENSE.
> Every stemmer is stateless and thread-safe, implements Stemmer, and is its
> own StemmerFactory. Input is expected lowercase, matching the algorithms'
> original contract; null input fails loudly.
> Parity is asserted against the original implementations: bundled fixtures
> sample the algorithms' vocabulary test data (up to 2000 word/stem pairs per
> stemmer, complete lists for Norwegian) regenerated by running the originals,
> so any behavioral drift fails the test. The Galician and Portuguese minimal
> stemmers are excluded because they build on the RSLP rule engine, which is
> its own effort.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
