[
https://issues.apache.org/jira/browse/OPENNLP-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristian Rickert updated OPENNLP-1927:
--------------------------------------
Description:
OpenNLP currently skips unsupported Hunspell directives, including {{ICONV}},
{{OCONV}}, {{COMPLEXPREFIXES}}, {{COMPOUNDRULE}}, {{IGNORE}}, and {{KEEPCASE}}.
Dictionaries can load successfully while producing different results from
Hunspell. This behavior is documented and was noted in the [review of PR
#1266|https://github.com/apache/opennlp/pull/1266#pullrequestreview-5122489404].
Implement the remaining dictionary behavior needed for Hunspell-compatible
stemming and morphological analysis. A valid dictionary using an unsupported
feature is a compatibility gap, not malformed input. Users should not need to
remove valid rules from their dictionaries.
h3. Acceptance criteria
* Audit the [Hunspell dictionary
format|https://github.com/hunspell/hunspell/blob/master/man/hunspell.5] and
identify missing behavior affecting stemming and analysis.
* Implement the six directives above and other identified gaps.
* Default to rejecting unsupported directives that could affect results, and
unknown directives whose effect cannot be established. Report the directive and
source location. Recognized metadata that does not affect these operations
should not cause rejection.
* Allow partial compatibility only through an explicit option that reports
skipped features. Continue rejecting malformed supported input in either mode.
* Verify results against a pinned Hunspell implementation using focused
fixtures and representative real dictionaries, respecting their licenses.
* Document supported behavior, loading options, and remaining limitations.
Spelling suggestions are separate functionality and are not part of this
stemmer ticket.
h3. Related work
* [OPENNLP-1893|https://issues.apache.org/jira/browse/OPENNLP-1893]: original
Hunspell stemmer support. Keep this issue closed.
* [PR #1266|https://github.com/apache/opennlp/pull/1266]: merged parsing and
analysis corrections. Release-note tracking for that merged work is separate
from the future compatibility work described here.
h3. Release notes
Hunspell dictionaries now load strictly by default:
HunspellDictionary.load(...) throws an IOException for an unsupported affix
directive. In 3.0.0-M6 unsupported directives were ignored. Pass
HunspellDictionary.LoadMode.ALLOW_PARTIAL to keep the earlier behaviour and
read the skipped directives through getUnsupportedDirectives().
HunspellStemmer.analyze is package-private in 3.0.0; the public API returns
stems.
A typed analysis enhancement is detailed in github issue #1
was:
OpenNLP currently skips unsupported Hunspell directives, including {{ICONV}},
{{OCONV}}, {{COMPLEXPREFIXES}}, {{COMPOUNDRULE}}, {{IGNORE}}, and {{KEEPCASE}}.
Dictionaries can load successfully while producing different results from
Hunspell. This behavior is documented and was noted in the [review of PR
#1266|https://github.com/apache/opennlp/pull/1266#pullrequestreview-5122489404].
Implement the remaining dictionary behavior needed for Hunspell-compatible
stemming and morphological analysis. A valid dictionary using an unsupported
feature is a compatibility gap, not malformed input. Users should not need to
remove valid rules from their dictionaries.
h3. Acceptance criteria
* Audit the [Hunspell dictionary
format|https://github.com/hunspell/hunspell/blob/master/man/hunspell.5] and
identify missing behavior affecting stemming and analysis.
* Implement the six directives above and other identified gaps.
* Default to rejecting unsupported directives that could affect results, and
unknown directives whose effect cannot be established. Report the directive and
source location. Recognized metadata that does not affect these operations
should not cause rejection.
* Allow partial compatibility only through an explicit option that reports
skipped features. Continue rejecting malformed supported input in either mode.
* Verify results against a pinned Hunspell implementation using focused
fixtures and representative real dictionaries, respecting their licenses.
* Document supported behavior, loading options, and remaining limitations.
Spelling suggestions are separate functionality and are not part of this
stemmer ticket.
h3. Related work
* [OPENNLP-1893|https://issues.apache.org/jira/browse/OPENNLP-1893]: original
Hunspell stemmer support. Keep this issue closed.
* [PR #1266|https://github.com/apache/opennlp/pull/1266]: merged parsing and
analysis corrections. Release-note tracking for that merged work is separate
from the future compatibility work described here.
> Complete Hunspell dictionary support for stemming and analysis
> --------------------------------------------------------------
>
> Key: OPENNLP-1927
> URL: https://issues.apache.org/jira/browse/OPENNLP-1927
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> OpenNLP currently skips unsupported Hunspell directives, including {{ICONV}},
> {{OCONV}}, {{COMPLEXPREFIXES}}, {{COMPOUNDRULE}}, {{IGNORE}}, and
> {{KEEPCASE}}. Dictionaries can load successfully while producing different
> results from Hunspell. This behavior is documented and was noted in the
> [review of PR
> #1266|https://github.com/apache/opennlp/pull/1266#pullrequestreview-5122489404].
> Implement the remaining dictionary behavior needed for Hunspell-compatible
> stemming and morphological analysis. A valid dictionary using an unsupported
> feature is a compatibility gap, not malformed input. Users should not need to
> remove valid rules from their dictionaries.
> h3. Acceptance criteria
> * Audit the [Hunspell dictionary
> format|https://github.com/hunspell/hunspell/blob/master/man/hunspell.5] and
> identify missing behavior affecting stemming and analysis.
> * Implement the six directives above and other identified gaps.
> * Default to rejecting unsupported directives that could affect results, and
> unknown directives whose effect cannot be established. Report the directive
> and source location. Recognized metadata that does not affect these
> operations should not cause rejection.
> * Allow partial compatibility only through an explicit option that reports
> skipped features. Continue rejecting malformed supported input in either mode.
> * Verify results against a pinned Hunspell implementation using focused
> fixtures and representative real dictionaries, respecting their licenses.
> * Document supported behavior, loading options, and remaining limitations.
> Spelling suggestions are separate functionality and are not part of this
> stemmer ticket.
> h3. Related work
> * [OPENNLP-1893|https://issues.apache.org/jira/browse/OPENNLP-1893]: original
> Hunspell stemmer support. Keep this issue closed.
> * [PR #1266|https://github.com/apache/opennlp/pull/1266]: merged parsing and
> analysis corrections. Release-note tracking for that merged work is separate
> from the future compatibility work described here.
> h3. Release notes
> Hunspell dictionaries now load strictly by default:
> HunspellDictionary.load(...) throws an IOException for an unsupported affix
> directive. In 3.0.0-M6 unsupported directives were ignored. Pass
> HunspellDictionary.LoadMode.ALLOW_PARTIAL to keep the earlier behaviour and
> read the skipped directives through getUnsupportedDirectives().
> HunspellStemmer.analyze is package-private in 3.0.0; the public API returns
> stems.
> A typed analysis enhancement is detailed in github issue #1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
