Kristian Rickert created OPENNLP-1927:
-----------------------------------------
Summary: Complete Hunspell dictionary support for stemming and
analysis
Key: OPENNLP-1927
URL: https://issues.apache.org/jira/browse/OPENNLP-1927
Project: OpenNLP
Issue Type: Improvement
Reporter: Kristian Rickert
Assignee: Kristian Rickert
OpenNLP currently skips unsupported Hunspell directives, including {{ICONV}},
{{OCONV}}, {{COMPLEXPREFIXES}}, {{COMPOUNDRULE}}, {{IGNORE}}, and {{KEEPCASE}}.
Dictionaries can load successfully while producing different results from
Hunspell. This behavior is documented and was noted in the [review of PR
#1266|https://github.com/apache/opennlp/pull/1266#pullrequestreview-5122489404].
Implement the remaining dictionary behavior needed for Hunspell-compatible
stemming and morphological analysis. A valid dictionary using an unsupported
feature is a compatibility gap, not malformed input. Users should not need to
remove valid rules from their dictionaries.
h3. Acceptance criteria
* Audit the [Hunspell dictionary
format|https://github.com/hunspell/hunspell/blob/master/man/hunspell.5] and
identify missing behavior affecting stemming and analysis.
* Implement the six directives above and other identified gaps.
* Default to rejecting unsupported directives that could affect results, and
unknown directives whose effect cannot be established. Report the directive and
source location. Recognized metadata that does not affect these operations
should not cause rejection.
* Allow partial compatibility only through an explicit option that reports
skipped features. Continue rejecting malformed supported input in either mode.
* Verify results against a pinned Hunspell implementation using focused
fixtures and representative real dictionaries, respecting their licenses.
* Document supported behavior, loading options, and remaining limitations.
Spelling suggestions are separate functionality and are not part of this
stemmer ticket.
h3. Related work
* [OPENNLP-1893|https://issues.apache.org/jira/browse/OPENNLP-1893]: original
Hunspell stemmer support. Keep this issue closed.
* [PR #1266|https://github.com/apache/opennlp/pull/1266]: merged parsing and
analysis corrections. Release-note tracking for that merged work is separate
from the future compatibility work described here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
