Kristian Rickert created OPENNLP-1905:
-----------------------------------------
Summary: Default-locale case folding breaks Morfologik dictionary
lookup
Key: OPENNLP-1905
URL: https://issues.apache.org/jira/browse/OPENNLP-1905
Project: OpenNLP
Issue Type: Bug
Reporter: Kristian Rickert
Assignee: Kristian Rickert
h2. Summary
Both Morfologik lookups fold the query word with the JVM default locale before
searching a prebuilt FSA dictionary. Under {{tr}} or {{az}} the Turkish dotless
i
means a capitalised word containing {{I}} folds to something the dictionary does
not contain, so the lookup misses. Nothing throws; lemmatization and POS tag
constraints silently stop working.
The dictionary is an external prebuilt artifact with fixed content, so a user
cannot work around this by editing their dictionary.
h2. Reproduction
Assertion failures observed against unfixed code with {{Locale.setDefault}} of
{{tr}}:
{noformat}
MorfologikLemmatizer.lemmatize expected <Illinois> but was <null>
MorfologikTagDictionary.getTags actual array was <null>
{noformat}
{{"Illinois".toLowerCase()}} under {{tr}} yields {{ıllinois}}, which is absent
from
an ASCII-lowercase FSA.
h2. Affected sites
|| Location || Expression ||
|
{{opennlp-extensions/opennlp-morfologik/src/main/java/opennlp/morfologik/lemmatizer/MorfologikLemmatizer.java:66}}
| {{lookup(word.toLowerCase())}} |
|
{{opennlp-extensions/opennlp-morfologik/src/main/java/opennlp/morfologik/tagdict/MorfologikTagDictionary.java:67}}
| {{word = word.toLowerCase()}} (case-insensitive mode) |
{{MorfologikTagDictionary}} is the more consequential of the two, because it
feeds
POS tagging constraints. Its failure does not merely lose a lemma, it changes
the
tagger's output depending on the JVM locale it runs under.
h2. Fix
Both folds become {{toLowerCase(Locale.ROOT)}}.
{{Locale.ROOT}} rather than the project's own {{StringUtil.toLowerCase}}: the
latter maps code points through {{Character.toLowerCase}}, which is context-free
and disagrees with {{String.toLowerCase(Locale.ROOT)}} on context-sensitive
mappings (Greek final sigma: {{ΟΔΟΣ}} folds to {{οδοσ}} rather than {{οδος}})
and
on one-to-many mappings such as {{ß}}. Existing FSA dictionaries were built
against
the English-locale fold, which is byte-identical to {{Locale.ROOT}}, so this
keeps
them working unchanged. The code-point fold would silently alter lookups against
already-published dictionaries.
h2. Test coverage
There was no locale test coverage anywhere in the project before this work.
There
is no {{Locale.setDefault}} in main or test sources.
{{MorfologikLocaleTest}} adds two tests, both verified to fail before the fix
and
pass after (the fix was stashed, the build rerun, the failure observed, then
restored). Failure messages are in the Reproduction section.
New fixture {{dictionaryLocaleSafety.txt}} and {{dictionaryLocaleSafety.info}}
in
the morfologik test resources, plus a {{createMorfologikDictionary(String)}}
overload on {{AbstractMorfologikTest}} (the existing no-arg method delegates to
it)
so the shared {{dictionaryWithLemma}} fixture is not perturbed.
h2. Build
{{mvn -o clean install}} over the full 23-module reactor: BUILD SUCCESS, zero
test
failures or errors, with checkstyle, rat and the existing forbiddenapis
configuration active.
h2. Note for the reviewer
No dictionary regeneration is required. That is the point of choosing
{{Locale.ROOT}} over the code-point fold. If a reviewer proposes switching to
{{StringUtil.toLowerCase}} for consistency with the rest of the codebase, that
would change lookup behaviour against every already-published FSA dictionary.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
