Jon Marius Venstad created OPENNLP-1520: -------------------------------------------
Summary: Generated Java code for stemmers is broken, and should be re-generated Key: OPENNLP-1520 URL: https://issues.apache.org/jira/browse/OPENNLP-1520 Project: OpenNLP Issue Type: Bug Components: Stemmer Affects Versions: 2.3.0 Reporter: Jon Marius Venstad The recursive stemming, which seems hard to actually trigger, but which is the intended usage of the {{methodObject }}and {{method}} in the {{Among}} class (called reflectively) is completely broken. First off, it tries to invoke a private method from outside the class (from a parent class, the {{{}SnowballProgram{}}}), which fails with an illegal access exception; if that worked, it would also have invoked _all_ such method calls on the _same, shared, static object_—not on the relevant stemmer instance. This was fixed 8 years ago, but it looks like the generated code in the opennlp-tools is 10 years old. I would urge you to re-generate that code. Commit that fixed the Java code generation: [https://github.com/snowballstem/snowball/commit/0f9d3d64ab965447a7f638b8ededc924f3efca75] Relevant sample stemmer with broken Java: [https://github.com/apache/opennlp/blob/main/opennlp-tools/src/main/java/opennlp/tools/stemmer/snowball/finnishStemmer.java] Stack trace showing illegal reflection access: {noformat} 2023-10-26 23:21:44.200 class opennlp.tools.stemmer.snowball.SnowballProgram cannot access a member of class opennlp.tools.stemmer.snowball.finnishStemmer with modifiers "private" exception=java.lang.IllegalAccessException: class opennlp.tools.stemmer.snowball.SnowballProgram cannot access a member of class opennlp.tools.stemmer.snowball.finnishStemmer with modifiers "private"at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:392) at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:674) at java.base/java.lang.reflect.Method.invoke(Method.java:560) at opennlp.tools.stemmer.snowball.SnowballProgram.find_among_b(SnowballProgram.java:353) at opennlp.tools.stemmer.snowball.finnishStemmer.r_case_ending(finnishStemmer.java:480) at opennlp.tools.stemmer.snowball.finnishStemmer.stem(finnishStemmer.java:1003) at opennlp.tools.stemmer.snowball.SnowballStemmer.stem(SnowballStemmer.java:131) at com.yahoo.language.opennlp.OpenNlpTokenizer.processToken(OpenNlpTokenizer.java:64) at com.yahoo.language.opennlp.OpenNlpTokenizer.lambda$tokenize$0(OpenNlpTokenizer.java:54) at com.yahoo.language.simple.SimpleTokenizer.tokenize(SimpleTokenizer.java:74) at com.yahoo.language.opennlp.OpenNlpTokenizer.tokenize(OpenNlpTokenizer.java:54) at com.yahoo.vespa.indexinglanguage.linguistics.LinguisticsAnnotator.annotate(LinguisticsAnnotator.java:76) ...{noformat} Best, Jon Marius Venstad, developer at [vespa.ai|http://vespa.ai/] -- This message was sent by Atlassian Jira (v8.20.10#820010)