[ https://issues.apache.org/jira/browse/HADOOP-18716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720086#comment-17720086 ]
Vinay Devadiga edited comment on HADOOP-18716 at 5/6/23 5:14 AM: ----------------------------------------------------------------- Hi [~ayushtkn] i dont have band width to raise PR , but here are most of the fixes explanation here are there tested it out . Any body can raise PR . h3. Appendix #1.The first major error which we encountered was the following stack trace - java.lang.NoClassDefFoundError: Could not initialize class com.google.inject.internal.cglib.core.$ReflectUtils at com.google.inject.internal.cglib.reflect.$FastClassEmitter.<init>(FastClassEmitter.java:67) at com.google.inject.internal.cglib.reflect.$FastClass$Generator.generateClass(FastClass.java:72) This error seems to be the lack of support of the current guice version 4.0 for jre 17 . so after upgrading the guice version to 5.1.0 reduced the {*}errors by the 300{*}. *Ref:* [https://github.com/google/guice/issues/1536] [https://github.com/google/guice/pull/1529/files] --<guice.version>4.0</guice.version> ++<guice.version>5.1.0</guice.version> ------------------------------------------------------------------------------------------------------ 2 . The second major error which we encountered the child jvms spawned by the maven do not have major add-opens which were causing various failures . ``` 1. java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts,TestJerseyClient 2. java.io.IOException: 0.0.0.0:43529: Unexpected HTTP response: code=500 != 200, op=MKDIRS, message=Server Error 4. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 6. java.lang.Exception: Unexpected exception, expected<org.apache.hadoop.yarn.webapp.WebAppException> but was<java.lang.NoClassDefFoundError> 7. org.apache.hadoop.service.ServiceStateException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues 8. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.ExceptionInInitializerError ``` and many more To fix this we use the arg line child property of <maven-surefire-plugin.argLine>-Xmx2048m -XX:+HeapDumpOnOutOfMemoryError</maven-surefire-plugin.argLine> We add the following - add-opens in the arg line and in the child projects --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.math=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED This fix brought down the errors by 2000. So the current count is {*}reduced to 500{*}. ------------------------------------------------------------------------------------------------------- 3. There were various mockito exceptions of these following stack trace java.lang.NullPointerException: Cannot enter synchronized block because "this.lock" is null Mockito cannot mock this class: class com.sun.jersey.api.client.ClientResponse. If you're not sure why you're getting this error, please report to the mailing list. Java : 17 JVM vendor name : Amazon.com Inc. JVM vendor version : 17.0.6+10-LTS JVM name : OpenJDK 64-Bit Server VM JVM version : 17.0.6+10-LTS JVM info : mixed mode, sharing OS name : Linux OS version : 4.14.309-231.529.amzn2.x86_64 You are seeing this disclaimer because Mockito is configured to create inlined mocks. You can learn about inline mocks and their limitations under item #39 of the Mockito class javadoc. Underlying exception : org.mockito.exceptions.base.MockitoException: Could not modify all classes [class com.sun.jersey.api.client.ClientResponse, class java.lang.Object] etc .These errors got fix after updating the mockito-core version and its scope <dependency> <groupId>org.mockito</groupId> <artifactId>mockito-core</artifactId> <version>2.28.2</version> <version>3.11.2</version> <scope>provided</scope> Other mockito exceptions are failing with following stack trace - Caused by: java.lang.UnsatisfiedLinkError: jdk.internal.misc.Unsafe.registerNatives()V at jdk.internal.misc.Unsafe.registerNatives(Native Method) at jdk.internal.misc.Unsafe.<clinit>(Unsafe.java:57) at java.base/java.lang.Class.forName0(Native Method) This is surely because we have not access to {{Unsafe}} . Ref: [https://github.com/jboss-javassist/javassist/issues/227] [https://github.com/mockito/mockito/issues/1207] [. https://github.com/mockito/mockito/issues/2164|https://github.com/mockito/mockito/issues/2164] This brought down *error failures to 200* . # There were some other minor errors like for [MetricsInvariantChecker resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/invariants/MetricsInvariantChecker.java] etc which used javascript engine for their testing, which is not supported in JRE 17 so the fix was add the following dependency and use graal.js as an alternative. Ref: [https://stackoverflow.com/questions/71481562/use-javascript-scripting-engine-in-java-17] <dependency> <groupId>org.graalvm.js</groupId> <artifactId>js</artifactId> <version>22.0.0</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.graalvm.js</groupId> <artifactId>js-scriptengine</artifactId> <version>22.0.0</version> <scope>provided</scope> </dependency> --this.scriptEngine = (Compilable) manager.getEngineByName("JavaScript"); ++this.scriptEngine = (Compilable) manager.getEngineByName("graal.js"); There were also some other failures - cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x59d016c9 This caused fixed by adding - --add-opens=java.base/sun.nio.ch=ALL-UNNAMED \ --add-opens=java.base/sun.nio.cs=ALL-UNNAMED \ Ref: [https://stackoverflow.com/questions/73465937/apache-spark-3-3-0-breaks-on-java-17-with-cannot-access-class-sun-nio-ch-direct] --------------------------------------------------------------------------------------------------- 5.The next set of errors which we fixed were IPC failures for these following classes TestIPC,TestAsyncIPC and TestRPCServerShutdown . We encountered the following stack trace - Caused by: java.lang.NoClassDefFoundError: org/bouncycastle/jce/provider/BouncyCastleProvider at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:244) ... 98 more Caused by: java.lang.ClassNotFoundException: org.bouncycastle.jce.provider.BouncyCastleProvider at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520) ... 99 more This was fixed by adding the following dependency which was missing JRE 17 runtime for Netty -{*}{{org.bouncycastle:bcpkix-jdk15on:1.65}}{*} Ref: [https://github.com/netty/netty/issues/10317] <dependency> <groupId>org.bouncycastle</groupId> <artifactId>bcprov-jdk15on</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.bouncycastle</groupId> <artifactId>bcpkix-jdk15on</artifactId> <scope>test</scope> </dependency> This reduced the {*}error count to 96{*}. --------------------------------------------------------------------------------------------------------- These are very rough numbers but this solves most of the issue . For some errors : fix is this have not tried it https://github.com/apache/hbase/pull/3443/files ---------------------------------------------------- Just trying to contribute to the community and my findings . was (Author: JIRAUSER290527): Hi [~ayushtkn] i dont have band width to raise PR , but here are most of the fixes explanation here are there tested it out . Any body can raise PR . h3. Appendix #1.The first major error which we encountered was the following stack trace - java.lang.NoClassDefFoundError: Could not initialize class com.google.inject.internal.cglib.core.$ReflectUtils at com.google.inject.internal.cglib.reflect.$FastClassEmitter.<init>(FastClassEmitter.java:67) at com.google.inject.internal.cglib.reflect.$FastClass$Generator.generateClass(FastClass.java:72) This error seems to be the lack of support of the current guice version 4.0 for jre 17 . so after upgrading the guice version to 5.1.0 reduced the {*}errors by the 300{*}. *Ref:* [https://github.com/google/guice/issues/1536] [https://github.com/google/guice/pull/1529/files] --<guice.version>4.0</guice.version> ++<guice.version>5.1.0</guice.version> ------------------------------------------------------------------------------------------------------ 2 . The second major error which we encountered the child jvms spawned by the maven do not have major add-opens which were causing various failures . ``` 1. java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts,TestJerseyClient 2. java.io.IOException: 0.0.0.0:43529: Unexpected HTTP response: code=500 != 200, op=MKDIRS, message=Server Error 4. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: 6. java.lang.Exception: Unexpected exception, expected<org.apache.hadoop.yarn.webapp.WebAppException> but was<java.lang.NoClassDefFoundError> 7. org.apache.hadoop.service.ServiceStateException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues 8. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.ExceptionInInitializerError ``` and many more To fix this we use the arg line child property of <maven-surefire-plugin.argLine>-Xmx2048m -XX:+HeapDumpOnOutOfMemoryError</maven-surefire-plugin.argLine> We add the following - add-opens in the arg line and in the child projects --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.math=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED This fix brought down the errors by 2000. So the current count is {*}reduced to 500{*}. ------------------------------------------------------------------------------------------------------- 3. There were various mockito exceptions of these following stack trace java.lang.NullPointerException: Cannot enter synchronized block because "this.lock" is null Mockito cannot mock this class: class com.sun.jersey.api.client.ClientResponse. If you're not sure why you're getting this error, please report to the mailing list. Java : 17 JVM vendor name : Amazon.com Inc. JVM vendor version : 17.0.6+10-LTS JVM name : OpenJDK 64-Bit Server VM JVM version : 17.0.6+10-LTS JVM info : mixed mode, sharing OS name : Linux OS version : 4.14.309-231.529.amzn2.x86_64 You are seeing this disclaimer because Mockito is configured to create inlined mocks. You can learn about inline mocks and their limitations under item #39 of the Mockito class javadoc. Underlying exception : org.mockito.exceptions.base.MockitoException: Could not modify all classes [class com.sun.jersey.api.client.ClientResponse, class java.lang.Object] etc .These errors got fix after updating the mockito-core version and its scope <dependency> <groupId>org.mockito</groupId> <artifactId>mockito-core</artifactId> <version>2.28.2</version> <version>3.11.2</version> <scope>provided</scope> Other mockito exceptions are failing with following stack trace - Caused by: java.lang.UnsatisfiedLinkError: jdk.internal.misc.Unsafe.registerNatives()V at jdk.internal.misc.Unsafe.registerNatives(Native Method) at jdk.internal.misc.Unsafe.<clinit>(Unsafe.java:57) at java.base/java.lang.Class.forName0(Native Method) This is surely because we have not access to {{Unsafe}} . Ref: [https://github.com/jboss-javassist/javassist/issues/227] [https://github.com/mockito/mockito/issues/1207] [. https://github.com/mockito/mockito/issues/2164|https://github.com/mockito/mockito/issues/2164] This brought down *error failures to 200* . # There were some other minor errors like for [MetricsInvariantChecker tests|https://code.amazon.com/packages/Aws157HadoopGit/blobs/4b4b79f64a65146acdc5586428e5438aa1639867/--/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/invariants/MetricsInvariantChecker.java] etc which used javascript engine for their testing, which is not supported in JRE 17 so the fix was add the following dependency and use graal.js as an alternative. Ref: [https://stackoverflow.com/questions/71481562/use-javascript-scripting-engine-in-java-17] <dependency> <groupId>org.graalvm.js</groupId> <artifactId>js</artifactId> <version>22.0.0</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.graalvm.js</groupId> <artifactId>js-scriptengine</artifactId> <version>22.0.0</version> <scope>provided</scope> </dependency> --this.scriptEngine = (Compilable) manager.getEngineByName("JavaScript"); ++this.scriptEngine = (Compilable) manager.getEngineByName("graal.js"); There were also some other failures - cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x59d016c9 This caused fixed by adding - --add-opens=java.base/sun.nio.ch=ALL-UNNAMED \ --add-opens=java.base/sun.nio.cs=ALL-UNNAMED \ Ref: [https://stackoverflow.com/questions/73465937/apache-spark-3-3-0-breaks-on-java-17-with-cannot-access-class-sun-nio-ch-direct] --------------------------------------------------------------------------------------------------- 5.The next set of errors which we fixed were IPC failures for these following classes TestIPC,TestAsyncIPC and TestRPCServerShutdown . We encountered the following stack trace - Caused by: java.lang.NoClassDefFoundError: org/bouncycastle/jce/provider/BouncyCastleProvider at io.netty.handler.ssl.util.SelfSignedCertificate.<init>(SelfSignedCertificate.java:244) ... 98 more Caused by: java.lang.ClassNotFoundException: org.bouncycastle.jce.provider.BouncyCastleProvider at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520) ... 99 more This was fixed by adding the following dependency which was missing JRE 17 runtime for Netty -{*}{{org.bouncycastle:bcpkix-jdk15on:1.65}}{*} Ref: [https://github.com/netty/netty/issues/10317] <dependency> <groupId>org.bouncycastle</groupId> <artifactId>bcprov-jdk15on</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.bouncycastle</groupId> <artifactId>bcpkix-jdk15on</artifactId> <scope>test</scope> </dependency> This reduced the {*}error count to 96{*}. --------------------------------------------------------------------------------------------------------- These are very rough numbers but this solves most of the issue . For some errors : fix is this have not tried it https://github.com/apache/hbase/pull/3443/files ---------------------------------------------------- Just trying to contribute to the community and my findings . > [JDK-17] Failed unit tests , with Java 17 runtime and compiled Java 8 > --------------------------------------------------------------------- > > Key: HADOOP-18716 > URL: https://issues.apache.org/jira/browse/HADOOP-18716 > Project: Hadoop Common > Issue Type: Bug > Reporter: Vinay Devadiga > Priority: Critical > > Compiled Hadoop - Hadoop branch 3.3.3 > mvn clean install - DskipTests=True > Java_Home -> points to Java-8 > maven version - 3.8.8 (Quite latest) > > Ran various whole test suit on my private cloud environment - > Changed Java_Home to Java-17 > > mvn surefire:test > > Out of 22k tests - 2.5 k tests failed . -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org