This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 4148fb5 [SPARK-37070][TEST] Pass all UTs in `mllib-local` and `mllib` with Java 17 4148fb5 is described below commit 4148fb58aada5bb7bc4835b39fe1baa07f9bacce Author: yangjie01 <yangji...@baidu.com> AuthorDate: Thu Oct 21 08:37:34 2021 -0700 [SPARK-37070][TEST] Pass all UTs in `mllib-local` and `mllib` with Java 17 ### What changes were proposed in this pull request? `Mockito` can't mock `j.u.Random` with Java 17 due to `module java.base does not export jdk.internal.util.random to unnamed module` and there are some UTs in `mllib-local` and `mllib` module failed with Java 17. So the main change of this pr is add `--add-exports=java.base/jdk.internal.util.random=ALL-UNNAMED` to `extraTestJavaArgs` to make all UTs passed in `mllib-local` and `mllib` module. ### Why are the changes needed? Pass UT with JDK 17 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass the Jenkins or GitHub Action - Manual test with Java 17 ``` mvn clean install -pl mllib-local mvn clean install -pl mllib ``` **Before** ``` Run completed in 714 milliseconds. Total number of tests run: 96 Suites: completed 9, aborted 0 Tests: succeeded 92, failed 4, canceled 0, ignored 0, pending 0 *** 4 TESTS FAILED *** Run completed in 10 minutes, 54 seconds. Total number of tests run: 1639 Suites: completed 206, aborted 0 Tests: succeeded 1635, failed 4, canceled 0, ignored 7, pending 0 *** 4 TESTS FAILED *** ``` All failed UTs has similar reasons: ``` - rand *** FAILED *** org.mockito.exceptions.base.MockitoException: Mockito cannot mock this class: class java.util.Random. Mockito can only mock non-private & non-final classes. If you're not sure why you're getting this error, please report to the mailing list. Java : 17 JVM vendor name : Azul Systems, Inc. JVM vendor version : 17+35-LTS JVM name : OpenJDK 64-Bit Server VM JVM version : 17+35-LTS JVM info : mixed mode, sharing OS name : Mac OS X OS version : 11.4 Underlying exception : java.lang.IllegalStateException: Cannot access annotation property public abstract boolean jdk.internal.util.random.RandomSupport$RandomGeneratorProperties.isStochastic() at org.scalatestplus.mockito.MockitoSugar.mock(MockitoSugar.scala:73) at org.scalatestplus.mockito.MockitoSugar.mock$(MockitoSugar.scala:72) at org.scalatestplus.mockito.MockitoSugar$.mock(MockitoSugar.scala:155) at org.apache.spark.ml.linalg.MatricesSuite.$anonfun$new$54(MatricesSuite.scala:807) at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226) ... Cause: java.lang.IllegalStateException: Cannot access annotation property public abstract boolean jdk.internal.util.random.RandomSupport$RandomGeneratorProperties.isStochastic() at net.bytebuddy.description.annotation.AnnotationDescription$ForLoadedAnnotation.getValue(AnnotationDescription.java:663) at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.handle(AnnotationAppender.java:298) at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.doAppend(AnnotationAppender.java:361) at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.append(AnnotationAppender.java:338) at net.bytebuddy.implementation.attribute.TypeAttributeAppender$ForInstrumentedType.apply(TypeAttributeAppender.java:93) at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForCreation.create(TypeWriter.java:5652) at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2166) at net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamicTypeBuilder.make(SubclassDynamicTypeBuilder.java:232) at net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamicTypeBuilder.make(SubclassDynamicTypeBuilder.java:204) at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase.make(DynamicType.java:3595) ... Cause: java.lang.IllegalAccessException: class net.bytebuddy.description.annotation.AnnotationDescription$ForLoadedAnnotation cannot access interface jdk.internal.util.random.RandomSupport$RandomGeneratorProperties (in module java.base) because module java.base does not export jdk.internal.util.random to unnamed module e55b13 at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:392) at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:674) at java.base/java.lang.reflect.Method.invoke(Method.java:560) at net.bytebuddy.description.annotation.AnnotationDescription$ForLoadedAnnotation.getValue(AnnotationDescription.java:642) at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.handle(AnnotationAppender.java:298) at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.doAppend(AnnotationAppender.java:361) at net.bytebuddy.implementation.attribute.AnnotationAppender$Default.append(AnnotationAppender.java:338) at net.bytebuddy.implementation.attribute.TypeAttributeAppender$ForInstrumentedType.apply(TypeAttributeAppender.java:93) at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForCreation.create(TypeWriter.java:5652) at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:2166) ... ``` **After** ``` Run completed in 1 second, 7 milliseconds. Total number of tests run: 96 Suites: completed 9, aborted 0 Tests: succeeded 96, failed 0, canceled 0, ignored 0, pending 0 All tests passed. Run completed in 10 minutes, 25 seconds. Total number of tests run: 1639 Suites: completed 206, aborted 0 Tests: succeeded 1639, failed 0, canceled 0, ignored 7, pending 0 All tests passed. ``` Closes #34355 from LuciferYang/SPARK-37070. Authored-by: yangjie01 <yangji...@baidu.com> Signed-off-by: Dongjoon Hyun <dongj...@apache.org> --- pom.xml | 6 ++++++ project/SparkBuild.scala | 6 +++++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index d2d4e8b..e6d47d7 100644 --- a/pom.xml +++ b/pom.xml @@ -285,6 +285,11 @@ <maven.build.timestamp.format>yyyy-MM-dd HH:mm:ss z</maven.build.timestamp.format> <!-- SPARK-36796 for JDK-17 test--> + <!-- + SPARK-37070 In order to enable the UTs in `mllib-local` and `mllib` to use `mockito` + to mock `j.u.Random`, "-add-exports=java.base/jdk.internal.util.random=ALL-UNNAMED" + is added. Should remove it when `mockito` can mock `j.u.Random` directly. + --> <extraJavaTestArgs> -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED @@ -298,6 +303,7 @@ --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED + --add-exports=java.base/jdk.internal.util.random=ALL-UNNAMED </extraJavaTestArgs> </properties> <repositories> diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala index 6d6ca4e..10c6dea 100644 --- a/project/SparkBuild.scala +++ b/project/SparkBuild.scala @@ -1138,7 +1138,11 @@ object TestSettings { "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED", "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED", "--add-opens=java.base/sun.security.action=ALL-UNNAMED", - "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED").mkString(" ") + "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED", + // SPARK-37070 In order to enable the UTs in `mllib-local` and `mllib` to use `mockito` + // to mock `j.u.Random`, "-add-exports=java.base/jdk.internal.util.random=ALL-UNNAMED" + // is added. Should remove it when `mockito` can mock `j.u.Random` directly. + "--add-exports=java.base/jdk.internal.util.random=ALL-UNNAMED").mkString(" ") s"-Xmx4g -Xss4m -XX:MaxMetaspaceSize=$metaspaceSize -XX:+UseParallelGC -XX:-UseDynamicNumberOfGCThreads -XX:ReservedCodeCacheSize=128m $extraTestJavaArgs" .split(" ").toSeq }, --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org