[ https://issues.apache.org/jira/browse/SPARK-35557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404182#comment-17404182 ]
Yang Jie edited comment on SPARK-35557 at 8/25/21, 5:43 AM: ------------------------------------------------------------ [~iemejia] I've been doing similar work recently: * Scala 2.13 and JDK 17-ea are easier to use together. We can pass most UTs by adding some ` --add-opens' rules, I think maybe we can try to use these args as the default for `spark.driver.extraJavaOptions` and `spark.executor.extraJavaOptions` when use JDK 17 * Scala 2.12 doesn't seem to fit well because there are some interfaces mismatch when using the reflection API with JDK 17-ea was (Author: luciferyang): [~iemejia] I've been doing similar work recently: * Scala 2.13 and JDK 17-ea are easier to use together. We can pass most UTs by adding some ` --add-opens' rules. * Scala 2.12 doesn't seem to fit well because there are some interfaces mismatch when using the reflection API with JDK 17-ea > Adapt uses of JDK 17 Internal APIs > ---------------------------------- > > Key: SPARK-35557 > URL: https://issues.apache.org/jira/browse/SPARK-35557 > Project: Spark > Issue Type: Sub-task > Components: Spark Core > Affects Versions: 3.2.0 > Reporter: Ismaël Mejía > Priority: Major > > I tried to run a Spark pipeline using the most recent 3.2.0-SNAPSHOT with > Spark 2.12.4 on Java 17 and I found this exception: > {code:java} > java.lang.ExceptionInInitializerError > at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit> > (ByteArrayMethods.java:54) > at org.apache.spark.internal.config.package$.<clinit> (package.scala:1149) > at org.apache.spark.SparkConf$.<clinit> (SparkConf.scala:654) > at org.apache.spark.SparkConf.contains (SparkConf.scala:455) > ... > Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make > private java.nio.DirectByteBuffer(long,int) accessible: module java.base does > not "opens java.nio" to unnamed module @110df513 > at java.lang.reflect.AccessibleObject.checkCanSetAccessible > (AccessibleObject.java:357) > at java.lang.reflect.AccessibleObject.checkCanSetAccessible > (AccessibleObject.java:297) > at java.lang.reflect.Constructor.checkCanSetAccessible (Constructor.java:188) > at java.lang.reflect.Constructor.setAccessible (Constructor.java:181) > at org.apache.spark.unsafe.Platform.<clinit> (Platform.java:56) > at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit> > (ByteArrayMethods.java:54) > at org.apache.spark.internal.config.package$.<clinit> (package.scala:1149) > at org.apache.spark.SparkConf$.<clinit> (SparkConf.scala:654) > at org.apache.spark.SparkConf.contains (SparkConf.scala:455)}} > {code} > It seems that Java 17 will be more strict about uses of JDK Internals > [https://openjdk.java.net/jeps/403] -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org