[
https://issues.apache.org/jira/browse/SPARK-35557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17441227#comment-17441227
]
Olivier Peyrusse commented on SPARK-35557:
------------------------------------------
Hello, everyone,
I see a similar issue when creating a basic local session
{{SparkSession.builder().appName("abc").config("spark.master",
"local").getOrCreate()}}, throwing because
bq. class org.apache.spark.storage.StorageUtils$ cannot access class
sun.nio.ch.DirectBuffer
Do you want me to open a specific ticket for this?
Link to the particular code:
https://github.com/apache/spark/blob/8ae88d01b46d581367d0047b50fcfb65078ab972/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala#L206-L223
>From what I see, in jdk17, we could directly call {{Unsafe#invokeCleaner}} as
>it accepts a ByteBuffer and internally casts it to a DirectBuffer. And this is
>the case since we call it with a {{MappedByteBuffer}} [2].
But it would require more tricks to make it work for jdk8 (certainly something
like loading the class from its name, using reflection to access the cleaner,
etc)
[2]
https://github.com/apache/spark/blob/8ae88d01b46d581367d0047b50fcfb65078ab972/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala#L234-L237
> Adapt uses of JDK 17 Internal APIs
> ----------------------------------
>
> Key: SPARK-35557
> URL: https://issues.apache.org/jira/browse/SPARK-35557
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.2.0
> Reporter: Ismaël Mejía
> Priority: Major
>
> I tried to run a Spark pipeline using the most recent 3.2.0-SNAPSHOT with
> Spark 2.12.4 on Java 17 and I found this exception:
> {code:java}
> java.lang.ExceptionInInitializerError
> at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit>
> (ByteArrayMethods.java:54)
> at org.apache.spark.internal.config.package$.<clinit> (package.scala:1149)
> at org.apache.spark.SparkConf$.<clinit> (SparkConf.scala:654)
> at org.apache.spark.SparkConf.contains (SparkConf.scala:455)
> ...
> Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make
> private java.nio.DirectByteBuffer(long,int) accessible: module java.base does
> not "opens java.nio" to unnamed module @110df513
> at java.lang.reflect.AccessibleObject.checkCanSetAccessible
> (AccessibleObject.java:357)
> at java.lang.reflect.AccessibleObject.checkCanSetAccessible
> (AccessibleObject.java:297)
> at java.lang.reflect.Constructor.checkCanSetAccessible (Constructor.java:188)
> at java.lang.reflect.Constructor.setAccessible (Constructor.java:181)
> at org.apache.spark.unsafe.Platform.<clinit> (Platform.java:56)
> at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit>
> (ByteArrayMethods.java:54)
> at org.apache.spark.internal.config.package$.<clinit> (package.scala:1149)
> at org.apache.spark.SparkConf$.<clinit> (SparkConf.scala:654)
> at org.apache.spark.SparkConf.contains (SparkConf.scala:455)}}
> {code}
> It seems that Java 17 will be more strict about uses of JDK Internals
> [https://openjdk.java.net/jeps/403]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
