attilapiros opened a new pull request, #56006:
URL: https://github.com/apache/spark/pull/56006
### What changes were proposed in this pull request?
Avoiding the `java.lang.UnsupportedOperationException` for `spark-shell
--remote` on JDK25.
This PR is simply adding `--sun-misc-unsafe-memory-access=allow` to the java
options which does not cause any issue for earlier JDK versions as
`-XX:+IgnoreUnrecognizedVMOptions` is already used.
### Why are the changes needed?
On JDK 25 when `spark-shell` is started with `--remote` flag we get an
exception at the start:
```
$ ./bin/spark-shell --remote sc://localhost:15002
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: package sun.security.action not in java.base
26/05/19 17:38:31 INFO BaseAllocator: Debug mode disabled. Enable with the
VM option -Darrow.memory.debug.allocator=true.
26/05/19 17:38:31 INFO DefaultAllocationManagerOption: allocation manager
type not specified, using netty as the default type
26/05/19 17:38:31 INFO CheckAllocator: Using DefaultAllocationManager at
memory/netty/DefaultAllocationManagerFactory.class
Exception in thread "main" java.lang.ExceptionInInitializerError
at
org.sparkproject.org.apache.arrow.memory.netty.DefaultAllocationManagerFactory.<clinit>(DefaultAllocationManagerFactory.java:26)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:467)
at java.base/java.lang.Class.forName(Class.java:458)
at
org.sparkproject.org.apache.arrow.memory.DefaultAllocationManagerOption.getFactory(DefaultAllocationManagerOption.java:105)
at
org.sparkproject.org.apache.arrow.memory.DefaultAllocationManagerOption.getDefaultAllocationManagerFactory(DefaultAllocationManagerOption.java:92)
at
org.sparkproject.org.apache.arrow.memory.BaseAllocator$Config.getAllocationManagerFactory(BaseAllocator.java:826)
at
org.sparkproject.org.apache.arrow.memory.ImmutableConfig.access$001(ImmutableConfig.java:20)
at
org.sparkproject.org.apache.arrow.memory.ImmutableConfig$InitShim.getAllocationManagerFactory(ImmutableConfig.java:80)
at
org.sparkproject.org.apache.arrow.memory.ImmutableConfig.<init>(ImmutableConfig.java:43)
at
org.sparkproject.org.apache.arrow.memory.ImmutableConfig$Builder.build(ImmutableConfig.java:492)
at
org.sparkproject.org.apache.arrow.memory.BaseAllocator.<clinit>(BaseAllocator.java:72)
at
org.apache.spark.sql.connect.SparkSession.<init>(SparkSession.scala:89)
at
org.apache.spark.sql.connect.SparkSession$Builder.tryCreateSessionFromClient(SparkSession.scala:1059)
at
org.apache.spark.sql.connect.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:1119)
at
org.apache.spark.sql.connect.SparkSession$.withLocalConnectServer(SparkSession.scala:949)
at
org.apache.spark.sql.connect.SparkSession$Builder.getOrCreate(SparkSession.scala:1118)
at
org.apache.spark.sql.application.ConnectRepl$.$anonfun$doMain$1(ConnectRepl.scala:91)
at
org.apache.spark.sql.connect.SparkSession$.withLocalConnectServer(SparkSession.scala:949)
at
org.apache.spark.sql.application.ConnectRepl$.doMain(ConnectRepl.scala:68)
at
org.apache.spark.sql.application.ConnectRepl$.main(ConnectRepl.scala:58)
at org.apache.spark.sql.application.ConnectRepl.main(ConnectRepl.scala)
at
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:565)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1033)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:226)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:95)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1171)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1180)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.UnsupportedOperationException
at
org.sparkproject.io.netty.buffer.EmptyByteBuf.memoryAddress(EmptyByteBuf.java:961)
at
org.sparkproject.io.netty.buffer.DuplicatedByteBuf.memoryAddress(DuplicatedByteBuf.java:115)
at
org.sparkproject.io.netty.buffer.UnsafeDirectLittleEndian.<init>(UnsafeDirectLittleEndian.java:45)
at
org.sparkproject.io.netty.buffer.PooledByteBufAllocatorL.<init>(PooledByteBufAllocatorL.java:47)
at
org.sparkproject.org.apache.arrow.memory.netty.NettyAllocationManager.<clinit>(NettyAllocationManager.java:54)
... 32 more
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Running the `spark-shell --remote` locally and `AmmoniteReplE2ESuite` on JDK
25 as:
```
./build/mvn test -pl sql/connect/client/jvm
-Dsuites="*.AmmoniteReplE2ESuite"
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
