This is an automated email from the ASF dual-hosted git repository.
attilapiros pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 47e5fa9e9378 [SPARK-56955][CONNECT] Replace -Dio.netty.noUnsafe=false
with --sun-misc-unsafe-memory-access=allow for a more complete JDK 25+ support
47e5fa9e9378 is described below
commit 47e5fa9e9378dcaec13fd750760d6cf28cefffc6
Author: Attila Zsolt Piros <[email protected]>
AuthorDate: Wed May 20 10:45:51 2026 -0700
[SPARK-56955][CONNECT] Replace -Dio.netty.noUnsafe=false with
--sun-misc-unsafe-memory-access=allow for a more complete JDK 25+ support
### What changes were proposed in this pull request?
Avoiding the `java.lang.UnsupportedOperationException` for `spark-shell
--remote` on JDK25.
This PR is simply replaces `-Dio.netty.noUnsafe=false` with
`--sun-misc-unsafe-memory-access=allow`.
This new flag does not cause any issue for earlier JDK versions as
`-XX:+IgnoreUnrecognizedVMOptions` is already used.
### Why are the changes needed?
On JDK 25 when `spark-shell` is started with `--remote` flag we get an
exception at the start:
```
$ ./bin/spark-shell --remote sc://localhost:15002
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: package sun.security.action not in java.base
26/05/19 17:38:31 INFO BaseAllocator: Debug mode disabled. Enable with the
VM option -Darrow.memory.debug.allocator=true.
26/05/19 17:38:31 INFO DefaultAllocationManagerOption: allocation manager
type not specified, using netty as the default type
26/05/19 17:38:31 INFO CheckAllocator: Using DefaultAllocationManager at
memory/netty/DefaultAllocationManagerFactory.class
Exception in thread "main" java.lang.ExceptionInInitializerError
at
org.sparkproject.org.apache.arrow.memory.netty.DefaultAllocationManagerFactory.<clinit>(DefaultAllocationManagerFactory.java:26)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:467)
at java.base/java.lang.Class.forName(Class.java:458)
at
org.sparkproject.org.apache.arrow.memory.DefaultAllocationManagerOption.getFactory(DefaultAllocationManagerOption.java:105)
at
org.sparkproject.org.apache.arrow.memory.DefaultAllocationManagerOption.getDefaultAllocationManagerFactory(DefaultAllocationManagerOption.java:92)
at
org.sparkproject.org.apache.arrow.memory.BaseAllocator$Config.getAllocationManagerFactory(BaseAllocator.java:826)
at
org.sparkproject.org.apache.arrow.memory.ImmutableConfig.access$001(ImmutableConfig.java:20)
at
org.sparkproject.org.apache.arrow.memory.ImmutableConfig$InitShim.getAllocationManagerFactory(ImmutableConfig.java:80)
at
org.sparkproject.org.apache.arrow.memory.ImmutableConfig.<init>(ImmutableConfig.java:43)
at
org.sparkproject.org.apache.arrow.memory.ImmutableConfig$Builder.build(ImmutableConfig.java:492)
at
org.sparkproject.org.apache.arrow.memory.BaseAllocator.<clinit>(BaseAllocator.java:72)
at
org.apache.spark.sql.connect.SparkSession.<init>(SparkSession.scala:89)
at
org.apache.spark.sql.connect.SparkSession$Builder.tryCreateSessionFromClient(SparkSession.scala:1059)
at
org.apache.spark.sql.connect.SparkSession$Builder.$anonfun$getOrCreate$1(SparkSession.scala:1119)
at
org.apache.spark.sql.connect.SparkSession$.withLocalConnectServer(SparkSession.scala:949)
at
org.apache.spark.sql.connect.SparkSession$Builder.getOrCreate(SparkSession.scala:1118)
at
org.apache.spark.sql.application.ConnectRepl$.$anonfun$doMain$1(ConnectRepl.scala:91)
at
org.apache.spark.sql.connect.SparkSession$.withLocalConnectServer(SparkSession.scala:949)
at
org.apache.spark.sql.application.ConnectRepl$.doMain(ConnectRepl.scala:68)
at
org.apache.spark.sql.application.ConnectRepl$.main(ConnectRepl.scala:58)
at
org.apache.spark.sql.application.ConnectRepl.main(ConnectRepl.scala)
at
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:565)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1033)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:226)
at
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:95)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1171)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1180)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.UnsupportedOperationException
at
org.sparkproject.io.netty.buffer.EmptyByteBuf.memoryAddress(EmptyByteBuf.java:961)
at
org.sparkproject.io.netty.buffer.DuplicatedByteBuf.memoryAddress(DuplicatedByteBuf.java:115)
at
org.sparkproject.io.netty.buffer.UnsafeDirectLittleEndian.<init>(UnsafeDirectLittleEndian.java:45)
at
org.sparkproject.io.netty.buffer.PooledByteBufAllocatorL.<init>(PooledByteBufAllocatorL.java:47)
at
org.sparkproject.org.apache.arrow.memory.netty.NettyAllocationManager.<clinit>(NettyAllocationManager.java:54)
... 32 more
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Running the `spark-shell --remote` locally and `AmmoniteReplE2ESuite` on
JDK 25 as:
```
./build/mvn test -pl sql/connect/client/jvm
-Dsuites="*.AmmoniteReplE2ESuite"
```
And also testing all the `ArrowConvertersSuite` tests:
```
$ build/sbt "sql/testOnly *ArrowConvertersSuite"
...
[info] Run completed in 4 seconds, 719 milliseconds.
[info] Total number of tests run: 43
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 43, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 127 s (0:02:07.0), completed May 20, 2026, 6:31:55 AM
$ java --version
openjdk 25.0.3 2026-04-21 LTS
OpenJDK Runtime Environment Temurin-25.0.3+9 (build 25.0.3+9-LTS)
OpenJDK 64-Bit Server VM Temurin-25.0.3+9 (build 25.0.3+9-LTS, mixed mode,
sharing)
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #56006 from attilapiros/SPARK-56955.
Authored-by: Attila Zsolt Piros <[email protected]>
Signed-off-by: attilapiros <[email protected]>
(cherry picked from commit dc1fde3f0389f07f155a999abf518744ea26f4cf)
Signed-off-by: attilapiros <[email protected]>
---
launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java | 2 +-
pom.xml | 2 +-
project/SparkBuild.scala | 2 +-
sql/connect/bin/spark-connect-scala-client | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
diff --git
a/launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java
b/launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java
index ec3c030723ce..feab84380422 100644
--- a/launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java
+++ b/launcher/src/main/java/org/apache/spark/launcher/JavaModuleOptions.java
@@ -46,7 +46,7 @@ public class JavaModuleOptions {
"-Dio.netty.tryReflectionSetAccessible=true",
"-Dio.netty.allocator.type=pooled",
"-Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE",
- "-Dio.netty.noUnsafe=false",
+ "--sun-misc-unsafe-memory-access=allow",
"--enable-native-access=ALL-UNNAMED"};
/**
diff --git a/pom.xml b/pom.xml
index efc29699b93e..9dc3f77e39fe 100644
--- a/pom.xml
+++ b/pom.xml
@@ -340,7 +340,7 @@
-Dio.netty.tryReflectionSetAccessible=true
-Dio.netty.allocator.type=pooled
-Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE
- -Dio.netty.noUnsafe=false
+ --sun-misc-unsafe-memory-access=allow
--enable-native-access=ALL-UNNAMED
-XX:+EnableDynamicAgentLoading
</extraJavaTestArgs>
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 5d59e1141327..541960a1f5df 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1921,7 +1921,7 @@ object TestSettings {
"-Dio.netty.tryReflectionSetAccessible=true",
"-Dio.netty.allocator.type=pooled",
"-Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE",
- "-Dio.netty.noUnsafe=false",
+ "--sun-misc-unsafe-memory-access=allow",
"--enable-native-access=ALL-UNNAMED",
"-XX:+EnableDynamicAgentLoading").mkString(" ")
s"-Xmx$heapSize -Xss4m -XX:MaxMetaspaceSize=$metaspaceSize
-XX:ReservedCodeCacheSize=128m -Dfile.encoding=UTF-8 $extraTestJavaArgs"
diff --git a/sql/connect/bin/spark-connect-scala-client
b/sql/connect/bin/spark-connect-scala-client
index 019a42a2ba47..5748a7ec125f 100755
--- a/sql/connect/bin/spark-connect-scala-client
+++ b/sql/connect/bin/spark-connect-scala-client
@@ -72,7 +72,7 @@ JVM_ARGS="-XX:+IgnoreUnrecognizedVMOptions \
-Dio.netty.tryReflectionSetAccessible=true \
-Dio.netty.allocator.type=pooled \
-Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE \
- -Dio.netty.noUnsafe=false \
+ --sun-misc-unsafe-memory-access=allow \
--enable-native-access=ALL-UNNAMED \
$SCJVM_ARGS"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
