[
https://issues.apache.org/jira/browse/SPARK-56955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anupam Yadav updated SPARK-56955:
---------------------------------
Description:
On JDK 25, spark-shell --remote fails to start due to Arrow/Netty memory
allocator incompatibility.
*Reproduced locally* on JDK 25 (Amazon Corretto 25.0.3) -- test passes on JDK
17 and 21 but fails consistently on JDK 25.
*Error:*
{code}
Exception in thread "main" java.lang.ExceptionInInitializerError
at
org.apache.arrow.memory.netty.DefaultAllocationManagerFactory.<clinit>(DefaultAllocationManagerFactory.java:26)
...
Caused by: java.lang.UnsupportedOperationException
at
org.sparkproject.connect.client.io.netty.buffer.EmptyByteBuf.memoryAddress(EmptyByteBuf.java:961)
at
org.sparkproject.connect.client.io.netty.buffer.DuplicatedByteBuf.memoryAddress(DuplicatedByteBuf.java:115)
at
org.sparkproject.connect.client.io.netty.buffer.UnsafeDirectLittleEndian.<init>(UnsafeDirectLittleEndian.java:45)
at
org.sparkproject.connect.client.io.netty.buffer.PooledByteBufAllocatorL.<init>(PooledByteBufAllocatorL.java:47)
{code}
*Environment:*
- JDK 25 (Amazon Corretto 25.0.3, also OpenJDK 25.0.3 on UBI 10 per
dongjoon-hyun)
- JDK 17 and 21: not affected
*Root cause:* Netty's {{EmptyByteBuf.memoryAddress()}} throws
{{UnsupportedOperationException}} on JDK 25. Arrow's
{{DefaultAllocationManagerFactory}} calls this during static initialization via
{{PooledByteBufAllocatorL}}. This affects any code path that initializes Arrow
(Spark Connect client, vectorized readers, etc.).
*Workaround:* PR #55999 skips the affected test on JDK 25+.
*Root fix:* Requires Arrow/Netty dependency update to a version that supports
JDK 25.
Originally reported by dongjoon-hyun on PR #55720.
was:
On JDK 25 + UBI 10 (Red Hat Enterprise Linux 10.1), spark-shell --remote fails
to start due to Arrow/Netty memory allocator incompatibility:
ExceptionInInitializerError in DefaultAllocationManagerFactory
Caused by: UnsupportedOperationException at EmptyByteBuf.memoryAddress()
This affects any code path that initializes Arrow (Spark Connect client,
vectorized readers, etc.).
Reported by dongjoon-hyun on PR #55720.
Workaround: skip affected tests on JDK 25+ (PR #55999).
Root fix: needs Arrow/Netty update to support JDK 25.
> Arrow/Netty memory allocator fails on JDK 25 (affects spark-shell --remote)
> ---------------------------------------------------------------------------
>
> Key: SPARK-56955
> URL: https://issues.apache.org/jira/browse/SPARK-56955
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.2.0
> Reporter: Anupam Yadav
> Priority: Major
>
> On JDK 25, spark-shell --remote fails to start due to Arrow/Netty memory
> allocator incompatibility.
> *Reproduced locally* on JDK 25 (Amazon Corretto 25.0.3) -- test passes on JDK
> 17 and 21 but fails consistently on JDK 25.
> *Error:*
> {code}
> Exception in thread "main" java.lang.ExceptionInInitializerError
> at
> org.apache.arrow.memory.netty.DefaultAllocationManagerFactory.<clinit>(DefaultAllocationManagerFactory.java:26)
> ...
> Caused by: java.lang.UnsupportedOperationException
> at
> org.sparkproject.connect.client.io.netty.buffer.EmptyByteBuf.memoryAddress(EmptyByteBuf.java:961)
> at
> org.sparkproject.connect.client.io.netty.buffer.DuplicatedByteBuf.memoryAddress(DuplicatedByteBuf.java:115)
> at
> org.sparkproject.connect.client.io.netty.buffer.UnsafeDirectLittleEndian.<init>(UnsafeDirectLittleEndian.java:45)
> at
> org.sparkproject.connect.client.io.netty.buffer.PooledByteBufAllocatorL.<init>(PooledByteBufAllocatorL.java:47)
> {code}
> *Environment:*
> - JDK 25 (Amazon Corretto 25.0.3, also OpenJDK 25.0.3 on UBI 10 per
> dongjoon-hyun)
> - JDK 17 and 21: not affected
> *Root cause:* Netty's {{EmptyByteBuf.memoryAddress()}} throws
> {{UnsupportedOperationException}} on JDK 25. Arrow's
> {{DefaultAllocationManagerFactory}} calls this during static initialization
> via {{PooledByteBufAllocatorL}}. This affects any code path that initializes
> Arrow (Spark Connect client, vectorized readers, etc.).
> *Workaround:* PR #55999 skips the affected test on JDK 25+.
> *Root fix:* Requires Arrow/Netty dependency update to a version that supports
> JDK 25.
> Originally reported by dongjoon-hyun on PR #55720.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
