[
https://issues.apache.org/jira/browse/CALCITE-7539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18082512#comment-18082512
]
Caican Cai commented on CALCITE-7539:
-------------------------------------
I investigated upgrading the Arrow adapter dependencies beyond 15.0.0, but the
upgrade is currently blocked by Calcite's CI/runtime compatibility requirements.
The investigation went through several candidate versions:
1. Arrow/Gandiva 18.0.0
This failed immediately on the JDK 8 CI job because Arrow 18 artifacts are
compiled for Java 11:
- class file version 55.0 = Java 11
- JDK 8 expects class file version 52.0
So Arrow 18 cannot be used while Calcite still runs JDK 8 CI.
2. Arrow/Gandiva 17.0.0 and 16.1.0
These versions still have Java 8 class file version 52, but they are not
actually safe on a JDK 8 runtime.
The CI failed with:
`NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;`
The reason is that those Arrow versions call the Java 9+ covariant method
signature:
`ByteBuffer.flip(): ByteBuffer`
But JDK 8 only has:
`ByteBuffer.flip(): Buffer`
Therefore, even though the bytecode version is Java 8-compatible, the runtime
API usage is not compatible with JDK 8.
3. Arrow/Gandiva 16.0.0
Arrow 16.0.0 avoids the `ByteBuffer.flip()` runtime issue, but it still fails
in Calcite CI because Gandiva native initialization fails:
`Failed to make LLVM module due to Could not create LLJIT instance: Symbols not
found: [ llvm_orc_registerEHFrameSectionWrapper ]`
This failure happens in existing Arrow adapter tests such as simple projection
tests, at:
`org.apache.arrow.gandiva.evaluator.Projector.make`
So this is not caused by a new test or a new Arrow type. It indicates a native
Gandiva / LLVM symbol compatibility issue in the CI environment.
Conclusion
The upgrade is not just a dependency version bump. There are multiple
compatibility blockers:
- 18.0.0 requires Java 11 bytecode;
- 17.0.0 and 16.1.0 are not safe on JDK 8 runtime because of Java 9+
`ByteBuffer.flip()` API usage;
- 16.0.0 still fails due to Gandiva native / LLVM symbol compatibility in CI.
Given that Calcite still runs JDK 8 jobs and the Arrow adapter relies on
Gandiva native libraries, I do not think this dependency upgrade is safe to
proceed with right now.
The practical decision is to keep Arrow/Gandiva at 15.0.0 for now, and handle
feature work such as CALCITE-6512 / CALCITE-7541 within the current dependency
baseline.
> Upgrade Arrow adapter dependencies to 16.0.0
> --------------------------------------------
>
> Key: CALCITE-7539
> URL: https://issues.apache.org/jira/browse/CALCITE-7539
> Project: Calcite
> Issue Type: Improvement
> Components: arrow-adapter
> Affects Versions: 1.41.0
> Reporter: Caican Cai
> Priority: Minor
> Labels: pull-request-available
> Fix For: 1.42.0
>
>
> Upgrade Arrow adapter dependencies to 16.0.0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
