This is an automated email from the ASF dual-hosted git repository.
CalvinKirs pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new be7c013c119 [fix](fe) Upgrade gRPC and remove unused LZ4 dependencies
(#67585)
be7c013c119 is described below
commit be7c013c119e4311f84794067d6a8c40e80cad17
Author: Calvin Kirs <[email protected]>
AuthorDate: Tue Sep 8 09:49:38 2026 +0800
[fix](fe) Upgrade gRPC and remove unused LZ4 dependencies (#67585)
### What problem does this PR solve?
Related PR: #67065
Problem Summary:
Upgrade the managed gRPC BOM from 1.65.1 to 1.75.0, matching #67065 and
replacing the `grpc-netty-shaded` version affected by CVE-2025-55163.
Exclude `org.lz4:lz4-pure-java` centrally from the managed
`odps-sdk-core` and `odps-sdk-table-api` dependencies. Neither
MaxCompute connector needs a replacement Java LZ4 dependency: the FE
uses the ODPS metadata/session APIs, and the BE uses Table API Arrow
readers/writers with ZSTD. The legacy SDK Tunnel LZ4 implementations are
outside the active Doris data path. The BE connector retains its local
Jackson exclusions.
Limit `flight-sql-jdbc-driver` to test scope. Only the FE test client
loads the JDBC driver; production code uses the retained `FlightServer`
and `FlightSqlClient` protocol libraries. The regression framework keeps
its independent JDBC driver dependency.
Add a ZSTD round-trip test through the ODPS Arrow writer/reader
factories, with the Java 17 direct-buffer access options required by the
test JVM. This verifies 1,024 rows including a null after removing LZ4,
alongside the existing isolated ODPS class-loading test.
### Release note
Upgrade gRPC to 1.75.0. Stop bundling the unused Java LZ4 dependency in
MaxCompute connectors and the test-only Flight SQL JDBC driver in the FE
runtime distribution.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (details below)
Validation of the final implementation in `d9613e32c86`:
- `EXTRA_FE_MODULES=maxcompute=be-java-extensions/max-compute-connector
./run-fe-ut.sh --run
'org.apache.doris.connector.maxcompute.*Test,org.apache.doris.maxcompute.*Test,!org.apache.doris.connector.maxcompute.OdpsLiveConnectivityTest'`:
146 FE and 14 BE connector tests passed, including ODPS class-loader
isolation and the new ZSTD round trip. The successful test runtime
classpaths were checked and contain no `net.jpountz.lz4` classes.
- `build.sh --fe` passed with `DISABLE_BE_JAVA_EXTENSIONS=OFF`,
`FE_MAVEN_THREADS=4`, `MVN_OPT=-Dmaven.build.cache.enabled=false`, and
`--be-extension-ignore
iceberg-metadata-scanner,hadoop-hudi-scanner,java-common,java-udf,jdbc-scanner,paimon-scanner,trino-connector-scanner,preload-extensions,hadoop-deps,java-writer`.
All 64 reactor modules passed, including Checkstyle, FE compilation and
FE/BE MaxCompute packaging. Maven wall time: 2m 31s.
- Both connector dependency trees contain no Java LZ4 artifacts. The
rebuilt FE plugin zip and BE connector jar contain no Java LZ4
jars/classes. The incremental output directory retained the previous
build's LZ4 jar; that stale artifact was removed and the runtime plugin
directory was rechecked.
- `output/fe/lib` contains `grpc-netty-shaded-1.75.0.jar`; no Flight SQL
JDBC driver jar is present under `output/fe`.
- `git diff --check` passed. Live MaxCompute connectivity, cluster
startup and regression tests were not run.
---
.../max-compute-connector/pom.xml | 3 ++
.../doris/maxcompute/MaxComputeJniWriterTest.java | 52 ++++++++++++++++++++++
fe/fe-core/pom.xml | 1 +
fe/pom.xml | 12 ++++-
4 files changed, 67 insertions(+), 1 deletion(-)
diff --git a/fe/be-java-extensions/max-compute-connector/pom.xml
b/fe/be-java-extensions/max-compute-connector/pom.xml
index 74c528fa2f1..37d6981ff79 100644
--- a/fe/be-java-extensions/max-compute-connector/pom.xml
+++ b/fe/be-java-extensions/max-compute-connector/pom.xml
@@ -104,6 +104,9 @@ under the License.
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
+ <argLine>
+ --add-opens=java.base/java.nio=ALL-UNNAMED
-Dio.netty.tryReflectionSetAccessible=true @{argLine}
+ </argLine>
<forkCount>${fe_ut_parallel}</forkCount>
<reuseForks>false</reuseForks>
<useFile>false</useFile>
diff --git
a/fe/be-java-extensions/max-compute-connector/src/test/java/org/apache/doris/maxcompute/MaxComputeJniWriterTest.java
b/fe/be-java-extensions/max-compute-connector/src/test/java/org/apache/doris/maxcompute/MaxComputeJniWriterTest.java
index 84cc4a79d6e..a795637240a 100644
---
a/fe/be-java-extensions/max-compute-connector/src/test/java/org/apache/doris/maxcompute/MaxComputeJniWriterTest.java
+++
b/fe/be-java-extensions/max-compute-connector/src/test/java/org/apache/doris/maxcompute/MaxComputeJniWriterTest.java
@@ -17,6 +17,16 @@
package org.apache.doris.maxcompute;
+import com.aliyun.odps.account.AliyunAccount;
+import com.aliyun.odps.table.arrow.ArrowReader;
+import com.aliyun.odps.table.arrow.ArrowReaderFactory;
+import com.aliyun.odps.table.arrow.ArrowWriter;
+import com.aliyun.odps.table.arrow.ArrowWriterFactory;
+import com.aliyun.odps.table.configuration.CompressionCodec;
+import com.aliyun.odps.table.configuration.ReaderOptions;
+import com.aliyun.odps.table.configuration.WriterOptions;
+import com.aliyun.odps.table.enviroment.Credentials;
+import com.aliyun.odps.table.enviroment.EnvironmentSettings;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.IntVector;
@@ -24,9 +34,51 @@ import org.apache.arrow.vector.VectorSchemaRoot;
import org.junit.Assert;
import org.junit.Test;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
import java.util.Collections;
public class MaxComputeJniWriterTest {
+ @Test
+ public void testZstdArrowRoundTrip() throws Exception {
+ EnvironmentSettings settings = EnvironmentSettings.newBuilder()
+ .withCredentials(Credentials.newBuilder()
+ .withAccount(new AliyunAccount("test-access-key",
"test-secret-key")).build())
+ .build();
+ WriterOptions writerOptions =
WriterOptions.newBuilder().withSettings(settings)
+ .withCompressionCodec(CompressionCodec.ZSTD).build();
+ try (BufferAllocator allocator = new RootAllocator();
+ VectorSchemaRoot root = VectorSchemaRoot.of(new IntVector("c",
allocator))) {
+ IntVector vector = (IntVector) root.getVector(0);
+ vector.allocateNew(1024);
+ for (int i = 0; i < 1024; i++) {
+ vector.set(i, i % 7);
+ }
+ vector.setNull(5);
+ root.setRowCount(1024);
+
+ ByteArrayOutputStream bytes = new ByteArrayOutputStream();
+ try (ArrowWriter writer =
ArrowWriterFactory.getRecordBatchWriter(bytes, writerOptions)) {
+ writer.writeBatch(root);
+ }
+
+ ReaderOptions readerOptions =
ReaderOptions.newBuilder().withSettings(settings)
+
.withBufferAllocator(allocator).withCompressionCodec(CompressionCodec.ZSTD)
+ .withReuseBatch(true).build();
+ try (ArrowReader reader = ArrowReaderFactory.getRecordBatchReader(
+ new ByteArrayInputStream(bytes.toByteArray()),
readerOptions)) {
+ Assert.assertTrue(reader.nextBatch());
+ VectorSchemaRoot decoded = reader.getCurrentValue();
+ Assert.assertEquals(root.getRowCount(), decoded.getRowCount());
+ IntVector decodedVector = (IntVector) decoded.getVector(0);
+ for (int i = 0; i < root.getRowCount(); i++) {
+ Assert.assertEquals(vector.getObject(i),
decodedVector.getObject(i));
+ }
+ Assert.assertFalse(reader.nextBatch());
+ }
+ }
+ }
+
@Test
public void testPrefixBufferBytesMeasuresLeadingRowsWithoutRebuild() {
try (BufferAllocator allocator = new RootAllocator();
diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml
index 30f143a2e88..8466cbfd061 100644
--- a/fe/fe-core/pom.xml
+++ b/fe/fe-core/pom.xml
@@ -805,6 +805,7 @@ under the License.
<dependency>
<groupId>org.apache.arrow</groupId>
<artifactId>flight-sql-jdbc-driver</artifactId>
+ <scope>test</scope>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
diff --git a/fe/pom.xml b/fe/pom.xml
index e0aee6b61a7..7c636abfacb 100644
--- a/fe/pom.xml
+++ b/fe/pom.xml
@@ -325,7 +325,7 @@ under the License.
<!-- NOTE: Using grpc-java whose version is newer than 1.34.0 will
break the build on CentOS 6 due to the obsolete GLIBC -->
<grpc-java.version>1.34.0</grpc-java.version>
<!--Need to ensure that the version is the same as in
arrow/java/pom.xml or compatible with it.-->
- <grpc.version>1.65.1</grpc.version>
+ <grpc.version>1.75.0</grpc.version>
<check.freamework.version>3.53.0</check.freamework.version>
<cel.version>0.12.0</cel.version>
<!-- FE-only Arrow 19 startup POC: align protobuf runtime with Arrow
Java 19's protobuf BOM. -->
@@ -1613,6 +1613,10 @@ under the License.
<artifactId>odps-sdk-core</artifactId>
<version>${maxcompute.version}</version>
<exclusions>
+ <exclusion>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-pure-java</artifactId>
+ </exclusion>
<exclusion>
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-vector</artifactId>
@@ -1635,6 +1639,12 @@ under the License.
<groupId>com.aliyun.odps</groupId>
<artifactId>odps-sdk-table-api</artifactId>
<version>${maxcompute.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-pure-java</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
<dependency>
<!-- Keep tea pinned for Alibaba SDK dependency chains that
allow com.aliyun:tea:[1.1.14,2.0.0). -->
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
