LuciferYang opened a new pull request, #365: URL: https://github.com/apache/doris-spark-connector/pull/365
## Summary Adds Apache Spark 4.1 support (JDK 17, Scala 2.13.17) **alongside** 4.0, with full unit + integration + E2E coverage. It reuses the shared `spark-4-base` module (the DataSource V2 API is stable 4.0→4.1); `spark-doris-connector-spark-4.1` is a thin adapter. > **Stacked on #363** (Spark 4.0), which is stacked on #364 (exactly-once retry fix). Please review/merge those first; this branch is rebased onto master as they land. ## New module - `spark-doris-connector-spark-4.1` — Spark 4.1.0 / Scala 2.13.17 adapter (mirrors `spark-4.0`). ## Dependency alignment for Spark 4.1 Per-profile (via properties); **2.x / 3.x / 4.0 are unchanged**. | Dependency | 4.0 | 4.1 | Why | | --- | --- | --- | --- | | Arrow | 18.1.0 | 18.3.0 | match Spark 4.1's Arrow | | Netty | 4.1.110 | 4.2.7 | **Spark 4.1 moved to Netty 4.2** | | gRPC | 1.65.0 | 1.69.0 | gRPC 1.69 supports Netty 4.2 | | adbc / protobuf | 0.18.0 / 3.25.1 | 0.18.0 / 3.25.1 | unchanged | The Netty 4.1→4.2 jump is the only 4.1-specific surprise: on the unshaded integration-test classpath the connector + gRPC must agree with Spark on a single Netty, so the `spark-4.1` profile pins Netty 4.2.7 and bumps gRPC to 1.69. The profile also adds Spark 4's JDK-17 `--add-opens` set. ## Other changes - `DorisConfig` no longer uses `commons-collections` (`MapUtils`) — Spark 4.1 stopped exporting it transitively; replaced with a plain null/empty check. - `build.sh`, `build-extension.yml`, `run-itcase.yml`, `run-e2ecase.yml`, `deploy_staging_jars.sh`: Spark 4.1 wiring mirroring 4.0. ## Test plan - [x] Build + unit tests for the Spark 4.1 module (JDK 17) - [x] Full Spark 4.1 integration suite (56 ITCase) + E2E (2 E2ECase, both thrift and arrow-flight) — green locally - [x] No regression to 2.4 / 3.x / 4.0 (per-profile overrides) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
