yihua opened a new issue, #19364: URL: https://github.com/apache/hudi/issues/19364
## Summary Apache Spark **4.2.0** has been officially released (GA). Hudi's `spark4.2` build profile currently targets `4.2.0-preview4` (added in #18621). This issue tracks upgrading the Spark 4.2 dependency to the GA release, including the CI bundle-validation Docker image rebuild. Part of the umbrella epic #16053 ([EPIC] Upgrading Spark in Hudi, HUDI-6427). ## Scope **Dependency upgrade** (`pom.xml`, `spark4.2` profile): - `spark42.version`: `4.2.0-preview4` → `4.2.0` - `hadoop.version`: `3.4.3` → `3.5.0` (matches what Spark 4.2.0 GA ships) - `lz4.version` (`at.yawk.lz4:lz4-java`): `1.10.4` → `1.11.0` (matches Spark 4.2.0 GA) All other transitive versions (Scala 2.13.18, Parquet 1.17.0, ORC 2.3.0, Avro 1.12.1, Jackson 2.21.2, Kafka 3.9.2, Log4j 2.25.4, slf4j 2.0.17, antlr 4.13.1) are unchanged between preview4 and GA — verified by diffing the Spark parent POMs. **CI bundle-validation Docker image rebuild**: - Rename base image / build script `flink1200hive313spark420previewscala213` → `flink1200hive313spark420scala213`, build with `SPARK_VERSION=4.2.0` / `HADOOP_VERSION=3.5.0`, and publish the new `apachehudi/hudi-ci-bundle-validation-base` image (see #18548 for the publish process). - Update `ci_run.sh`, `run_docker_java17.sh`, `validate.sh`, `.github/workflows/bot.yml`, and `.asf.yaml` required checks: `spark4.2.0-preview4` → `spark4.2.0`. ## Verification - Spark 4.2.0, `hadoop-client-api` 3.5.0, and `lz4-java` 1.11.0 resolve transitively from Maven Central; `spark-4.2.0-bin-hadoop3.tgz` and `hadoop-3.5.0.tar.gz` are available on archive.apache.org. - Full compile + bundle validation + docker-java17 test run via the CI Spark 4.2 matrix jobs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
