voonhous opened a new issue, #19887:
URL: https://github.com/apache/hudi/issues/19887
**What needs to be done:**
Two tasks, in this order:
1. **Build and push Spark 3.5.9 variants of the `apachehudi` demo/IT images
to Docker Hub.** Two image sets exist at 3.5.3 and both need a 3.5.9 twin:
`hudi-hadoop_2.8.4-hive_2.3.10-spark{base,adhoc,master,worker}_3.5.3` and
`hudi-hadoop_3.3.4-hive_3.1.3-spark{base,adhoc,master,worker}_3.5.3`.
`docker/build_docker_images.sh --spark-version 3.5.9 --multi-arch` builds and
pushes both arches in one pass (second run with `--hadoop-version 3.3.4
--hive-version 3.1.3`). Needs push rights on the `apachehudi` org.
2. **Repoint every 3.5.3 pin under `docker/`, `hudi-integ-test/pom.xml` and
`bot.yml` to 3.5.9** once the images are up. Full list in the dropdown. Order
matters: the `integration-tests` job runs `docker compose up` on
`docker-compose_hadoop284_hive2310_spark353_amd64.yml` on every run, so
repointing before the push fails the job on the image pull.
**Why this task is needed:**
#19884 moves `spark35.version` to 3.5.9 and the `integration-tests` Spark
tarball with it. The docker stack stays on 3.5.3, so the demo containers and
the docker-cluster ITs run a Hudi bundle compiled against 3.5.9 on a 3.5.3
runtime, the direction that can hit a missing API. It is latent today: all nine
ITs on that stack (`ITTestHoodieDemo`, `ITTestHoodieSanity`,
`ITTestHoodieSyncCommand`) are `@Disabled` and report `Tests run: 9, Skipped:
9`. But the stack is still pulled and started on every run, `setup_demo.sh`
ships it to users, and the parked `integration-tests-hive-sync` job pins the
same `sparkadhoc_3.5.3` image. #19884 left this out deliberately because none
of the 3.5.9 images exist yet.
<details>
<summary>Every 3.5.3 pin to move</summary>
- `docker/hoodie/hadoop/pom.xml:70` `docker.spark.version`
- `docker/build_docker_images.sh:24` `SPARK_VERSION` default
-
`docker/hoodie/hadoop/{spark_base,sparkadhoc,sparkmaster,sparkworker}/Dockerfile`
`ARG SPARK_VERSION=3.5.3`
-
`docker/compose/docker-compose_hadoop284_hive2310_spark353_{amd64,arm64}.yml`
and `docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml`: rename to
`spark359`, move the four `image:` tags and `CLUSTER_NAME` in each
- `docker/setup_demo.sh:22-24`, `docker/stop_demo.sh:23-25`
`COMPOSE_FILE_NAME`
- `docker/README.md:73-92`
- `hudi-integ-test/pom.xml:413` and `:573` (`m1-mac` profile)
`dockerCompose.file`
- `.github/workflows/bot.yml:1568-1569` (commented
`integration-tests-hive-sync` matrix: `composePrefix`, `sparkAdhocImage`)
</details>
<details>
<summary>Docker Hub state as of 2026-09-10</summary>
| repository | tags |
|---|---|
| `hudi-hadoop_2.8.4-hive_2.3.10-spark{base,adhoc,master,worker}_3.5.3` |
`latest`, `1.2.0-SNAPSHOT` (2026-04-17) |
| `hudi-hadoop_3.3.4-hive_3.1.3-sparkadhoc_3.5.3` | `latest`, `4e6ebba2...`
(2025-10-17) |
| `hudi-hadoop_2.8.4-hive_2.3.10-sparkadhoc_3.5.5` | no such repository |
| `hudi-hadoop_2.8.4-hive_2.3.10-sparkadhoc_3.5.9` | no such repository |
The compose files pull `:latest`, and the Spark version is part of the
repository name, so pushing 3.5.9 under new names does not disturb anything
currently pinned.
</details>
**Related issues:** #19884 (the bump), #19883 (tarball download; lists this
drift), #19524 (epic). Out of scope: the bundle-validation lanes at
spark3.5.0/spark3.5.1 use a different image (`hudi-ci-bundle-validation-base`)
and stay on #19883.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
