This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 30a4ba33e385 docs(docker): document build_docker_images.sh flags
(#18687)
30a4ba33e385 is described below
commit 30a4ba33e38595eb6c1ef8327cba7ae0296aa92a
Author: 201573 <[email protected]>
AuthorDate: Thu Jul 23 16:17:11 2026 +0800
docs(docker): document build_docker_images.sh flags (#18687)
* docs(docker): document build_docker_images.sh flags
* docs(docker): clarify multi-arch prerequisites
* docs(docker): align demo example with setup_demo.sh
* docs(docker): mention setup_demo.sh compose override
* docs(docker): address review findings on demo build notes
- note the default versions built when no flags are passed
- clarify build_docker_images.sh builds with docker directly, unlike
build_local_docker_images.sh which goes through Maven
- warn that a plain setup_demo.sh run pulls Docker Hub images over
locally built ones; use ./setup_demo.sh dev to keep local images
- use shell language tag on the buildx code fence for consistency
---------
Co-authored-by: voon <[email protected]>
---
docker/README.md | 107 +++++++++++++++++++++++--------------------------------
1 file changed, 44 insertions(+), 63 deletions(-)
diff --git a/docker/README.md b/docker/README.md
index f655f42dca8b..f445a87fb4d4 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -59,6 +59,37 @@ To build all docker images locally, you can run the script:
./build_local_docker_images.sh
```
+To build the Docker demo images with `docker` directly, rather than through
the Maven build like
+`build_local_docker_images.sh` above, run the script from under
`<HUDI_REPO_DIR>/docker`:
+
+```shell
+# With no flags, builds Hadoop 2.8.4 / Spark 3.5.3 / Hive 2.3.10, matching
+# docker-compose_hadoop284_hive2310_spark353_{amd64,arm64}.yml
+./build_docker_images.sh
+```
+
+You can override the Hadoop, Spark, and Hive versions from the command line.
If you plan to use `setup_demo.sh`,
+build the image set matching the default compose files first. For other flows,
use one of the supported version
+combinations under `docker/compose`.
+
+```shell
+# Matches setup_demo.sh and
+# docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml
+./build_docker_images.sh --hadoop-version 3.3.4 --spark-version 3.5.3
--hive-version 3.1.3
+
+# Another supported combination is
+# docker-compose_hadoop340_hive313_spark401_{amd64,arm64}.yml
+./build_docker_images.sh --hadoop-version 3.4.0 --spark-version 4.0.1
--hive-version 3.1.3
+```
+
+`setup_demo.sh` currently defaults to
`docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml`. If you build a
+different image set for the demo flow, update `COMPOSE_FILE_NAME` in
`setup_demo.sh` to point to the matching compose
+file before running the script. Run `./setup_demo.sh dev` to use your locally
built images; a plain run pulls the
+Docker Hub images over them.
+
+By default, the script builds images for the current machine architecture and
derives the version tag from the root
+`pom.xml`. Use `--version-tag` to set an explicit tag if needed.
+
To build a single image target, you can run
```shell
@@ -120,15 +151,10 @@ Please refer to the [Docker Demo Docs
page](https://hudi.apache.org/docs/docker_
## Building Multi-Arch Images
-NOTE: The steps below require some code changes. Support for multi-arch builds
in a fully automated manner is being
-tracked by [HUDI-3601](https://issues.apache.org/jira/browse/HUDI-3601).
+The `build_docker_images.sh` script supports multi-arch image builds through
Docker `buildx`. First ensure a `buildx`
+builder is set up locally:
-By default, the docker images are built for x86_64 (amd64) architecture.
Docker `buildx` allows you to build multi-arch
-images, link them together with a manifest file, and push them all to a
registry – with a single command. Let's say we
-want to build for arm64 architecture. First we need to ensure that `buildx`
setup is done locally. Please follow the
-below steps (referred from https://www.docker.com/blog/multi-arch-images):
-
-```
+```shell
# List builders
~ ❯❯❯ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS
@@ -155,63 +181,18 @@ Status: running
Platforms: linux/amd64, linux/arm64, linux/arm/v7, linux/arm/v6
```
-Now goto `<HUDI_REPO_DIR>/docker/hoodie/hadoop` and change the `Dockerfile` to
pull dependent images corresponding to
-arm64. For example, in [base/Dockerfile](./hoodie/hadoop/base/Dockerfile)
(which pulls jdk11 image), change the
-line `FROM openjdk:11-jdk-slim-bullseye` to `FROM
arm64v8/openjdk:11-jdk-slim-bullseye`.
+Then run the script from under `<HUDI_REPO_DIR>/docker`:
-Then, from under `<HUDI_REPO_DIR>/docker/hoodie/hadoop` directory, execute the
following command to build as well as
-push the image to the dockerhub repo:
-
-```
-# Run under hoodie/hadoop, the <tag> is optional, "latest" by default
-docker buildx build <image_folder_name> --platform <comma-separated,platforms>
-t <hub-user>/<repo-name>[:<tag>] --push
-
-# For example, to build the Java 11 base image
-docker buildx build base_java11 --platform linux/arm64 -t
apachehudi/hudi-hadoop_2.8.4-base-java11:linux-arm64-0.10.1 --push
-```
-
-Note: the base image is now tagged per Java variant (`-base-java11` /
`-base-java17`). Downstream Dockerfiles
-select the variant via the `BASE_IMAGE_TAG` build arg (default `java11`). If
you also need the Java 17 base for
-arm64, repeat the build against `base_java17` and tag it as
`...-base-java17:<tag>`.
-
-Once the base image is pushed then you could do something similar for other
images.
-Change [hive](./hoodie/hadoop/hive_base/Dockerfile) dockerfile to pull the
base image with tag corresponding to
-linux/arm64 platform.
-
-```
-# Change below line in the Dockerfile
-FROM apachehudi/hudi-hadoop_${HADOOP_VERSION}-base-${BASE_IMAGE_TAG}:latest
-# as shown below (pin to the same Java variant you built above, e.g. java11)
-FROM --platform=linux/arm64
apachehudi/hudi-hadoop_${HADOOP_VERSION}-base-java11:linux-arm64-0.10.1
+```shell
+./build_docker_images.sh --multi-arch
-# and then build & push from under hoodie/hadoop dir
-docker buildx build hive_base --platform linux/arm64 -t
apachehudi/hudi-hadoop_2.8.4-hive_2.3.3:linux-arm64-0.10.1 --push
+# Example with explicit component versions
+./build_docker_images.sh --hadoop-version 3.4.0 --spark-version 4.0.1
--hive-version 3.1.3 --multi-arch
```
-Similarly, for images that are dependent on hive (e.g. [base
spark](./hoodie/hadoop/spark_base/Dockerfile)
-, [sparkmaster](./hoodie/hadoop/sparkmaster/Dockerfile),
[sparkworker](./hoodie/hadoop/sparkworker/Dockerfile)
-and [sparkadhoc](./hoodie/hadoop/sparkadhoc/Dockerfile)), change the
corresponding Dockerfile to pull the base hive
-image with tag corresponding to arm64. Then build and push using `docker
buildx` command.
-
-For the sake of completeness, here is a
[patch](https://gist.github.com/xushiyan/cec16585e884cf0693250631a1d10ec2) which
-shows what changes to make in Dockerfiles (assuming tag is named
`linux-arm64-0.10.1`), and below is the list
-of `docker buildx` commands.
-
-```
-docker buildx build base_java11 --platform linux/arm64 -t
apachehudi/hudi-hadoop_2.8.4-base-java11:linux-arm64-0.10.1 --push
-docker buildx build datanode --platform linux/arm64 --build-arg
BASE_IMAGE_TAG=java11 -t
apachehudi/hudi-hadoop_2.8.4-datanode:linux-arm64-0.10.1 --push
-docker buildx build historyserver --platform linux/arm64 -t
apachehudi/hudi-hadoop_2.8.4-history:linux-arm64-0.10.1 --push
-docker buildx build hive_base --platform linux/arm64 -t
apachehudi/hudi-hadoop_2.8.4-hive_2.3.3:linux-arm64-0.10.1 --push
-docker buildx build namenode --platform linux/arm64 -t
apachehudi/hudi-hadoop_2.8.4-namenode:linux-arm64-0.10.1 --push
-docker buildx build prestobase --platform linux/arm64 -t
apachehudi/hudi-hadoop_2.8.4-prestobase_0.217:linux-arm64-0.10.1 --push
-docker buildx build spark_base --platform linux/arm64 -t
apachehudi/hudi-hadoop_2.8.4-hive_2.3.3-sparkbase_2.4.4:linux-arm64-0.10.1
--push
-docker buildx build sparkadhoc --platform linux/arm64 -t
apachehudi/hudi-hadoop_2.8.4-hive_2.3.3-sparkadhoc_2.4.4:linux-arm64-0.10.1
--push
-docker buildx build sparkmaster --platform linux/arm64 -t
apachehudi/hudi-hadoop_2.8.4-hive_2.3.3-sparkmaster_2.4.4:linux-arm64-0.10.1
--push
-docker buildx build sparkworker --platform linux/arm64 -t
apachehudi/hudi-hadoop_2.8.4-hive_2.3.3-sparkworker_2.4.4:linux-arm64-0.10.1
--push
-```
+When `--multi-arch` is enabled, the script builds and pushes the amd64 and
arm64 variants in one pass. Use
+`--version-tag <tag>` to override the image tag used for the push.
-Once all the required images are pushed to the dockerhub repos, then we need
to do one additional change
-in [docker compose](./compose/docker-compose_hadoop284_hive233_spark244.yml)
file.
-Apply [this
patch](https://gist.github.com/codope/3dd986de5e54f0650dd74b6032e4456c) to the
docker compose file so
-that [setup_demo](./setup_demo.sh) pulls images with the correct tag for
arm64. And now we should be ready to run the
-setup script and follow the docker demo.
+Note that `--multi-arch` uses `docker buildx build --push` and the image names
in the script are hardcoded to the
+`apachehudi/...` Docker Hub repositories, so this flow requires push access to
those repositories. No Dockerfile
+changes are needed for the current amd64 plus arm64 image set in this
repository.
