deepakpanda93 opened a new pull request, #19489:
URL: https://github.com/apache/hudi/pull/19489
### Describe the issue this Pull Request addresses
Supersedes #14101, which was opened against a
`docker/build_docker_images.sh` that has since been
rewritten on master. Rebasing it would have reverted that work, so this is a
fresh port of the same
idea onto the current script. All review comments from #14101 are addressed
here.
Publishing the Hudi demo images to the `apachehudi` Docker Hub organisation
is a manual
`docker push` per image today, as documented under *Upload Updated Image to
Repository on Docker
Hub*. There is no scripted path, so a release publishes nine images by hand
across two tags each.
### Summary and Changelog
`docker/build_and_publish_docker_images.sh` replaces
`docker/build_docker_images.sh` and does both
jobs, which is what @xushiyan asked for on #14101 rather than a second
script duplicating the image
list and version variables.
- `--publish <true|false>`, default `false`, so a plain run only ever writes
to the local Docker
daemon. Named rather than positional, so the command states what it is
about to do.
- `--hadoop-version`, `--spark-version`, `--hive-version`, `--version-tag`,
all optional.
- `--multi-arch <true|false>`, default `false`.
- The base image follows the Spark version, so a 4.x release selects
`base_java17`.
- Every build passes `HADOOP_VERSION`, `SPARK_VERSION`, `HIVE_VERSION` and
`BASE_IMAGE_TAG`. The
Dockerfiles carry differing `ARG` defaults, so without this an image can
get a name that does not
describe its contents.
- Every image is tagged twice: `:latest` and the version from the root
`pom.xml`, overridable with
`--version-tag`.
- The push step reports a per-tag summary and exits non-zero if any push
failed, so a partial
publish is not read as success.
- `--help` prints the usage, including who publishing is for.
Two things worth reviewer attention:
**Default versions changed.** They were Hadoop 2.8.4 / Hive 2.3.10, but
`setup_demo.sh` runs
`docker-compose_hadoop334_hive313_spark353`, so a bare build produced a set
the demo never
referenced. Defaults are now Hadoop 3.3.4 / Hive 3.1.3 / Spark 3.5.3 to
match. The old combination
is still reachable with the version flags.
**Multi-arch without publishing.** `buildx` cannot leave a multi-platform
result nowhere: it either
pushes or loads. With `--publish true` it pushes, which is also what makes
each image visible to the
next one, since every image after the base starts `FROM` the previous one.
Without it the images are
loaded into the local **containerd** image store, using the builder from the
current docker context.
Both are required: the legacy image store cannot hold a multi-platform
image, and a
`docker-container` builder resolves `FROM` against registries only, so it
silently builds the later
images against whatever is already on Docker Hub rather than against the
run. The script checks for
the containerd store and stops with an explanation instead.
**One unrelated bug fixed because it blocked the above.** `base_java11`
copied
`/etc/hadoop/mapred-site.xml.template` unconditionally, a file Hadoop 3.x
does not ship, so Hadoop
3.x could not be built on the Java 11 base at all. The copy is now guarded.
The guard is POSIX
`[ -f ... ]` rather than the `[[ -f ... ]]` used by the legacy `base`
module, because the Dockerfile
`RUN` executes under `/bin/sh`, which is `dash` in these images; `[[` is not
a command there, so the
condition is always false and the copy would be skipped even when the file
exists. Happy to split
this into its own PR if reviewers prefer.
`docker/README.md` is updated throughout: a publishing section stating this
is a maintainer action,
the version and multi-arch flags, and what keeping multi-arch images locally
requires.
**Review round.** Three places where the README and the script had drifted
apart:
- The version override section told readers to edit the variable block at
the top of the script,
while the script takes `--hadoop-version`, `--spark-version` and
`--hive-version`. It documents
the flags now; no hand editing instruction remains.
- The version tag was described only as an exported `VERSION_TAG`, while
`--version-tag` also
exists. Both work, so the README states both and the order they resolve
in: flag, then
environment variable, then the version from the root `pom.xml`.
- The base image section had been narrowed to `base_java11` while the script
still selects
`base_java17` for a 4.x `--spark-version`. The section describes both
again, and notes that the
demo compose files are all Spark 3.x so the Java 17 base is only reached
on request.
### Impact
Publishing the demo image set becomes one command for maintainers. A plain
build now produces the
images `setup_demo.sh` actually consumes. Hadoop 3.x is buildable on the
Java 11 base.
`build_docker_images.sh` is removed, so anyone invoking it by name will need
to switch. Nothing in
the repository referenced it other than the README.
### Risk Level
low
### Documentation Update
`docker/README.md` is updated in this PR. No Hudi website change needed.
### Verification
**Full image set built.** A default run built all nine images on Docker
28.1.1 / macOS arm64, each
carrying both tags, `:latest` and the `pom.xml` version:
```
apachehudi/hudi-hadoop_3.3.4-base-java11 :latest
:1.3.0-SNAPSHOT
apachehudi/hudi-hadoop_3.3.4-datanode :latest
:1.3.0-SNAPSHOT
apachehudi/hudi-hadoop_3.3.4-namenode :latest
:1.3.0-SNAPSHOT
apachehudi/hudi-hadoop_3.3.4-history :latest
:1.3.0-SNAPSHOT
apachehudi/hudi-hadoop_3.3.4-hive_3.1.3 :latest
:1.3.0-SNAPSHOT
apachehudi/hudi-hadoop_3.3.4-hive_3.1.3-sparkbase_3.5.3 :latest
:1.3.0-SNAPSHOT
apachehudi/hudi-hadoop_3.3.4-hive_3.1.3-sparkadhoc_3.5.3 :latest
:1.3.0-SNAPSHOT
apachehudi/hudi-hadoop_3.3.4-hive_3.1.3-sparkmaster_3.5.3 :latest
:1.3.0-SNAPSHOT
apachehudi/hudi-hadoop_3.3.4-hive_3.1.3-sparkworker_3.5.3 :latest
:1.3.0-SNAPSHOT
```
These names are exactly what
`docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml`
references, which the previous defaults did not produce.
**Argument handling** was checked by stubbing `docker` on `PATH` and
asserting the resulting
command lines:
| run | `docker build` | `buildx` | `buildx --push` | `buildx --load` |
`docker push` |
| --- | --- | --- | --- | --- | --- |
| *(default)* | 9 | 0 | 0 | 0 | 0 |
| `--publish true` | 9 | 0 | 0 | 0 | 18 |
| `--multi-arch true` | 0 | 9 | 0 | 9 | 0 |
| `--multi-arch true --publish true` | 0 | 9 | 9 | 0 | 0 |
**Review round checks.** The three README corrections were verified against
the script rather than
by reading it:
```
$ ./build_and_publish_docker_images.sh --version-tag t1 --spark-version
4.0.1 --publish false
Using Java 17 base image for Spark 4.0.1
Building hoodie/hadoop/base_java17 as:
- apachehudi/hudi-hadoop_3.3.4-base-java17:latest
```
| version tag invocation | resulting tag |
| --- | --- |
| `--version-tag from-flag` | `from-flag` |
| `VERSION_TAG=from-env` | `from-env` |
| both together | `from-flag` |
`bash -n` and `sh -n` both pass on the script, and every flag it parses is
documented in the README.
**Multi-arch** was exercised for `base_java11` and `datanode` on
`linux/amd64,linux/arm64`,
confirming the Hadoop 3.3.4 guard and that `datanode` resolves its parent
from the local containerd
store rather than pulling it. The remaining seven were not built multi-arch
end to end, as that run
is dominated by ~660 MB Apache mirror downloads per architecture.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
