[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. IMPALA-12436: Create multi-platform builder images Updates the toolchain build image creation process to support publishing multi-platform (aarch64 and amd64) images. Uses docker buildx to produce multi-platform images. This requires QEMU on the builder; these builds are quick, so the overhead of emulating arm is minimal. Currently assumes images are built on amd64 as not all builds specify `--platform`. Moves the publishing step to happen within buildx, because buildx does not support loading multi-platform images in the local Docker registry. This does mean that a build failure could result in publishing only some of the images. These are not updated frequently and are usually preceded by a test run, so this risk seems minimal. The jenkins/build.sh script avoids this issue by first running a build without publishing, then when it succeeds running again with publishing (which will largely pull image layers from the build cache so should be very fast). Python 2.7 ran into build issues installing awscli on ARM, so OS targeting ARM were updated to ensure they include Python 3.8+ and install a newer version of awscli in Python 3. New options were added to buildall.py: - registry: publish images to the specified registry - multi: enables multi-platform builds on RedHat 8 and Ubuntu 20/22 Multi-platform builds require QEMU and a non-default buildx builder. Jenkins job is updated to install QEMU, create a builder, and build multi-platform images. Multi-platform images can't be loaded into a local Docker context (where they would be listed via docker images); we continue to load them there by default with --load, but disable it when doing multi-platform builds. The only way to get images out of the builder for multi-platform builds is to specify a registry. Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Reviewed-on: http://gerrit.cloudera.org:8080/20469 Reviewed-by: Joe McDonnell Tested-by: Michael Smith --- M docker/README.md M docker/all/postinstall.sh M docker/buildall.py M docker/redhat8.df M docker/ubuntu2004.df M docker/ubuntu2204.df M jenkins/build.sh 7 files changed, 87 insertions(+), 32 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Michael Smith: Verified -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 8 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 7: Verified+1 Verified these containers can be used to build current and older versions of native-toolchain. -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 20 Sep 2023 20:50:22 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/20469/6/docker/buildall.py File docker/buildall.py: http://gerrit.cloudera.org:8080/#/c/20469/6/docker/buildall.py@29 PS6, Line 29: ARM_PLATFORMS = ['redhat8', 'redhat9', 'ubuntu2004', 'ubuntu2204'] > Maybe this should be an environment variable? Although we'll need to make c Not going to change it, if we're producing multi-platform containers we should always do it for all platforms we support. -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 20 Sep 2023 17:52:20 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 7: Code-Review+2 This looks good to me. Sometimes we use the docker images for older builds of native toolchain for older Impala branches. Probably we want to verify that these images continue to work for older toolchains (this is mainly about Redhat 8 and Ubuntu 20). I don't think the version of Python 3 or any of the other changes will matter for those purposes. -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 19 Sep 2023 19:36:48 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/20469/5/jenkins/build.sh File jenkins/build.sh: http://gerrit.cloudera.org:8080/#/c/20469/5/jenkins/build.sh@59 PS5, Line 59: # Non-default builder only used for multi-platform bui > I didn't think so, but I'm open to adding it. Adding the param helped decouple this change from updating the Jenkins environment to a newer Docker version. -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 12 Sep 2023 22:28:37 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Hello Laszlo Gaal, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20469 to look at the new patch set (#7). Change subject: IMPALA-12436: Create multi-platform builder images .. IMPALA-12436: Create multi-platform builder images Updates the toolchain build image creation process to support publishing multi-platform (aarch64 and amd64) images. Uses docker buildx to produce multi-platform images. This requires QEMU on the builder; these builds are quick, so the overhead of emulating arm is minimal. Currently assumes images are built on amd64 as not all builds specify `--platform`. Moves the publishing step to happen within buildx, because buildx does not support loading multi-platform images in the local Docker registry. This does mean that a build failure could result in publishing only some of the images. These are not updated frequently and are usually preceded by a test run, so this risk seems minimal. The jenkins/build.sh script avoids this issue by first running a build without publishing, then when it succeeds running again with publishing (which will largely pull image layers from the build cache so should be very fast). Python 2.7 ran into build issues installing awscli on ARM, so OS targeting ARM were updated to ensure they include Python 3.8+ and install a newer version of awscli in Python 3. New options were added to buildall.py: - registry: publish images to the specified registry - multi: enables multi-platform builds on RedHat 8 and Ubuntu 20/22 Multi-platform builds require QEMU and a non-default buildx builder. Jenkins job is updated to install QEMU, create a builder, and build multi-platform images. Multi-platform images can't be loaded into a local Docker context (where they would be listed via docker images); we continue to load them there by default with --load, but disable it when doing multi-platform builds. The only way to get images out of the builder for multi-platform builds is to specify a registry. Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb --- M docker/README.md M docker/all/postinstall.sh M docker/buildall.py M docker/redhat8.df M docker/ubuntu2004.df M docker/ubuntu2204.df M jenkins/build.sh 7 files changed, 87 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/69/20469/7 -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/20469/6/docker/buildall.py File docker/buildall.py: http://gerrit.cloudera.org:8080/#/c/20469/6/docker/buildall.py@29 PS6, Line 29: ARM_PLATFORMS = ['redhat8', 'redhat9', 'ubuntu2004', 'ubuntu2204'] Maybe this should be an environment variable? Although we'll need to make changes to other images to build on ARM. Things I've noticed: - the centos 7 tag we use doesn't have ARM images - Python 2.7 seems to fail hard building PyYAML on ARM, so they all need to pull in python3-devel and pip3 -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 12 Sep 2023 21:39:00 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Hello Laszlo Gaal, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20469 to look at the new patch set (#6). Change subject: IMPALA-12436: Create multi-platform builder images .. IMPALA-12436: Create multi-platform builder images Updates the toolchain build image creation process to support publishing multi-platform (aarch64 and amd64) images. Uses docker buildx to produce multi-platform images. This requires QEMU on the builder; these builds are quick, so the overhead of emulating arm is minimal. Currently assumes images are built on amd64 as not all builds specify `--platform`. Moves the publishing step to happen within buildx, because buildx does not support loading multi-platform images in the local Docker registry. This does mean that a build failure could result in publishing only some of the images. These are not updated frequently and are usually preceded by a test run, so this risk seems minimal. The jenkins/build.sh script avoids this issue by first running a build without publishing, then when it succeeds running again with publishing (which will largely pull image layers from the build cache so should be very fast). Python 2.7 ran into build issues installing awscli on ARM, so OS targeting ARM were updated to ensure they include Python 3.8+ and install a newer version of awscli in Python 3. New options were added to buildall.py: - registry: publish images to the specified registry - multi: enables multi-platform builds on RedHat 8 and Ubuntu 20/22 Multi-platform builds require QEMU and a non-default buildx builder. Jenkins job is updated to install QEMU, create a builder, and build multi-platform images. Multi-platform images can't be loaded into a local Docker context (where they would be listed via docker images); we continue to load them there by default with --load, but disable it when doing multi-platform builds. The only way to get images out of the builder for multi-platform builds is to specify a registry. Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb --- M docker/README.md M docker/all/postinstall.sh M docker/buildall.py M docker/redhat8.df M docker/ubuntu2004.df M docker/ubuntu2204.df M jenkins/build.sh 7 files changed, 78 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/69/20469/6 -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 6: (3 comments) http://gerrit.cloudera.org:8080/#/c/20469/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20469/5//COMMIT_MSG@17 PS5, Line 17: Moves the publishing step to happen within buildx, because buildx does : not support loading multi-platform images in the local Docker registry. : This does mean that a build failure could result in publishing only some : of the images. These are not updated frequently and are usually preceded : by a test run, so this risk seems minimal. > Can you update this to describe how we avoid this problem with the two pass Done http://gerrit.cloudera.org:8080/#/c/20469/5/docker/README.md File docker/README.md: http://gerrit.cloudera.org:8080/#/c/20469/5/docker/README.md@45 PS5, Line 45: and > Nit: and Done http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py File docker/buildall.py: http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py@58 PS5, Line 58: build_cmd += ['--builder', args.builder] > Does that impact the x86 single-arch build? Or is that basically fine becau I think it only impacts when you use a docker-container builder (for multi-platform builds), the default "docker" builder works fine. -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 6 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 12 Sep 2023 17:09:07 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Michael Smith has posted comments on this change. (
http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20469/5/jenkins/build.sh
File jenkins/build.sh:
http://gerrit.cloudera.org:8080/#/c/20469/5/jenkins/build.sh@59
PS5, Line 59: ./buildall.py --multi --builder=${BUILDER} ${DISTRO_PARAM}
> Would it make sense to have an environment variable / Jenkins parameter tha
I didn't think so, but I'm open to adding it.
If you push to a registry, then we want to build with multi. If you don't, then
we don't really need the output. Main scenario might be to test building a
single platform, but if you want to test that without multi it's easy enough to
do locally.
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Reviewer: Michael Smith
Gerrit-Comment-Date: Tue, 12 Sep 2023 17:05:07 +
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/20469 )
Change subject: IMPALA-12436: Create multi-platform builder images
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20469/5/jenkins/build.sh
File jenkins/build.sh:
http://gerrit.cloudera.org:8080/#/c/20469/5/jenkins/build.sh@59
PS5, Line 59: ./buildall.py --multi --builder=${BUILDER} ${DISTRO_PARAM}
Would it make sense to have an environment variable / Jenkins parameter that
toggles whether to do the multi-arch build or a single arch build?
--
To view, visit http://gerrit.cloudera.org:8080/20469
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb
Gerrit-Change-Number: 20469
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Reviewer: Michael Smith
Gerrit-Comment-Date: Tue, 12 Sep 2023 00:23:21 +
Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/20469/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20469/5//COMMIT_MSG@17 PS5, Line 17: Moves the publishing step to happen within buildx, because buildx does : not support loading multi-platform images in the local Docker registry. : This does mean that a build failure could result in publishing only some : of the images. These are not updated frequently and are usually preceded : by a test run, so this risk seems minimal. Can you update this to describe how we avoid this problem with the two passes? http://gerrit.cloudera.org:8080/#/c/20469/5/docker/README.md File docker/README.md: http://gerrit.cloudera.org:8080/#/c/20469/5/docker/README.md@45 PS5, Line 45: acd Nit: and http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py File docker/buildall.py: http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py@58 PS5, Line 58: build_cmd += ['--builder', args.builder] > Building everything in the same builder seems to require a fairly new versi Does that impact the x86 single-arch build? Or is that basically fine because it is using the default builder? -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 12 Sep 2023 00:00:41 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py File docker/buildall.py: http://gerrit.cloudera.org:8080/#/c/20469/5/docker/buildall.py@58 PS5, Line 58: build_cmd += ['--builder', args.builder] Building everything in the same builder seems to require a fairly new version of buildx. Docker 20 wasn't new enough; I'm using Docker 24 locally. -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 23:23:05 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py File docker/buildall.py: http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@29 PS2, Line 29: ARM_PLATFORMS = ['redhat8', 'redhat9', 'ubuntu2004', 'ubuntu2204'] : : def main(): > Yeah, the normal path is fine, so it's only the --multi path that would be Done http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@32 PS2, Line 32: nfig(leve > I agree that we don't really need Redhat 9 ARM at the moment, so even if we Ack -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 21:52:53 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Hello Laszlo Gaal, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20469 to look at the new patch set (#5). Change subject: IMPALA-12436: Create multi-platform builder images .. IMPALA-12436: Create multi-platform builder images Updates the toolchain build image creation process to support publishing multi-platform (aarch64 and amd64) images. Uses docker buildx to produce multi-platform images. This requires QEMU on the builder; these builds are quick, so the overhead of emulating arm is minimal. Currently assumes images are built on amd64 as not all builds specify `--platform`. Moves the publishing step to happen within buildx, because buildx does not support loading multi-platform images in the local Docker registry. This does mean that a build failure could result in publishing only some of the images. These are not updated frequently and are usually preceded by a test run, so this risk seems minimal. Python 2.7 ran into build issues installing awscli on ARM, so OS targeting ARM were updated to ensure they include Python 3.8+ and install a newer version of awscli in Python 3. New options were added to buildall.py: - registry: publish images to the specified registry - multi: enables multi-platform builds on RedHat 8 and Ubuntu 20/22 Multi-platform builds require QEMU and a non-default buildx builder. Jenkins job is updated to install QEMU, create a builder, and build multi-platform images. Multi-platform images can't be loaded into a local Docker context (where they would be listed via docker images); we continue to load them there by default with --load, but disable it when doing multi-platform builds. The only way to get images out of the builder for multi-platform builds is to specify a registry. Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb --- M docker/README.md M docker/all/postinstall.sh M docker/buildall.py M docker/redhat8.df M docker/ubuntu2004.df M docker/ubuntu2204.df M jenkins/build.sh 7 files changed, 73 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/69/20469/5 -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py File docker/buildall.py: http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@29 PS2, Line 29: # apt install binfmt-support qemu-user-static qemu-system-x86 : # then create a docker-container builder : # docker buildx create --use > I think the errors you get from Docker are pretty clear. These are only req Yeah, the normal path is fine, so it's only the --multi path that would be a problem. I noticed that we have a README.md file in this directory with some instructions for building the docker images. Maybe we could add a section there about what to install, how to get a builder, how to invoke buildall.py? http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@32 PS2, Line 32: 'redhat8' > We could if we think we'll move CDW to it sometime soon. I was mostly tryin I agree that we don't really need Redhat 9 ARM at the moment, so even if we produce the docker images, we can probably skip the full build until later. -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 21:34:20 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20469 ) Change subject: IMPALA-12436: Create multi-platform builder images .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py File docker/buildall.py: http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@29 PS2, Line 29: # apt install binfmt-support qemu-user-static qemu-system-x86 : # then create a docker-container builder : # docker buildx create --use > Nit: Is there anything we can do to detect if these are appropriately speci I think the errors you get from Docker are pretty clear. These are only required if specifying '--multi', I'll update the comment to clarify that. I made sure most local development workflows would not require QEMU or a new builder, so these should only be needed if testing out multi-platform builds specifically. http://gerrit.cloudera.org:8080/#/c/20469/2/docker/buildall.py@32 PS2, Line 32: 'redhat8' > Should we include redhat9? We could if we think we'll move CDW to it sometime soon. I was mostly trying to keep native-toolchain build varieties down to ones we would use. http://gerrit.cloudera.org:8080/#/c/20469/2/jenkins/build.sh File jenkins/build.sh: http://gerrit.cloudera.org:8080/#/c/20469/2/jenkins/build.sh@65 PS2, Line 65: popd > Something we could do to make sure we only push if all images succeeded is Done -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 11 Sep 2023 21:12:04 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-12436: Create multi-platform builder images
Hello Laszlo Gaal, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20469 to look at the new patch set (#3). Change subject: IMPALA-12436: Create multi-platform builder images .. IMPALA-12436: Create multi-platform builder images Updates the toolchain build image creation process to support publishing multi-platform (aarch64 and amd64) images. Uses docker buildx to produce multi-platform images. This requires QEMU on the builder; these builds are quick, so the overhead of emulating arm is minimal. Currently assumes images are built on amd64 as not all builds specify `--platform`. Moves the publishing step to happen within buildx, because buildx does not support loading multi-platform images in the local Docker registry. This does mean that a build failure could result in publishing only some of the images. These are not updated frequently and are usually preceded by a test run, so this risk seems minimal. Python 2.7 ran into build issues installing awscli on ARM, so OS targeting ARM were updated to ensure they include Python 3.8+ and install a newer version of awscli in Python 3. New options were added to buildall.py: - registry: publish images to the specified registry - multi: enables multi-platform builds on RedHat 8 and Ubuntu 20/22 Multi-platform builds require QEMU and a non-default buildx builder. Jenkins job is updated to install QEMU, create a builder, and build multi-platform images. Multi-platform images can't be loaded into a local Docker context (where they would be listed via docker images); we continue to load them there by default with --load, but disable it when doing multi-platform builds. The only way to get images out of the builder for multi-platform builds is to specify a registry. Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb --- M docker/all/postinstall.sh M docker/buildall.py M docker/redhat8.df M docker/ubuntu2004.df M docker/ubuntu2204.df M jenkins/build.sh 6 files changed, 54 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/69/20469/3 -- To view, visit http://gerrit.cloudera.org:8080/20469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I058deb877f26b48a2d87a65710143bba9a42a5fb Gerrit-Change-Number: 20469 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
