This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push: new 9a4e1144532 Add instructions for Go custom container builds, link from 2.55.0 release notes. (#30290) 9a4e1144532 is described below commit 9a4e1144532291244cfe6fde7f99655de1e9c88d Author: Robert Burke <lostl...@users.noreply.github.com> AuthorDate: Tue Feb 13 13:23:26 2024 -0800 Add instructions for Go custom container builds, link from 2.55.0 release notes. (#30290) * Stopgap message in release notes. * Recommend 2.54.0 not 2.53.0. * Add from scratch instructions. * remove specific go version * ws * redundant line * rm more ws --------- Co-authored-by: lostluck <13907733+lostl...@users.noreply.github.com> --- CHANGES.md | 3 ++ .../en/documentation/runtime/environments.md | 40 +++++++++++++++++++++- 2 files changed, 42 insertions(+), 1 deletion(-) diff --git a/CHANGES.md b/CHANGES.md index bda5937a93d..607f1a1185f 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -72,6 +72,9 @@ * X behavior was changed ([#X](https://github.com/apache/beam/issues/X)). * Arrow version was bumped to 15.0.0 from 5.0.0 ([#30181](https://github.com/apache/beam/pull/30181)). +* Go SDK users who build custom worker containers may run into issues with the move to distroless containers as a base (see Security Fixes). + * The issue stems from distroless containers lacking additional tools, which current custom container processes may rely on. + * See https://beam.apache.org/documentation/runtime/environments/#from-scratch-go for instructions on building and using a custom container. ## Deprecations diff --git a/website/www/site/content/en/documentation/runtime/environments.md b/website/www/site/content/en/documentation/runtime/environments.md index c860816e300..d9a42db29e2 100644 --- a/website/www/site/content/en/documentation/runtime/environments.md +++ b/website/www/site/content/en/documentation/runtime/environments.md @@ -29,7 +29,7 @@ You may want to customize container images for many reasons, including: * Launching third-party software in the worker environment * Further customizing the execution environment - This guide describes how to create and use customized containers for the Beam SDK. + This guide describes how to create and use customized containers for the Beam SDKs. ### Prerequisites @@ -218,6 +218,44 @@ Beam offers a way to provide your own custom container image. The easiest way to docker push "${IMAGE_NAME}:${TAG}" ``` +#### Building a compatible container image from scratch (Go) {#from-scratch-go} + +From the 2.55.0 release, the Beam Go SDK has moved to using [distroless images](https://github.com/GoogleContainerTools/distroless) as a base. +These images have a reduced security attack surface by not including common tools and utilities. +This may cause difficulties customizing the image with using one of the above approaches. +As a fallback, it's possible to build a custom image from scratch, by building a matching boot loader, and setting +that as the container's entry point. + +For example, if it's preferable to use alpine as the container OS your multi-stage docker file might +look like the following: + +``` +FROM golang:latest-alpine AS build_base + +# Set the Current Working Directory inside the container +WORKDIR /tmp/beam + +# Build the Beam Go bootloader, to the local directory, matching your Beam version. +# Similar go targets exist for other SDK languages. +RUN GOBIN=`pwd` go install github.com/apache/beam/sdks/v2/go/container@v2.53.0 + +# Set the real base image. +FROM alpine:3.9 +RUN apk add ca-certificates + +# The following are required for the container to operate correctly. +# Copy the boot loader `container` to the image. +COPY --from=build_base /tmp/beam/container /opt/apache/beam/boot + +# Set the container to use the newly built boot loader. +ENTRYPOINT ["/opt/apache/beam/boot"] +``` + +Build and push the new image as when [modifying an existing base image](#modify-existing-base-image) above. + +>**NOTE**: Java and Python require additional dependencies, such as their runtimes, and SDK packages for +> a valid container image. The bootloader isn't sufficient for creating a custom container for these SDKs. + ## Running pipelines with custom container images {#running-pipelines} The common method for providing a container image requires using the