On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <[email protected]> wrote:
> Hi team > > I am working on improving docker container support for Beam. We would like > to publish prebuilt containers for each release version and daily snapshot. > Current work focuses on release images only and it would be part of the > release process. > > The release images will be pushed to GCR which is publicly > accessible(pullable). We will use the following locations. > *Repository*: gcr.io/beam > *Project*: apache-beam-testing > More details, including naming and tagging scheme, can be found at wiki > <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> > which > is written by several contributors. > > I would like to discuss these two questions. > *1. How many tests do we need to run before pushing images to gcr*? > Publishing artifacts is the last step of the release process, so at this > moment, we already verified all codebase. In addition, many Jenkins tests > use containers, so it is already verified several times. Do we need to run > it again? > In a docker repository, one container image can have multiple tags. One possibility is that on the last step of the release process, after sufficient testing, we place a production tag on an image that was already pushed with a dev tag. For example a dev tag may look like: gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look like: gcr.io/apache-beam/python37:2.16.0 and both will refer to the same image at the end. We should also plan what the process of updating the container image will look like, if we need to release the image with additional changes, and how we will test these changes before the final push (or placing production tag). > > *2. How many tests do we need to run to validate pushed images?* > When we push the images, we assume the images would work and pass all the > tests. After pushing, we should confirm the images are pullable and > useable. I suggest we run several tests on dataflow with each pushed image. > What do you think? > I think it makes sense to do - Beam runners that use SDK container images should have some continuously running tests, which periodically check that all supported images are pullable and still compatible with the runner. This work can be refined later as we explore more during our release > process. > Please comment or edit the wiki page or reply to this email with your > opinions. > > Thanks, > Hannah >
