Hi Hannah, Thank you, I know how to build the containers locally, but not how to publish them!
The cwiki says "Publishing images to gcr.io/beam requires permissions in apache-beam-testing project." Can I get access to the testing project (at least temporarily) and what would I need to setup to run the publish target that is shown on cwiki? Thanks, Thomas On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <[email protected]> wrote: > Hi Thomas > > I haven't uploaded any snapshot images yet. Here is how you can create one > from head. > > cd [...]/beam/ > # For Python > > ./gradlew :sdks:python:container:py{version}:docker *where version is > {2,35,36,37}* > # For Java > > ./gradlew -p sdks/java/container docker > # For Go > > ./gradlew -p sdks/go/container docker > > The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot from > head. > > Please let me know if you have any questions. > Hannah > > On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <[email protected]> wrote: > >> I actually found something in [1], but it is 2.15 unfortunately. >> >> [1] >> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30 >> >> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <[email protected]> wrote: >> >>> Thanks for working on this. Do you happen to have publicly accessible >>> snapshots published for your testing currently (even when the final >>> location isn't sorted out)? >>> >>> I would like to use a 2.16 based Python SDK image for working on my >>> downstream project, but could not find anything in >>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot >>> >>> Thanks, >>> Thomas >>> >>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev < >>> [email protected]> wrote: >>> >>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <[email protected]> >>>> wrote: >>>> >>>>> Hi team >>>>> >>>>> I am working on improving docker container support for Beam. We would >>>>> like to publish prebuilt containers for each release version and daily >>>>> snapshot. Current work focuses on release images only and it would be part >>>>> of the release process. >>>>> >>>>> The release images will be pushed to GCR which is publicly >>>>> accessible(pullable). We will use the following locations. >>>>> *Repository*: gcr.io/beam >>>>> *Project*: apache-beam-testing >>>>> More details, including naming and tagging scheme, can be found at >>>>> wiki >>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> >>>>> which >>>>> is written by several contributors. >>>>> >>>>> I would like to discuss these two questions. >>>>> *1. How many tests do we need to run before pushing images to gcr*? >>>>> Publishing artifacts is the last step of the release process, so at >>>>> this moment, we already verified all codebase. In addition, many Jenkins >>>>> tests use containers, so it is already verified several times. Do we need >>>>> to run it again? >>>>> >>>> >>>> In a docker repository, one container image can have multiple tags. One >>>> possibility is that on the last step of the release process, after >>>> sufficient testing, we place a production tag on an image that was already >>>> pushed with a dev tag. >>>> >>>> For example a dev tag may look like: >>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look >>>> like: >>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same >>>> image at the end. >>>> >>>> We should also plan what the process of updating the container image >>>> will look like, if we need to release the image with additional changes, >>>> and how we will test these changes before the final push (or placing >>>> production tag). >>>> >>>> >>>>> >>>>> *2. How many tests do we need to run to validate pushed images?* >>>>> When we push the images, we assume the images would work and pass all >>>>> the tests. After pushing, we should confirm the images are pullable and >>>>> useable. I suggest we run several tests on dataflow with each pushed >>>>> image. >>>>> What do you think? >>>>> >>>> >>>> I think it makes sense to do - Beam runners that use SDK container >>>> images should have some continuously running tests, which periodically >>>> check that all supported images are pullable and still compatible with the >>>> runner. >>>> >>>> This work can be refined later as we explore more during our release >>>>> process. >>>>> Please comment or edit the wiki page or reply to this email with your >>>>> opinions. >>>>> >>>>> Thanks, >>>>> Hannah >>>>> >>>>
