Awesome, thank you!
On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <[email protected]> wrote: > Hi Thomas > > I created snapshot images from head as of around 2PM today. > You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot. > > Thanks, > Hannah > > On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <[email protected]> wrote: > >> Hi Hannah, >> >> Thank you, I know how to build the containers locally, but not how to >> publish them! >> >> The cwiki says "Publishing images to gcr.io/beam requires permissions in >> apache-beam-testing project." >> >> Can I get access to the testing project (at least temporarily) and what >> would I need to setup to run the publish target that is shown on cwiki? >> >> Thanks, >> Thomas >> >> >> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <[email protected]> >> wrote: >> >>> Hi Thomas >>> >>> I haven't uploaded any snapshot images yet. Here is how you can create >>> one from head. >>> > cd [...]/beam/ >>> # For Python >>> > ./gradlew :sdks:python:container:py{version}:docker *where version is >>> {2,35,36,37}* >>> # For Java >>> > ./gradlew -p sdks/java/container docker >>> # For Go >>> > ./gradlew -p sdks/go/container docker >>> >>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot from >>> head. >>> >>> Please let me know if you have any questions. >>> Hannah >>> >>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <[email protected]> wrote: >>> >>>> I actually found something in [1], but it is 2.15 unfortunately. >>>> >>>> [1] >>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30 >>>> >>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <[email protected]> wrote: >>>> >>>>> Thanks for working on this. Do you happen to have publicly accessible >>>>> snapshots published for your testing currently (even when the final >>>>> location isn't sorted out)? >>>>> >>>>> I would like to use a 2.16 based Python SDK image for working on my >>>>> downstream project, but could not find anything in >>>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot >>>>> >>>>> Thanks, >>>>> Thomas >>>>> >>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev < >>>>> [email protected]> wrote: >>>>> >>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi team >>>>>>> >>>>>>> I am working on improving docker container support for Beam. We >>>>>>> would like to publish prebuilt containers for each release version and >>>>>>> daily snapshot. Current work focuses on release images only and it >>>>>>> would be >>>>>>> part of the release process. >>>>>>> >>>>>>> The release images will be pushed to GCR which is publicly >>>>>>> accessible(pullable). We will use the following locations. >>>>>>> *Repository*: gcr.io/beam >>>>>>> *Project*: apache-beam-testing >>>>>>> More details, including naming and tagging scheme, can be found at >>>>>>> wiki >>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process> >>>>>>> which >>>>>>> is written by several contributors. >>>>>>> >>>>>>> I would like to discuss these two questions. >>>>>>> *1. How many tests do we need to run before pushing images to gcr*? >>>>>>> Publishing artifacts is the last step of the release process, so at >>>>>>> this moment, we already verified all codebase. In addition, many Jenkins >>>>>>> tests use containers, so it is already verified several times. Do we >>>>>>> need >>>>>>> to run it again? >>>>>>> >>>>>> >>>>>> In a docker repository, one container image can have multiple tags. >>>>>> One possibility is that on the last step of the release process, after >>>>>> sufficient testing, we place a production tag on an image that was >>>>>> already >>>>>> pushed with a dev tag. >>>>>> >>>>>> For example a dev tag may look like: >>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look >>>>>> like: >>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same >>>>>> image at the end. >>>>>> >>>>>> We should also plan what the process of updating the container image >>>>>> will look like, if we need to release the image with additional changes, >>>>>> and how we will test these changes before the final push (or placing >>>>>> production tag). >>>>>> >>>>>> >>>>>>> >>>>>>> *2. How many tests do we need to run to validate pushed images?* >>>>>>> When we push the images, we assume the images would work and pass >>>>>>> all the tests. After pushing, we should confirm the images are pullable >>>>>>> and >>>>>>> useable. I suggest we run several tests on dataflow with each pushed >>>>>>> image. >>>>>>> What do you think? >>>>>>> >>>>>> >>>>>> I think it makes sense to do - Beam runners that use SDK container >>>>>> images should have some continuously running tests, which periodically >>>>>> check that all supported images are pullable and still compatible with >>>>>> the >>>>>> runner. >>>>>> >>>>>> This work can be refined later as we explore more during our release >>>>>>> process. >>>>>>> Please comment or edit the wiki page or reply to this email with >>>>>>> your opinions. >>>>>>> >>>>>>> Thanks, >>>>>>> Hannah >>>>>>> >>>>>>
