We do link from https://beam.apache.org/documentation/runtime/environments/.
On Tue, Nov 19, 2019 at 10:59 AM Robert Bradshaw <rober...@google.com> wrote: > We should probably add a link to these from our site as well, for > visibility. > > On Tue, Nov 19, 2019 at 10:56 AM Kyle Weaver <kcwea...@google.com> wrote: > > > > +1 Thanks for bringing that up Chad, I had the same problem locating the > docker images on Docker hub (searching "apachebeam" is the only way that > seems to work). > > > > Another little thing I noticed is that > https://hub.docker.com/u/apachebeam is mostly empty. It'd be good to at > least add the Beam logo as a profile picture and a link to the website in > the description. > > > > On Tue, Nov 19, 2019 at 10:21 AM Chad Dombrova <chad...@gmail.com> > wrote: > >> > >> Hi all, > >> I think it might be good to update the description of the beam docker > images and add some descriptive tags, because searching for "apache beam" > in docker hub does not turn up anything: > https://hub.docker.com/search?q=apache%20beam&type=image. > >> > >> I clicked through 10 pages worth and couldn't find it. Maybe I missed > something, but it clearly shouldn't be this hard. I did eventually manage > to find it through the docs. > >> > >> Also, googling "apache beam python docker" also does not yield anything > useful. In fact, it turns up an unofficial apache beam docker hub image. > >> > >> One thing I noticed is that images designated as "Official Images" get > listed first, so it would be good to get that done as well. > >> > >> thanks! > >> chad > >> > >> > >> > >> > >> On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang <hannahji...@google.com> > wrote: > >>> > >>> Hi team > >>> > >>> I haven't received any objections, so will proceed with settings > mentioned in a previous email. > >>> > >>> A reminder to PMC members, please let me know your docker hub id if > you want to be an admin. > >>> > >>> Thanks, > >>> Hannah > >>> > >>> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka <goe...@google.com> wrote: > >>>> > >>>> Please ignore the previous email. I was looking at the older document > in the mail thread. > >>>> > >>>> On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka <goe...@google.com> > wrote: > >>>>> > >>>>> I think sdk in the name is obsolete as they are all under sdks name > space. > >>>>> > >>>>> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang <hannahji...@google.com> > wrote: > >>>>>> > >>>>>> Hi Team > >>>>>> > >>>>>> Thanks for all the comments about beam containers. > >>>>>> After considering various opinions and investigating gcr and docker > hub, we decided to push images to docker hub. > >>>>>> > >>>>>> Each image will have two tags, {version}_rc and {version}. > {version} tag will be added after the release candidate image is verified. > >>>>>> Meanwhile, we will have latest tag for each repository, which > always points to the most recent verified release image, so users can pull > it by default. > >>>>>> > >>>>>> Docker hub doesn't support leveled repository, which means we > should follow repository:tag format. > >>>>>> it's too general if we use {language_version} as repository for SDK > images. (version is added when we support multiple versions.) > >>>>>> So I would like to include sdk to repository. Images generated at > local will also have the same name. > >>>>>> Here are some examples: > >>>>>> > >>>>>> python2.7_sdk:2.15.0 > >>>>>> java_sdk:2.15.0_rc > >>>>>> go_sdk:latest > >>>>>> > >>>>>> I will proceed with this format if there is no strong opposition by > tomorrow noon(PST). > >>>>>> > >>>>>> To PMC members: > >>>>>> Permission control will follow the pypi model. All interested PMC > members will be added as admins and release managers will be granted push > permission. > >>>>>> Please let me know your docker id if you want to be added as an > admin. > >>>>>> > >>>>>> Thanks, > >>>>>> Hannah > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise <t...@apache.org> wrote: > >>>>>>> > >>>>>>> This will greatly simplify trying out portable runners: > https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster > >>>>>>> > >>>>>>> Can't wait for following to disappear from the instructions page: > ./gradlew :sdks:python:container:docker > >>>>>>> > >>>>>>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise <t...@apache.org> > wrote: > >>>>>>>> > >>>>>>>> Awesome, thank you! > >>>>>>>> > >>>>>>>> > >>>>>>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang < > hannahji...@google.com> wrote: > >>>>>>>>> > >>>>>>>>> Hi Thomas > >>>>>>>>> > >>>>>>>>> I created snapshot images from head as of around 2PM today. > >>>>>>>>> You can pull images from > gcr.io/apache-beam-testing/beam/sdks/snapshot. > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Hannah > >>>>>>>>> > >>>>>>>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <t...@apache.org> > wrote: > >>>>>>>>>> > >>>>>>>>>> Hi Hannah, > >>>>>>>>>> > >>>>>>>>>> Thank you, I know how to build the containers locally, but not > how to publish them! > >>>>>>>>>> > >>>>>>>>>> The cwiki says "Publishing images to gcr.io/beam requires > permissions in apache-beam-testing project." > >>>>>>>>>> > >>>>>>>>>> Can I get access to the testing project (at least temporarily) > and what would I need to setup to run the publish target that is shown on > cwiki? > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Thomas > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang < > hannahji...@google.com> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Hi Thomas > >>>>>>>>>>> > >>>>>>>>>>> I haven't uploaded any snapshot images yet. Here is how you > can create one from head. > >>>>>>>>>>> > cd [...]/beam/ > >>>>>>>>>>> # For Python > >>>>>>>>>>> > ./gradlew :sdks:python:container:py{version}:docker where > version is {2,35,36,37} > >>>>>>>>>>> # For Java > >>>>>>>>>>> > ./gradlew -p sdks/java/container docker > >>>>>>>>>>> # For Go > >>>>>>>>>>> > ./gradlew -p sdks/go/container docker > >>>>>>>>>>> > >>>>>>>>>>> The 2.15 one is just for testing, not a real 2.15.0, nor a > snapshot from head. > >>>>>>>>>>> > >>>>>>>>>>> Please let me know if you have any questions. > >>>>>>>>>>> Hannah > >>>>>>>>>>> > >>>>>>>>>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <t...@apache.org> > wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> I actually found something in [1], but it is 2.15 > unfortunately. > >>>>>>>>>>>> > >>>>>>>>>>>> [1] > https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30 > >>>>>>>>>>>> > >>>>>>>>>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <t...@apache.org> > wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks for working on this. Do you happen to have publicly > accessible snapshots published for your testing currently (even when the > final location isn't sorted out)? > >>>>>>>>>>>>> > >>>>>>>>>>>>> I would like to use a 2.16 based Python SDK image for > working on my downstream project, but could not find anything in > gcr.io/apache-beam-testing/beam/sdks/rc/snapshot > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> Thomas > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev < > valen...@google.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang < > hannahji...@google.com> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi team > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I am working on improving docker container support for > Beam. We would like to publish prebuilt containers for each release version > and daily snapshot. Current work focuses on release images only and it > would be part of the release process. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> The release images will be pushed to GCR which is publicly > accessible(pullable). We will use the following locations. > >>>>>>>>>>>>>>> Repository: gcr.io/beam > >>>>>>>>>>>>>>> Project: apache-beam-testing > >>>>>>>>>>>>>>> More details, including naming and tagging scheme, can be > found at wiki which is written by several contributors. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I would like to discuss these two questions. > >>>>>>>>>>>>>>> 1. How many tests do we need to run before pushing images > to gcr? > >>>>>>>>>>>>>>> Publishing artifacts is the last step of the release > process, so at this moment, we already verified all codebase. In addition, > many Jenkins tests use containers, so it is already verified several times. > Do we need to run it again? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> In a docker repository, one container image can have > multiple tags. One possibility is that on the last step of the release > process, after sufficient testing, we place a production tag on an image > that was already pushed with a dev tag. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> For example a dev tag may look like: > >>>>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag > may look like: > >>>>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to > the same image at the end. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> We should also plan what the process of updating the > container image will look like, if we need to release the image with > additional changes, and how we will test these changes before the final > push (or placing production tag). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 2. How many tests do we need to run to validate pushed > images? > >>>>>>>>>>>>>>> When we push the images, we assume the images would work > and pass all the tests. After pushing, we should confirm the images are > pullable and useable. I suggest we run several tests on dataflow with each > pushed image. What do you think? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I think it makes sense to do - Beam runners that use SDK > container images should have some continuously running tests, which > periodically check that all supported images are pullable and still > compatible with the runner. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> This work can be refined later as we explore more during > our release process. > >>>>>>>>>>>>>>> Please comment or edit the wiki page or reply to this > email with your opinions. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>> Hannah >
