I agree, I think their are few things which have to be though through as part of Portable image release.
* Where to host the images. We can ofcourse have an alias for the image which can point to a different location but the hosting location have to be sort through. * Validation process for the images. * Backward compatibility for the images. Though we can just tag them with release name. I might not have immediate bandwidth to do this so we need to prioritize based on other items we have. On Tue, May 28, 2019 at 5:24 PM Ahmet Altay <al...@google.com> wrote: > Could we first figure out the process (where to push, how to push, > permissions needed, how to validate etc.) as part of the snapshots and > update the release guide based on that? > > On Tue, May 28, 2019 at 2:43 AM Robert Bradshaw <rober...@google.com> > wrote: > >> In the future (read, next release) the SDK will likely have reference >> to the containers, so this will have to be part of the release. > > > Who is working on this change? Could they help with figuring out the > publishing the containers part? > > >> But I >> agree for 2.13 it should be more about figuring out the process and >> not necessarily holding back. >> >> On Mon, May 27, 2019 at 7:42 PM Ankur Goenka <goe...@google.com> wrote: >> > >> > +1 >> > We can release the images with 2.13 but we should not block 2.13 >> release for this. >> > >> > On Mon, May 27, 2019, 8:39 AM Thomas Weise <t...@apache.org> wrote: >> >> >> >> +1 >> >> >> >> >> >> On Mon, May 27, 2019 at 6:56 AM Ismaël Mejía <ieme...@gmail.com> >> wrote: >> >>> >> >>> +1 >> >>> >> >>> On Mon, May 27, 2019 at 3:35 PM Maximilian Michels <m...@apache.org> >> wrote: >> >>> > >> >>> > +1 >> >>> > >> >>> > On 27.05.19 14:04, Robert Bradshaw wrote: >> >>> > > Sounds like everyone's onboard with the plan. Any chance we could >> >>> > > publish these for the upcoming 2.13 release? >> >>> > > >> >>> > > On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy <lgaj...@apache.org> >> wrote: >> >>> > >> >> >>> > >> +1 to have a registry for images accessible to anyone. For >> snapshot images, I agree that gcr + apache-beam-testing project seems a >> good and easy way to start with. >> >>> > >> >> >>> > >> Łukasz >> >>> > >> >> >>> > >> wt., 22 sty 2019 o 19:43 Mark Liu <mark...@google.com> >> napisał(a): >> >>> > >>> >> >>> > >>> +1 to have an official Beam released container image. >> >>> > >>> >> >>> > >>> Also I would propose to add a verification step to (or after) >> the release process to do smoke check. Python have ValidatesContainer test >> that runs basic pipeline using newly built container for verification. >> Other sdk languages can do similar thing or add a common framework. >> >>> > >>> >> >>> > >>> Mark >> >>> > >>> >> >>> > >>> On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold < >> amyrv...@google.com> wrote: >> >>> > >>>> >> >>> > >>>> +1 This would be great. gcr.io seems like a good option for >> snapshots due to the permissions from jenkins to upload and ability to keep >> snapshots around. >> >>> > >>>> >> >>> > >>>> On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang < >> ruo...@google.com> wrote: >> >>> > >>>>> >> >>> > >>>>> +1 This would be a great thing to have. >> >>> > >>>>> >> >>> > >>>>> On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka < >> goe...@google.com> wrote: >> >>> > >>>>>> >> >>> > >>>>>> grc.io seems to be a good option. Given that we don't need >> the hosting server name in the image name makes it easily changeable later. >> >>> > >>>>>> >> >>> > >>>>>> Docker container for Apache Flink is named "flink" and they >> have different tags for different releases and configurations >> https://hub.docker.com/_/flink .We can follow a similar model and can >> name the image as "beam" (beam doesn't seem to be taken on docker hub) and >> use tags to distinguish Java/Python/Go and versions etc. >> >>> > >>>>>> >> >>> > >>>>>> Tags will look like: >> >>> > >>>>>> java-SNAPSHOT >> >>> > >>>>>> java-2.10.1 >> >>> > >>>>>> python2-SNAPSHOT >> >>> > >>>>>> python2-2.10.1 >> >>> > >>>>>> go-SNAPSHOT >> >>> > >>>>>> go-2.10.1 >> >>> > >>>>>> >> >>> > >>>>>> >> >>> > >>>>>> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay < >> al...@google.com> wrote: >> >>> > >>>>>>> >> >>> > >>>>>>> For snapshots, we could use gcr.io. Permission would not >> be a problem since Jenkins is already correctly setup. The cost will be >> covered under apache-beam-testing project. And since this is only for >> snapshots, it will be only for temporary artifacts not for release >> artifacts. >> >>> > >>>>>>> >> >>> > >>>>>>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev < >> valen...@google.com> wrote: >> >>> > >>>>>>>> >> >>> > >>>>>>>> +1, releasing containers is a useful process that we need >> to build in Beam and it is required for FnApi users. Among other reasons, >> having officially-released Beam SDK harness container images will make it >> easier for users to do simple customizations to container images, as they >> will be able to use container image released by Beam as a base image. >> >>> > >>>>>>>> >> >>> > >>>>>>>> Good point about potential storage limitations on Bintray. >> With Beam Release cadence we may quickly exceed the 10 GB quota. It may >> also affect our decisions as to which images we want to release, for >> example: do we want to only release one container image with Python 3 >> interpreter, or do we want to release a container image for each Python 3 >> minor version that Beam is compatible with. >> >>> > >>>>>>> >> >>> > >>>>>>> >> >>> > >>>>>>> Probably worth a separate discussion. I would favor first >> releasing a python 3 compatible version before figuring out how we would >> target multiple python 3 versions. >> >>> > >>>>>>> >> >>> > >>>>>>> >> >>> > >>>>>>>> >> >>> > >>>>>>>> >> >>> > >>>>>>>> On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka < >> goe...@google.com> wrote: >> >>> > >>>>>>>>> >> >>> > >>>>>>>>> >> >>> > >>>>>>>>> >> >>> > >>>>>>>>> On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay < >> al...@google.com> wrote: >> >>> > >>>>>>>>>> >> >>> > >>>>>>>>>> >> >>> > >>>>>>>>>> >> >>> > >>>>>>>>>> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka < >> goe...@google.com> wrote: >> >>> > >>>>>>>>>>> >> >>> > >>>>>>>>>>> - Could we start from snapshots first and then do it >> for releases? >> >>> > >>>>>>>>>>> +1, releasing snapsots first makes sense to me. >> >>> > >>>>>>>>>>> - For snapshots, do we need to clean old containers >> after a while? Otherwise I guess we will accumulate lots of containers. >> >>> > >>>>>>>>>>> For snap shots we can maintain a single snapshot image >> from git HEAD daily. Docker has the internal image container id which >> changes everytime an image is changed and pulls new images as needed. >> >>> > >>>>>>>>>> >> >>> > >>>>>>>>>> >> >>> > >>>>>>>>>> There is a potential use this may not work with. If a >> user picks up a snaphsot build and want to use it until the next release >> arrives. I guess in that case the user can copy the snapshotted container >> image and rely on that. >> >>> > >>>>>>>>>> >> >>> > >>>>>>>>> >> >>> > >>>>>>>>> Yes, that should be reasonable. >> >>> > >>>>>>>>>>> >> >>> > >>>>>>>>>>> - Do we also need additional code changes for snapshots >> and releases to default to these specific containers? There could be a >> version based mechanism to resolve the correct container to use. >> >>> > >>>>>>>>>>> The current image defaults have username in it. We >> should be ok by just updating the default image url to published image url. >> >>> > >>>>>>>>>>> >> >>> > >>>>>>>>>>> We should also check for pricing and details about >> Apache-Bintray agreement before pushing images and changing defaults. >> >>> > >>>>>>>>>> >> >>> > >>>>>>>>>> >> >>> > >>>>>>>>>> There is information on bintray's pricing page about >> open source projects [1]. I do not know if there is a special >> apache-bintray agreement or not. If there is no special agreement there is >> a 10GB storage limit for using bintray. >> >>> > >>>>>>>>> >> >>> > >>>>>>>>> As each image can easily run into Gigs, 10GB might not be >> sufficient for future proofing. >> >>> > >>>>>>>>> We can also register docker image to docker image >> registry and not have bintray in the name to later host images on a >> different vendor for future proofing. >> >>> > >>>>>>>>> >> >>> > >>>>>>>>>> >> >>> > >>>>>>>>>> [1] >> https://bintray.com/account/pricing?tab=account&type=pricing >> >>> > >>>>>>>>>> >> >>> > >>>>>>>>>>> >> >>> > >>>>>>>>>>> >> >>> > >>>>>>>>>>> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay < >> al...@google.com> wrote: >> >>> > >>>>>>>>>>>> >> >>> > >>>>>>>>>>>> This sounds like a good idea. Some questions: >> >>> > >>>>>>>>>>>> >> >>> > >>>>>>>>>>>> - Could we start from snapshots first and then do it >> for releases? >> >>> > >>>>>>>>>>>> - For snapshots, do we need to clean old containers >> after a while? Otherwise I guess we will accumulate lots of containers. >> >>> > >>>>>>>>>>>> - Do we also need additional code changes for >> snapshots and releases to default to these specific containers? There could >> be a version based mechanism to resolve the correct container to use. >> >>> > >>>>>>>>>>>> >> >>> > >>>>>>>>>>>> On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka < >> goe...@google.com> wrote: >> >>> > >>>>>>>>>>>>> >> >>> > >>>>>>>>>>>>> Hi All, >> >>> > >>>>>>>>>>>>> >> >>> > >>>>>>>>>>>>> As portability/FnApi is taking shape and are >> compatible with ULR and Flink. I wanted to discuss the release plan release >> of SDKHarness Docker images. Of-course users can create their own images >> but it will be useful to have a default image available out of box. >> >>> > >>>>>>>>>>>>> Pre build image are a must for making FnApi available >> for users and not just the developers. >> >>> > >>>>>>>>>>>>> The other purpose of these images is to be server as >> base image layer for building custom images. >> >>> > >>>>>>>>>>>>> >> >>> > >>>>>>>>>>>>> Apache already have bintray repositories for beam. >> >>> > >>>>>>>>>>>>> https://bintray.com/apache/beam-snapshots-docker >> >>> > >>>>>>>>>>>>> https://bintray.com/apache/beam-docker >> >>> > >>>>>>>>>>>>> >> >>> > >>>>>>>>>>>>> Shall we start pushing Python/Java/Go SDK Harness >> containers to https://bintray.com/apache/beam-docker for beam release >> and maintain daily snapshot at >> https://bintray.com/apache/beam-snapshots-docker ? >> >>> > >>>>>>>>>>>>> >> >>> > >>>>>>>>>>>>> Thanks, >> >>> > >>>>>>>>>>>>> Ankur >> >>> > >>>>> >> >>> > >>>>> >> >>> > >>>>> >> >>> > >>>>> -- >> >>> > >>>>> ================ >> >>> > >>>>> Ruoyun Huang >> >>> > >>>>> >> >
