I agree, I think their are few things which have to be though through as
part of Portable image release.

* Where to host the images. We can ofcourse have an alias for the image
which can point to a different location but the hosting location have to be
sort through.
* Validation process for the images.
* Backward compatibility for the images. Though we can just tag them with
release name.

I might not have immediate bandwidth to do this so we need to prioritize
based on other items we have.


On Tue, May 28, 2019 at 5:24 PM Ahmet Altay <al...@google.com> wrote:

> Could we first figure out the process (where to push, how to push,
> permissions needed, how to validate etc.) as part of the snapshots and
> update the release guide based on that?
>
> On Tue, May 28, 2019 at 2:43 AM Robert Bradshaw <rober...@google.com>
> wrote:
>
>> In the future (read, next release) the SDK will likely have reference
>> to the containers, so this will have to be part of the release.
>
>
> Who is working on this change? Could they help with figuring out the
> publishing the containers part?
>
>
>> But I
>> agree for 2.13 it should be more about figuring out the process and
>> not necessarily holding back.
>>
>> On Mon, May 27, 2019 at 7:42 PM Ankur Goenka <goe...@google.com> wrote:
>> >
>> > +1
>> > We can release the images with 2.13 but we should not block 2.13
>> release for this.
>> >
>> > On Mon, May 27, 2019, 8:39 AM Thomas Weise <t...@apache.org> wrote:
>> >>
>> >> +1
>> >>
>> >>
>> >> On Mon, May 27, 2019 at 6:56 AM Ismaël Mejía <ieme...@gmail.com>
>> wrote:
>> >>>
>> >>> +1
>> >>>
>> >>> On Mon, May 27, 2019 at 3:35 PM Maximilian Michels <m...@apache.org>
>> wrote:
>> >>> >
>> >>> > +1
>> >>> >
>> >>> > On 27.05.19 14:04, Robert Bradshaw wrote:
>> >>> > > Sounds like everyone's onboard with the plan. Any chance we could
>> >>> > > publish these for the upcoming 2.13 release?
>> >>> > >
>> >>> > > On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy <lgaj...@apache.org>
>> wrote:
>> >>> > >>
>> >>> > >> +1 to have a registry for images accessible to anyone. For
>> snapshot images, I agree that gcr + apache-beam-testing project seems a
>> good and easy way to start with.
>> >>> > >>
>> >>> > >> Łukasz
>> >>> > >>
>> >>> > >> wt., 22 sty 2019 o 19:43 Mark Liu <mark...@google.com>
>> napisał(a):
>> >>> > >>>
>> >>> > >>> +1 to have an official Beam released container image.
>> >>> > >>>
>> >>> > >>> Also I would propose to add a verification step to (or after)
>> the release process to do smoke check. Python have ValidatesContainer test
>> that runs basic pipeline using newly built container for verification.
>> Other sdk languages can do similar thing or add a common framework.
>> >>> > >>>
>> >>> > >>> Mark
>> >>> > >>>
>> >>> > >>> On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold <
>> amyrv...@google.com> wrote:
>> >>> > >>>>
>> >>> > >>>> +1 This would be great. gcr.io seems like a good option for
>> snapshots due to the permissions from jenkins to upload and ability to keep
>> snapshots around.
>> >>> > >>>>
>> >>> > >>>> On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang <
>> ruo...@google.com> wrote:
>> >>> > >>>>>
>> >>> > >>>>> +1 This would be a great thing to have.
>> >>> > >>>>>
>> >>> > >>>>> On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka <
>> goe...@google.com> wrote:
>> >>> > >>>>>>
>> >>> > >>>>>> grc.io seems to be a good option. Given that we don't need
>> the hosting server name in the image name makes it easily changeable later.
>> >>> > >>>>>>
>> >>> > >>>>>> Docker container for Apache Flink is named "flink" and they
>> have different tags for different releases and configurations
>> https://hub.docker.com/_/flink .We can follow a similar model and can
>> name the image as "beam" (beam doesn't seem to be taken on docker hub) and
>> use tags to distinguish Java/Python/Go and versions etc.
>> >>> > >>>>>>
>> >>> > >>>>>> Tags will look like:
>> >>> > >>>>>> java-SNAPSHOT
>> >>> > >>>>>> java-2.10.1
>> >>> > >>>>>> python2-SNAPSHOT
>> >>> > >>>>>> python2-2.10.1
>> >>> > >>>>>> go-SNAPSHOT
>> >>> > >>>>>> go-2.10.1
>> >>> > >>>>>>
>> >>> > >>>>>>
>> >>> > >>>>>> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay <
>> al...@google.com> wrote:
>> >>> > >>>>>>>
>> >>> > >>>>>>> For snapshots, we could use gcr.io. Permission would not
>> be a problem since Jenkins is already correctly setup. The cost will be
>> covered under apache-beam-testing project. And since this is only for
>> snapshots, it will be only for temporary artifacts not for release
>> artifacts.
>> >>> > >>>>>>>
>> >>> > >>>>>>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>> >>> > >>>>>>>>
>> >>> > >>>>>>>> +1, releasing containers is a useful process that we need
>> to build in Beam and it is required for FnApi users. Among other reasons,
>> having officially-released Beam SDK harness container images will make it
>> easier for users to do simple customizations to  container images, as they
>> will be able to use container image released by Beam as a base image.
>> >>> > >>>>>>>>
>> >>> > >>>>>>>> Good point about potential storage limitations on Bintray.
>> With Beam Release cadence we may quickly exceed the 10 GB quota. It may
>> also affect our decisions as to which images we want to release, for
>> example: do we want to only release one container image with Python 3
>> interpreter, or do we want to release a container image for each Python 3
>> minor version that Beam is compatible with.
>> >>> > >>>>>>>
>> >>> > >>>>>>>
>> >>> > >>>>>>> Probably worth a separate discussion. I would favor first
>> releasing a python 3 compatible version before figuring out how we would
>> target multiple python 3 versions.
>> >>> > >>>>>>>
>> >>> > >>>>>>>
>> >>> > >>>>>>>>
>> >>> > >>>>>>>>
>> >>> > >>>>>>>> On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka <
>> goe...@google.com> wrote:
>> >>> > >>>>>>>>>
>> >>> > >>>>>>>>>
>> >>> > >>>>>>>>>
>> >>> > >>>>>>>>> On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay <
>> al...@google.com> wrote:
>> >>> > >>>>>>>>>>
>> >>> > >>>>>>>>>>
>> >>> > >>>>>>>>>>
>> >>> > >>>>>>>>>> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka <
>> goe...@google.com> wrote:
>> >>> > >>>>>>>>>>>
>> >>> > >>>>>>>>>>> - Could we start from snapshots first and then do it
>> for releases?
>> >>> > >>>>>>>>>>> +1, releasing snapsots first makes sense to me.
>> >>> > >>>>>>>>>>> - For snapshots, do we need to clean old containers
>> after a while? Otherwise I guess we will accumulate lots of containers.
>> >>> > >>>>>>>>>>> For snap shots we can maintain a single snapshot image
>> from git HEAD daily. Docker has the internal image container id which
>> changes everytime an image is changed and pulls new images as needed.
>> >>> > >>>>>>>>>>
>> >>> > >>>>>>>>>>
>> >>> > >>>>>>>>>> There is a potential use this may not work with. If a
>> user picks up a snaphsot build and want to use it until the next release
>> arrives. I guess in that case the user can copy the snapshotted container
>> image and rely on that.
>> >>> > >>>>>>>>>>
>> >>> > >>>>>>>>>
>> >>> > >>>>>>>>> Yes, that should be reasonable.
>> >>> > >>>>>>>>>>>
>> >>> > >>>>>>>>>>> - Do we also need additional code changes for snapshots
>> and releases to default to these specific containers? There could be a
>> version based mechanism to resolve the correct container to use.
>> >>> > >>>>>>>>>>> The current image defaults have username in it. We
>> should be ok by just updating the default image url to published image url.
>> >>> > >>>>>>>>>>>
>> >>> > >>>>>>>>>>> We should also check for pricing and details about
>> Apache-Bintray agreement before pushing images and changing defaults.
>> >>> > >>>>>>>>>>
>> >>> > >>>>>>>>>>
>> >>> > >>>>>>>>>> There is information on bintray's pricing page about
>> open source projects [1]. I do not know if there is a special
>> apache-bintray agreement or not. If there is no special agreement there is
>> a 10GB storage limit for using bintray.
>> >>> > >>>>>>>>>
>> >>> > >>>>>>>>> As each image can easily run into Gigs, 10GB might not be
>> sufficient for future proofing.
>> >>> > >>>>>>>>> We can also register docker image to docker image
>> registry and not have bintray in the name to later host images on a
>> different vendor for future proofing.
>> >>> > >>>>>>>>>
>> >>> > >>>>>>>>>>
>> >>> > >>>>>>>>>> [1]
>> https://bintray.com/account/pricing?tab=account&type=pricing
>> >>> > >>>>>>>>>>
>> >>> > >>>>>>>>>>>
>> >>> > >>>>>>>>>>>
>> >>> > >>>>>>>>>>> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay <
>> al...@google.com> wrote:
>> >>> > >>>>>>>>>>>>
>> >>> > >>>>>>>>>>>> This sounds like a good idea. Some questions:
>> >>> > >>>>>>>>>>>>
>> >>> > >>>>>>>>>>>> - Could we start from snapshots first and then do it
>> for releases?
>> >>> > >>>>>>>>>>>> - For snapshots, do we need to clean old containers
>> after a while? Otherwise I guess we will accumulate lots of containers.
>> >>> > >>>>>>>>>>>> - Do we also need additional code changes for
>> snapshots and releases to default to these specific containers? There could
>> be a version based mechanism to resolve the correct container to use.
>> >>> > >>>>>>>>>>>>
>> >>> > >>>>>>>>>>>> On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka <
>> goe...@google.com> wrote:
>> >>> > >>>>>>>>>>>>>
>> >>> > >>>>>>>>>>>>> Hi All,
>> >>> > >>>>>>>>>>>>>
>> >>> > >>>>>>>>>>>>> As portability/FnApi is taking shape and are
>> compatible with ULR and Flink. I wanted to discuss the release plan release
>> of SDKHarness Docker images. Of-course users can create their own images
>> but it will be useful to have a default image available out of box.
>> >>> > >>>>>>>>>>>>> Pre build image are a must for making FnApi available
>> for users and not just the developers.
>> >>> > >>>>>>>>>>>>> The other purpose of these images is to be server as
>> base image layer for building custom images.
>> >>> > >>>>>>>>>>>>>
>> >>> > >>>>>>>>>>>>> Apache already have bintray repositories for beam.
>> >>> > >>>>>>>>>>>>> https://bintray.com/apache/beam-snapshots-docker
>> >>> > >>>>>>>>>>>>> https://bintray.com/apache/beam-docker
>> >>> > >>>>>>>>>>>>>
>> >>> > >>>>>>>>>>>>> Shall we start pushing Python/Java/Go SDK Harness
>> containers to https://bintray.com/apache/beam-docker for beam release
>> and maintain daily snapshot at
>> https://bintray.com/apache/beam-snapshots-docker ?
>> >>> > >>>>>>>>>>>>>
>> >>> > >>>>>>>>>>>>> Thanks,
>> >>> > >>>>>>>>>>>>> Ankur
>> >>> > >>>>>
>> >>> > >>>>>
>> >>> > >>>>>
>> >>> > >>>>> --
>> >>> > >>>>> ================
>> >>> > >>>>> Ruoyun  Huang
>> >>> > >>>>>
>>
>

Reply via email to