+1
On Mon, May 27, 2019 at 3:35 PM Maximilian Michels <m...@apache.org> wrote: > > +1 > > On 27.05.19 14:04, Robert Bradshaw wrote: > > Sounds like everyone's onboard with the plan. Any chance we could > > publish these for the upcoming 2.13 release? > > > > On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy <lgaj...@apache.org> wrote: > >> > >> +1 to have a registry for images accessible to anyone. For snapshot > >> images, I agree that gcr + apache-beam-testing project seems a good and > >> easy way to start with. > >> > >> Łukasz > >> > >> wt., 22 sty 2019 o 19:43 Mark Liu <mark...@google.com> napisał(a): > >>> > >>> +1 to have an official Beam released container image. > >>> > >>> Also I would propose to add a verification step to (or after) the release > >>> process to do smoke check. Python have ValidatesContainer test that runs > >>> basic pipeline using newly built container for verification. Other sdk > >>> languages can do similar thing or add a common framework. > >>> > >>> Mark > >>> > >>> On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold <amyrv...@google.com> wrote: > >>>> > >>>> +1 This would be great. gcr.io seems like a good option for snapshots > >>>> due to the permissions from jenkins to upload and ability to keep > >>>> snapshots around. > >>>> > >>>> On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang <ruo...@google.com> wrote: > >>>>> > >>>>> +1 This would be a great thing to have. > >>>>> > >>>>> On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka <goe...@google.com> wrote: > >>>>>> > >>>>>> grc.io seems to be a good option. Given that we don't need the hosting > >>>>>> server name in the image name makes it easily changeable later. > >>>>>> > >>>>>> Docker container for Apache Flink is named "flink" and they have > >>>>>> different tags for different releases and configurations > >>>>>> https://hub.docker.com/_/flink .We can follow a similar model and can > >>>>>> name the image as "beam" (beam doesn't seem to be taken on docker hub) > >>>>>> and use tags to distinguish Java/Python/Go and versions etc. > >>>>>> > >>>>>> Tags will look like: > >>>>>> java-SNAPSHOT > >>>>>> java-2.10.1 > >>>>>> python2-SNAPSHOT > >>>>>> python2-2.10.1 > >>>>>> go-SNAPSHOT > >>>>>> go-2.10.1 > >>>>>> > >>>>>> > >>>>>> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay <al...@google.com> wrote: > >>>>>>> > >>>>>>> For snapshots, we could use gcr.io. Permission would not be a problem > >>>>>>> since Jenkins is already correctly setup. The cost will be covered > >>>>>>> under apache-beam-testing project. And since this is only for > >>>>>>> snapshots, it will be only for temporary artifacts not for release > >>>>>>> artifacts. > >>>>>>> > >>>>>>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev > >>>>>>> <valen...@google.com> wrote: > >>>>>>>> > >>>>>>>> +1, releasing containers is a useful process that we need to build > >>>>>>>> in Beam and it is required for FnApi users. Among other reasons, > >>>>>>>> having officially-released Beam SDK harness container images will > >>>>>>>> make it easier for users to do simple customizations to container > >>>>>>>> images, as they will be able to use container image released by Beam > >>>>>>>> as a base image. > >>>>>>>> > >>>>>>>> Good point about potential storage limitations on Bintray. With Beam > >>>>>>>> Release cadence we may quickly exceed the 10 GB quota. It may also > >>>>>>>> affect our decisions as to which images we want to release, for > >>>>>>>> example: do we want to only release one container image with Python > >>>>>>>> 3 interpreter, or do we want to release a container image for each > >>>>>>>> Python 3 minor version that Beam is compatible with. > >>>>>>> > >>>>>>> > >>>>>>> Probably worth a separate discussion. I would favor first releasing a > >>>>>>> python 3 compatible version before figuring out how we would target > >>>>>>> multiple python 3 versions. > >>>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka <goe...@google.com> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay <al...@google.com> > >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka <goe...@google.com> > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> - Could we start from snapshots first and then do it for releases? > >>>>>>>>>>> +1, releasing snapsots first makes sense to me. > >>>>>>>>>>> - For snapshots, do we need to clean old containers after a > >>>>>>>>>>> while? Otherwise I guess we will accumulate lots of containers. > >>>>>>>>>>> For snap shots we can maintain a single snapshot image from git > >>>>>>>>>>> HEAD daily. Docker has the internal image container id which > >>>>>>>>>>> changes everytime an image is changed and pulls new images as > >>>>>>>>>>> needed. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> There is a potential use this may not work with. If a user picks > >>>>>>>>>> up a snaphsot build and want to use it until the next release > >>>>>>>>>> arrives. I guess in that case the user can copy the snapshotted > >>>>>>>>>> container image and rely on that. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> Yes, that should be reasonable. > >>>>>>>>>>> > >>>>>>>>>>> - Do we also need additional code changes for snapshots and > >>>>>>>>>>> releases to default to these specific containers? There could be > >>>>>>>>>>> a version based mechanism to resolve the correct container to use. > >>>>>>>>>>> The current image defaults have username in it. We should be ok > >>>>>>>>>>> by just updating the default image url to published image url. > >>>>>>>>>>> > >>>>>>>>>>> We should also check for pricing and details about Apache-Bintray > >>>>>>>>>>> agreement before pushing images and changing defaults. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> There is information on bintray's pricing page about open source > >>>>>>>>>> projects [1]. I do not know if there is a special apache-bintray > >>>>>>>>>> agreement or not. If there is no special agreement there is a 10GB > >>>>>>>>>> storage limit for using bintray. > >>>>>>>>> > >>>>>>>>> As each image can easily run into Gigs, 10GB might not be > >>>>>>>>> sufficient for future proofing. > >>>>>>>>> We can also register docker image to docker image registry and not > >>>>>>>>> have bintray in the name to later host images on a different vendor > >>>>>>>>> for future proofing. > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> [1] https://bintray.com/account/pricing?tab=account&type=pricing > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay <al...@google.com> > >>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> This sounds like a good idea. Some questions: > >>>>>>>>>>>> > >>>>>>>>>>>> - Could we start from snapshots first and then do it for > >>>>>>>>>>>> releases? > >>>>>>>>>>>> - For snapshots, do we need to clean old containers after a > >>>>>>>>>>>> while? Otherwise I guess we will accumulate lots of containers. > >>>>>>>>>>>> - Do we also need additional code changes for snapshots and > >>>>>>>>>>>> releases to default to these specific containers? There could be > >>>>>>>>>>>> a version based mechanism to resolve the correct container to > >>>>>>>>>>>> use. > >>>>>>>>>>>> > >>>>>>>>>>>> On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka <goe...@google.com> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi All, > >>>>>>>>>>>>> > >>>>>>>>>>>>> As portability/FnApi is taking shape and are compatible with > >>>>>>>>>>>>> ULR and Flink. I wanted to discuss the release plan release of > >>>>>>>>>>>>> SDKHarness Docker images. Of-course users can create their own > >>>>>>>>>>>>> images but it will be useful to have a default image available > >>>>>>>>>>>>> out of box. > >>>>>>>>>>>>> Pre build image are a must for making FnApi available for users > >>>>>>>>>>>>> and not just the developers. > >>>>>>>>>>>>> The other purpose of these images is to be server as base image > >>>>>>>>>>>>> layer for building custom images. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Apache already have bintray repositories for beam. > >>>>>>>>>>>>> https://bintray.com/apache/beam-snapshots-docker > >>>>>>>>>>>>> https://bintray.com/apache/beam-docker > >>>>>>>>>>>>> > >>>>>>>>>>>>> Shall we start pushing Python/Java/Go SDK Harness containers to > >>>>>>>>>>>>> https://bintray.com/apache/beam-docker for beam release and > >>>>>>>>>>>>> maintain daily snapshot at > >>>>>>>>>>>>> https://bintray.com/apache/beam-snapshots-docker ? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> Ankur > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> ================ > >>>>> Ruoyun Huang > >>>>>
