I think sdk in the name is obsolete as they are all under sdks name space.
On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang <hannahji...@google.com> wrote:
> Hi Team
>
> Thanks for all the comments about beam containers.
> After considering various opinions and investigating gcr and docker hub,
> we decided to push images to docker hub.
>
> Each image will have two tags, {version}_rc and {version}. {version} tag
> will be added after the release candidate image is verified.
> Meanwhile, we will have* latest* tag for each repository, which always
> points to the most recent verified release image, so users can pull it by
> default.
>
> Docker hub doesn't support leveled repository, which means we should
> follow *repository:tag* format.
> it's too general if we use {language_version} as repository for SDK
> images. (version is added when we support multiple versions.)
> So I would like to include *sdk* to repository. Images generated at local
> will also have the same name.
> Here are some examples:
>
> - python2.7_sdk:2.15.0
> - java_sdk:2.15.0_rc
> - go_sdk:latest
>
> I will proceed with this format if there is no strong opposition by
> tomorrow noon(PST).
>
> *To PMC members*:
> Permission control will follow the pypi model. All interested PMC members
> will be added as admins and release managers will be granted push
> permission.
> Please let me know your *docker id* if you want to be added as an admin.
>
> Thanks,
> Hannah
>
>
>
>
>
>
>
>
> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise <t...@apache.org> wrote:
>
>> This will greatly simplify trying out portable runners:
>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>
>> Can't wait for following to disappear from the instructions page: ./gradlew
>> :sdks:python:container:docker
>>
>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise <t...@apache.org> wrote:
>>
>>> Awesome, thank you!
>>>
>>>
>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <hannahji...@google.com>
>>> wrote:
>>>
>>>> Hi Thomas
>>>>
>>>> I created snapshot images from head as of around 2PM today.
>>>> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>>>
>>>> Thanks,
>>>> Hannah
>>>>
>>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <t...@apache.org> wrote:
>>>>
>>>>> Hi Hannah,
>>>>>
>>>>> Thank you, I know how to build the containers locally, but not how to
>>>>> publish them!
>>>>>
>>>>> The cwiki says "Publishing images to gcr.io/beam requires permissions
>>>>> in apache-beam-testing project."
>>>>>
>>>>> Can I get access to the testing project (at least temporarily) and
>>>>> what would I need to setup to run the publish target that is shown on
>>>>> cwiki?
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>>
>>>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <hannahji...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Thomas
>>>>>>
>>>>>> I haven't uploaded any snapshot images yet. Here is how you can
>>>>>> create one from head.
>>>>>> > cd [...]/beam/
>>>>>> # For Python
>>>>>> > ./gradlew :sdks:python:container:py{version}:docker *where version
>>>>>> is {2,35,36,37}*
>>>>>> # For Java
>>>>>> > ./gradlew -p sdks/java/container docker
>>>>>> # For Go
>>>>>> > ./gradlew -p sdks/go/container docker
>>>>>>
>>>>>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
>>>>>> from head.
>>>>>>
>>>>>> Please let me know if you have any questions.
>>>>>> Hannah
>>>>>>
>>>>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <t...@apache.org> wrote:
>>>>>>
>>>>>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>>>>>
>>>>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <t...@apache.org> wrote:
>>>>>>>
>>>>>>>> Thanks for working on this. Do you happen to have publicly
>>>>>>>> accessible snapshots published for your testing currently (even when
>>>>>>>> the
>>>>>>>> final location isn't sorted out)?
>>>>>>>>
>>>>>>>> I would like to use a 2.16 based Python SDK image for working on my
>>>>>>>> downstream project, but could not find anything in
>>>>>>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Thomas
>>>>>>>>
>>>>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>>>>>>> valen...@google.com> wrote:
>>>>>>>>
>>>>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>>>>>> hannahji...@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi team
>>>>>>>>>>
>>>>>>>>>> I am working on improving docker container support for Beam. We
>>>>>>>>>> would like to publish prebuilt containers for each release version
>>>>>>>>>> and
>>>>>>>>>> daily snapshot. Current work focuses on release images only and it
>>>>>>>>>> would be
>>>>>>>>>> part of the release process.
>>>>>>>>>>
>>>>>>>>>> The release images will be pushed to GCR which is publicly
>>>>>>>>>> accessible(pullable). We will use the following locations.
>>>>>>>>>> *Repository*: gcr.io/beam
>>>>>>>>>> *Project*: apache-beam-testing
>>>>>>>>>> More details, including naming and tagging scheme, can be found
>>>>>>>>>> at wiki
>>>>>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process>
>>>>>>>>>> which
>>>>>>>>>> is written by several contributors.
>>>>>>>>>>
>>>>>>>>>> I would like to discuss these two questions.
>>>>>>>>>> *1. How many tests do we need to run before pushing images to gcr*
>>>>>>>>>> ?
>>>>>>>>>> Publishing artifacts is the last step of the release process, so
>>>>>>>>>> at this moment, we already verified all codebase. In addition, many
>>>>>>>>>> Jenkins
>>>>>>>>>> tests use containers, so it is already verified several times. Do we
>>>>>>>>>> need
>>>>>>>>>> to run it again?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In a docker repository, one container image can have multiple
>>>>>>>>> tags. One possibility is that on the last step of the release
>>>>>>>>> process,
>>>>>>>>> after sufficient testing, we place a production tag on an image that
>>>>>>>>> was
>>>>>>>>> already pushed with a dev tag.
>>>>>>>>>
>>>>>>>>> For example a dev tag may look like:
>>>>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may
>>>>>>>>> look like:
>>>>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the
>>>>>>>>> same image at the end.
>>>>>>>>>
>>>>>>>>> We should also plan what the process of updating the container
>>>>>>>>> image will look like, if we need to release the image with
>>>>>>>>> additional changes, and how we will test these changes before the
>>>>>>>>> final
>>>>>>>>> push (or placing production tag).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *2. How many tests do we need to run to validate pushed images?*
>>>>>>>>>> When we push the images, we assume the images would work and pass
>>>>>>>>>> all the tests. After pushing, we should confirm the images are
>>>>>>>>>> pullable and
>>>>>>>>>> useable. I suggest we run several tests on dataflow with each pushed
>>>>>>>>>> image.
>>>>>>>>>> What do you think?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think it makes sense to do - Beam runners that use SDK
>>>>>>>>> container images should have some continuously running tests, which
>>>>>>>>> periodically check that all supported images are pullable and still
>>>>>>>>> compatible with the runner.
>>>>>>>>>
>>>>>>>>> This work can be refined later as we explore more during our
>>>>>>>>>> release process.
>>>>>>>>>> Please comment or edit the wiki page or reply to this email with
>>>>>>>>>> your opinions.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Hannah
>>>>>>>>>>
>>>>>>>>>
