Hi Thomas
I created snapshot images from head as of around 2PM today.
You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.
Thanks,
Hannah
On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <t...@apache.org> wrote:
> Hi Hannah,
>
> Thank you, I know how to build the containers locally, but not how to
> publish them!
>
> The cwiki says "Publishing images to gcr.io/beam requires permissions in
> apache-beam-testing project."
>
> Can I get access to the testing project (at least temporarily) and what
> would I need to setup to run the publish target that is shown on cwiki?
>
> Thanks,
> Thomas
>
>
> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <hannahji...@google.com>
> wrote:
>
>> Hi Thomas
>>
>> I haven't uploaded any snapshot images yet. Here is how you can create
>> one from head.
>> > cd [...]/beam/
>> # For Python
>> > ./gradlew :sdks:python:container:py{version}:docker *where version is
>> {2,35,36,37}*
>> # For Java
>> > ./gradlew -p sdks/java/container docker
>> # For Go
>> > ./gradlew -p sdks/go/container docker
>>
>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot from
>> head.
>>
>> Please let me know if you have any questions.
>> Hannah
>>
>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <t...@apache.org> wrote:
>>
>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>
>>> [1]
>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>
>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <t...@apache.org> wrote:
>>>
>>>> Thanks for working on this. Do you happen to have publicly accessible
>>>> snapshots published for your testing currently (even when the final
>>>> location isn't sorted out)?
>>>>
>>>> I would like to use a 2.16 based Python SDK image for working on my
>>>> downstream project, but could not find anything in
>>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>>> valen...@google.com> wrote:
>>>>
>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <hannahji...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi team
>>>>>>
>>>>>> I am working on improving docker container support for Beam. We would
>>>>>> like to publish prebuilt containers for each release version and daily
>>>>>> snapshot. Current work focuses on release images only and it would be
>>>>>> part
>>>>>> of the release process.
>>>>>>
>>>>>> The release images will be pushed to GCR which is publicly
>>>>>> accessible(pullable). We will use the following locations.
>>>>>> *Repository*: gcr.io/beam
>>>>>> *Project*: apache-beam-testing
>>>>>> More details, including naming and tagging scheme, can be found at
>>>>>> wiki
>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process>
>>>>>> which
>>>>>> is written by several contributors.
>>>>>>
>>>>>> I would like to discuss these two questions.
>>>>>> *1. How many tests do we need to run before pushing images to gcr*?
>>>>>> Publishing artifacts is the last step of the release process, so at
>>>>>> this moment, we already verified all codebase. In addition, many Jenkins
>>>>>> tests use containers, so it is already verified several times. Do we need
>>>>>> to run it again?
>>>>>>
>>>>>
>>>>> In a docker repository, one container image can have multiple tags.
>>>>> One possibility is that on the last step of the release process, after
>>>>> sufficient testing, we place a production tag on an image that was
>>>>> already
>>>>> pushed with a dev tag.
>>>>>
>>>>> For example a dev tag may look like:
>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look
>>>>> like:
>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same
>>>>> image at the end.
>>>>>
>>>>> We should also plan what the process of updating the container image
>>>>> will look like, if we need to release the image with additional changes,
>>>>> and how we will test these changes before the final push (or placing
>>>>> production tag).
>>>>>
>>>>>
>>>>>>
>>>>>> *2. How many tests do we need to run to validate pushed images?*
>>>>>> When we push the images, we assume the images would work and pass all
>>>>>> the tests. After pushing, we should confirm the images are pullable and
>>>>>> useable. I suggest we run several tests on dataflow with each pushed
>>>>>> image.
>>>>>> What do you think?
>>>>>>
>>>>>
>>>>> I think it makes sense to do - Beam runners that use SDK container
>>>>> images should have some continuously running tests, which periodically
>>>>> check that all supported images are pullable and still compatible with
>>>>> the
>>>>> runner.
>>>>>
>>>>> This work can be refined later as we explore more during our release
>>>>>> process.
>>>>>> Please comment or edit the wiki page or reply to this email with your
>>>>>> opinions.
>>>>>>
>>>>>> Thanks,
>>>>>> Hannah
>>>>>>
>>>>>
