Hi Thomas
I haven't uploaded any snapshot images yet. Here is how you can create one
from head.
> cd [...]/beam/
# For Python
> ./gradlew :sdks:python:container:py{version}:docker *where version is
{2,35,36,37}*
# For Java
> ./gradlew -p sdks/java/container docker
# For Go
> ./gradlew -p sdks/go/container docker
The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot from
head.
Please let me know if you have any questions.
Hannah
On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <[email protected]> wrote:
> I actually found something in [1], but it is 2.15 unfortunately.
>
> [1]
> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>
> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <[email protected]> wrote:
>
>> Thanks for working on this. Do you happen to have publicly accessible
>> snapshots published for your testing currently (even when the final
>> location isn't sorted out)?
>>
>> I would like to use a 2.16 based Python SDK image for working on my
>> downstream project, but could not find anything in
>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>
>> Thanks,
>> Thomas
>>
>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <[email protected]>
>> wrote:
>>
>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <[email protected]>
>>> wrote:
>>>
>>>> Hi team
>>>>
>>>> I am working on improving docker container support for Beam. We would
>>>> like to publish prebuilt containers for each release version and daily
>>>> snapshot. Current work focuses on release images only and it would be part
>>>> of the release process.
>>>>
>>>> The release images will be pushed to GCR which is publicly
>>>> accessible(pullable). We will use the following locations.
>>>> *Repository*: gcr.io/beam
>>>> *Project*: apache-beam-testing
>>>> More details, including naming and tagging scheme, can be found at wiki
>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process>
>>>> which
>>>> is written by several contributors.
>>>>
>>>> I would like to discuss these two questions.
>>>> *1. How many tests do we need to run before pushing images to gcr*?
>>>> Publishing artifacts is the last step of the release process, so at
>>>> this moment, we already verified all codebase. In addition, many Jenkins
>>>> tests use containers, so it is already verified several times. Do we need
>>>> to run it again?
>>>>
>>>
>>> In a docker repository, one container image can have multiple tags. One
>>> possibility is that on the last step of the release process, after
>>> sufficient testing, we place a production tag on an image that was already
>>> pushed with a dev tag.
>>>
>>> For example a dev tag may look like:
>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look
>>> like:
>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same
>>> image at the end.
>>>
>>> We should also plan what the process of updating the container image
>>> will look like, if we need to release the image with additional changes,
>>> and how we will test these changes before the final push (or placing
>>> production tag).
>>>
>>>
>>>>
>>>> *2. How many tests do we need to run to validate pushed images?*
>>>> When we push the images, we assume the images would work and pass all
>>>> the tests. After pushing, we should confirm the images are pullable and
>>>> useable. I suggest we run several tests on dataflow with each pushed image.
>>>> What do you think?
>>>>
>>>
>>> I think it makes sense to do - Beam runners that use SDK container
>>> images should have some continuously running tests, which periodically
>>> check that all supported images are pullable and still compatible with the
>>> runner.
>>>
>>> This work can be refined later as we explore more during our release
>>>> process.
>>>> Please comment or edit the wiki page or reply to this email with your
>>>> opinions.
>>>>
>>>> Thanks,
>>>> Hannah
>>>>
>>>
