[Reminder] Beam 2.16.0 Release Soon

[Mark Liu]

Hi all,

Reminder, the plan to cut the release branch is on 09/11 (less than a week
from now)[1]. Please mark all release blocking issues with fix version
2.16[2].

[1]
https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
[2]
https://issues.apache.org/jira/browse/BEAM-8157?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.16.0

Re: Improve container support

[Ankur Goenka]

Please ignore the previous email. I was looking at the older document in
the mail thread.

On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka  wrote:

> I think sdk in the name is obsolete as they are all under sdks name space.
>
> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang 
> wrote:
>
>> Hi Team
>>
>> Thanks for all the comments about beam containers.
>> After considering various opinions and investigating gcr and docker hub,
>> we decided to push images to docker hub.
>>
>> Each image will have two tags, {version}_rc and {version}. {version} tag
>> will be added after the release candidate image is verified.
>> Meanwhile, we will have* latest* tag for each repository, which always
>> points to the most recent verified release image, so users can pull it by
>> default.
>>
>> Docker hub doesn't support leveled repository, which means we should
>> follow *repository:tag* format.
>> it's too general if we use {language_version} as repository for SDK
>> images. (version is added when we support multiple versions.)
>> So I would like to include *sdk* to repository. Images generated at
>> local will also have the same name.
>> Here are some examples:
>>
>>- python2.7_sdk:2.15.0
>>- java_sdk:2.15.0_rc
>>- go_sdk:latest
>>
>> I will proceed with this format if there is no strong opposition by
>> tomorrow noon(PST).
>>
>> *To PMC members*:
>> Permission control will follow the pypi model. All interested PMC members
>> will be added as admins and release managers will be granted push
>> permission.
>> Please let me know your *docker id* if you want to be added as an admin.
>>
>> Thanks,
>> Hannah
>>
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:
>>
>>> This will greatly simplify trying out portable runners:
>>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>>
>>> Can't wait for following to disappear from the instructions page: ./gradlew
>>> :sdks:python:container:docker
>>>
>>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:
>>>
 Awesome, thank you!


 On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang 
 wrote:

> Hi Thomas
>
> I created snapshot images from head as of around 2PM today.
> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot
> .
>
> Thanks,
> Hannah
>
> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:
>
>> Hi Hannah,
>>
>> Thank you, I know how to build the containers locally, but not how to
>> publish them!
>>
>> The cwiki says "Publishing images to gcr.io/beam requires
>> permissions in apache-beam-testing project."
>>
>> Can I get access to the testing project (at least temporarily) and
>> what would I need to setup to run the publish target that is shown on 
>> cwiki?
>>
>> Thanks,
>> Thomas
>>
>>
>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang 
>> wrote:
>>
>>> Hi Thomas
>>>
>>> I haven't uploaded any snapshot images yet. Here is how you can
>>> create one from head.
>>> > cd [...]/beam/
>>> # For Python
>>> > ./gradlew :sdks:python:container:py{version}:docker *where
>>> version is {2,35,36,37}*
>>> # For Java
>>> > ./gradlew -p sdks/java/container docker
>>> # For Go
>>> > ./gradlew -p sdks/go/container docker
>>>
>>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
>>> from head.
>>>
>>> Please let me know if you have any questions.
>>> Hannah
>>>
>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise  wrote:
>>>
 I actually found something in [1], but it is 2.15 unfortunately.

 [1]
 https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30

 On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise 
 wrote:

> Thanks for working on this. Do you happen to have publicly
> accessible snapshots published for your testing currently (even when 
> the
> final location isn't sorted out)?
>
> I would like to use a 2.16 based Python SDK image for working on
> my downstream project, but could not find anything in
> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>
> Thanks,
> Thomas
>
> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>> hannahji...@google.com> wrote:
>>
>>> Hi team
>>>
>>> I am working on improving docker container support for Beam. We
>>> would like to publish prebuilt containers for each release version 
>>> and
>>> daily snapshot. Current work focuses on release images only and it 
>>> would be
>>> part of the release process.

Re: Possible Python SDK performance regression

[Ahmet Altay]

On Thu, Sep 5, 2019 at 4:15 PM Thomas Weise  wrote:

> The workload is quite different. What I have is streaming with state and
> timers.
>
>
>
> On Thu, Sep 5, 2019 at 3:47 PM Pablo Estrada  wrote:
>
>> We only recently started running Chicago Taxi Example. +Michał Walenia
>>  I don't see it in the dashboards. Do you
>> know if it's possible to see any trends in the data?
>>
>> We have a few tests running now:
>> - Combine tests:
>> https://apache-beam-testing.appspot.com/explore?dashboard=5763764733345792=201943890=1334074373
>> - GBK tests:
>> https://apache-beam-testing.appspot.com/explore?dashboard=5763764733345792=201943890=1334074373
>>
>> They don't seem to show a very drastic jump either, but they aren't very
>> old.
>>
>> There is also work ongoing to add alerting for this sort of regressions
>> by Kasia and Kamil (added). The work is not there yet (it's in progress).
>> Best
>> -P.
>>
>> On Thu, Sep 5, 2019 at 3:35 PM Thomas Weise  wrote:
>>
>>> It probably won't be practical to do a bisect due to the high cost of
>>> each iteration with our fork/deploy setup.
>>>
>>> Perhaps it is time to setup something with the synthetic source that
>>> works just with Beam as dependency.
>>>
>>
I agree with this.

Pablo, Kasia, Kamil, does the new benchmarks give us a easy to use
framework for using synthetic source in benchmarks?


>
>>> On Thu, Sep 5, 2019 at 3:23 PM Ahmet Altay  wrote:
>>>
 There are a few in this dashboard [1], but not very useful in this case
 because they do not go back more than a month and not very comprehensive. I
 do not see a jump there. Thomas, would it be possible to bisect to find
 what commit caused the regression?

 +Pablo Estrada  do we have any python on flink
 benchmarks for chicago example?
 +Alan Myrvold  +Yifan Zou  It
 would be good to have alerts on benchmarks. Do we have such an ability
 today?

 [1] https://apache-beam-testing.appspot.com/dashboard-admin

 On Thu, Sep 5, 2019 at 3:15 PM Thomas Weise  wrote:

> Hi,
>
> Are there any performance tests run for the Python SDK as part of
> release verification (or otherwise as well)?
>
> I see what appears to be a regression in master (compared to 2.14)
> with our in-house application (~ 25% jump in cpu utilization and
> corresponds drop in throughput).
>
> I wanted to see if there is anything available to verify that within
> Beam.
>
> Thanks,
> Thomas
>
>

Re: Improve container support

[Ankur Goenka]

I think sdk in the name is obsolete as they are all under sdks name space.

On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang  wrote:

> Hi Team
>
> Thanks for all the comments about beam containers.
> After considering various opinions and investigating gcr and docker hub,
> we decided to push images to docker hub.
>
> Each image will have two tags, {version}_rc and {version}. {version} tag
> will be added after the release candidate image is verified.
> Meanwhile, we will have* latest* tag for each repository, which always
> points to the most recent verified release image, so users can pull it by
> default.
>
> Docker hub doesn't support leveled repository, which means we should
> follow *repository:tag* format.
> it's too general if we use {language_version} as repository for SDK
> images. (version is added when we support multiple versions.)
> So I would like to include *sdk* to repository. Images generated at local
> will also have the same name.
> Here are some examples:
>
>- python2.7_sdk:2.15.0
>- java_sdk:2.15.0_rc
>- go_sdk:latest
>
> I will proceed with this format if there is no strong opposition by
> tomorrow noon(PST).
>
> *To PMC members*:
> Permission control will follow the pypi model. All interested PMC members
> will be added as admins and release managers will be granted push
> permission.
> Please let me know your *docker id* if you want to be added as an admin.
>
> Thanks,
> Hannah
>
>
>
>
>
>
>
>
> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:
>
>> This will greatly simplify trying out portable runners:
>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>
>> Can't wait for following to disappear from the instructions page: ./gradlew
>> :sdks:python:container:docker
>>
>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:
>>
>>> Awesome, thank you!
>>>
>>>
>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang 
>>> wrote:
>>>
 Hi Thomas

 I created snapshot images from head as of around 2PM today.
 You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.

 Thanks,
 Hannah

 On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:

> Hi Hannah,
>
> Thank you, I know how to build the containers locally, but not how to
> publish them!
>
> The cwiki says "Publishing images to gcr.io/beam requires permissions
> in apache-beam-testing project."
>
> Can I get access to the testing project (at least temporarily) and
> what would I need to setup to run the publish target that is shown on 
> cwiki?
>
> Thanks,
> Thomas
>
>
> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang 
> wrote:
>
>> Hi Thomas
>>
>> I haven't uploaded any snapshot images yet. Here is how you can
>> create one from head.
>> > cd [...]/beam/
>> # For Python
>> > ./gradlew :sdks:python:container:py{version}:docker *where version
>> is {2,35,36,37}*
>> # For Java
>> > ./gradlew -p sdks/java/container docker
>> # For Go
>> > ./gradlew -p sdks/go/container docker
>>
>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
>> from head.
>>
>> Please let me know if you have any questions.
>> Hannah
>>
>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise  wrote:
>>
>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>
>>> [1]
>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>
>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise  wrote:
>>>
 Thanks for working on this. Do you happen to have publicly
 accessible snapshots published for your testing currently (even when 
 the
 final location isn't sorted out)?

 I would like to use a 2.16 based Python SDK image for working on my
 downstream project, but could not find anything in
 gcr.io/apache-beam-testing/beam/sdks/rc/snapshot

 Thanks,
 Thomas

 On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
> hannahji...@google.com> wrote:
>
>> Hi team
>>
>> I am working on improving docker container support for Beam. We
>> would like to publish prebuilt containers for each release version 
>> and
>> daily snapshot. Current work focuses on release images only and it 
>> would be
>> part of the release process.
>>
>> The release images will be pushed to GCR which is publicly
>> accessible(pullable). We will use the following locations.
>> *Repository*: gcr.io/beam
>> *Project*: apache-beam-testing
>> More details, including naming and tagging scheme, can be found
>> at 

Re: Possible Python SDK performance regression

[Thomas Weise]

The workload is quite different. What I have is streaming with state and
timers.



On Thu, Sep 5, 2019 at 3:47 PM Pablo Estrada  wrote:

> We only recently started running Chicago Taxi Example. +Michał Walenia
>  I don't see it in the dashboards. Do you
> know if it's possible to see any trends in the data?
>
> We have a few tests running now:
> - Combine tests:
> https://apache-beam-testing.appspot.com/explore?dashboard=5763764733345792=201943890=1334074373
> - GBK tests:
> https://apache-beam-testing.appspot.com/explore?dashboard=5763764733345792=201943890=1334074373
>
> They don't seem to show a very drastic jump either, but they aren't very
> old.
>
> There is also work ongoing to add alerting for this sort of regressions by
> Kasia and Kamil (added). The work is not there yet (it's in progress).
> Best
> -P.
>
> On Thu, Sep 5, 2019 at 3:35 PM Thomas Weise  wrote:
>
>> It probably won't be practical to do a bisect due to the high cost of
>> each iteration with our fork/deploy setup.
>>
>> Perhaps it is time to setup something with the synthetic source that
>> works just with Beam as dependency.
>>
>> On Thu, Sep 5, 2019 at 3:23 PM Ahmet Altay  wrote:
>>
>>> There are a few in this dashboard [1], but not very useful in this case
>>> because they do not go back more than a month and not very comprehensive. I
>>> do not see a jump there. Thomas, would it be possible to bisect to find
>>> what commit caused the regression?
>>>
>>> +Pablo Estrada  do we have any python on flink
>>> benchmarks for chicago example?
>>> +Alan Myrvold  +Yifan Zou  It
>>> would be good to have alerts on benchmarks. Do we have such an ability
>>> today?
>>>
>>> [1] https://apache-beam-testing.appspot.com/dashboard-admin
>>>
>>> On Thu, Sep 5, 2019 at 3:15 PM Thomas Weise  wrote:
>>>
 Hi,

 Are there any performance tests run for the Python SDK as part of
 release verification (or otherwise as well)?

 I see what appears to be a regression in master (compared to 2.14) with
 our in-house application (~ 25% jump in cpu utilization and corresponds
 drop in throughput).

 I wanted to see if there is anything available to verify that within
 Beam.

 Thanks,
 Thomas

Re: Possible Python SDK performance regression

[Pablo Estrada]

We only recently started running Chicago Taxi Example. +Michał Walenia
 I don't see it in the dashboards. Do you know
if it's possible to see any trends in the data?

We have a few tests running now:
- Combine tests:
https://apache-beam-testing.appspot.com/explore?dashboard=5763764733345792=201943890=1334074373
- GBK tests:
https://apache-beam-testing.appspot.com/explore?dashboard=5763764733345792=201943890=1334074373

They don't seem to show a very drastic jump either, but they aren't very
old.

There is also work ongoing to add alerting for this sort of regressions by
Kasia and Kamil (added). The work is not there yet (it's in progress).
Best
-P.

On Thu, Sep 5, 2019 at 3:35 PM Thomas Weise  wrote:

> It probably won't be practical to do a bisect due to the high cost of each
> iteration with our fork/deploy setup.
>
> Perhaps it is time to setup something with the synthetic source that works
> just with Beam as dependency.
>
> On Thu, Sep 5, 2019 at 3:23 PM Ahmet Altay  wrote:
>
>> There are a few in this dashboard [1], but not very useful in this case
>> because they do not go back more than a month and not very comprehensive. I
>> do not see a jump there. Thomas, would it be possible to bisect to find
>> what commit caused the regression?
>>
>> +Pablo Estrada  do we have any python on flink
>> benchmarks for chicago example?
>> +Alan Myrvold  +Yifan Zou  It
>> would be good to have alerts on benchmarks. Do we have such an ability
>> today?
>>
>> [1] https://apache-beam-testing.appspot.com/dashboard-admin
>>
>> On Thu, Sep 5, 2019 at 3:15 PM Thomas Weise  wrote:
>>
>>> Hi,
>>>
>>> Are there any performance tests run for the Python SDK as part of
>>> release verification (or otherwise as well)?
>>>
>>> I see what appears to be a regression in master (compared to 2.14) with
>>> our in-house application (~ 25% jump in cpu utilization and corresponds
>>> drop in throughput).
>>>
>>> I wanted to see if there is anything available to verify that within
>>> Beam.
>>>
>>> Thanks,
>>> Thomas
>>>
>>>

Re: Possible Python SDK performance regression

[Thomas Weise]

It probably won't be practical to do a bisect due to the high cost of each
iteration with our fork/deploy setup.

Perhaps it is time to setup something with the synthetic source that works
just with Beam as dependency.

On Thu, Sep 5, 2019 at 3:23 PM Ahmet Altay  wrote:

> There are a few in this dashboard [1], but not very useful in this case
> because they do not go back more than a month and not very comprehensive. I
> do not see a jump there. Thomas, would it be possible to bisect to find
> what commit caused the regression?
>
> +Pablo Estrada  do we have any python on flink
> benchmarks for chicago example?
> +Alan Myrvold  +Yifan Zou  It
> would be good to have alerts on benchmarks. Do we have such an ability
> today?
>
> [1] https://apache-beam-testing.appspot.com/dashboard-admin
>
> On Thu, Sep 5, 2019 at 3:15 PM Thomas Weise  wrote:
>
>> Hi,
>>
>> Are there any performance tests run for the Python SDK as part of release
>> verification (or otherwise as well)?
>>
>> I see what appears to be a regression in master (compared to 2.14) with
>> our in-house application (~ 25% jump in cpu utilization and corresponds
>> drop in throughput).
>>
>> I wanted to see if there is anything available to verify that within Beam.
>>
>> Thanks,
>> Thomas
>>
>>

Re: Improve container support

[Hannah Jiang]

Hi Team

Thanks for all the comments about beam containers.
After considering various opinions and investigating gcr and docker hub, we
decided to push images to docker hub.

Each image will have two tags, {version}_rc and {version}. {version} tag
will be added after the release candidate image is verified.
Meanwhile, we will have* latest* tag for each repository, which always
points to the most recent verified release image, so users can pull it by
default.

Docker hub doesn't support leveled repository, which means we should follow
*repository:tag* format.
it's too general if we use {language_version} as repository for SDK images.
(version is added when we support multiple versions.)
So I would like to include *sdk* to repository. Images generated at local
will also have the same name.
Here are some examples:

   - python2.7_sdk:2.15.0
   - java_sdk:2.15.0_rc
   - go_sdk:latest

I will proceed with this format if there is no strong opposition by
tomorrow noon(PST).

*To PMC members*:
Permission control will follow the pypi model. All interested PMC members
will be added as admins and release managers will be granted push
permission.
Please let me know your *docker id* if you want to be added as an admin.

Thanks,
Hannah








On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:

> This will greatly simplify trying out portable runners:
> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>
> Can't wait for following to disappear from the instructions page: ./gradlew
> :sdks:python:container:docker
>
> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:
>
>> Awesome, thank you!
>>
>>
>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang 
>> wrote:
>>
>>> Hi Thomas
>>>
>>> I created snapshot images from head as of around 2PM today.
>>> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>>
>>> Thanks,
>>> Hannah
>>>
>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:
>>>
 Hi Hannah,

 Thank you, I know how to build the containers locally, but not how to
 publish them!

 The cwiki says "Publishing images to gcr.io/beam requires permissions
 in apache-beam-testing project."

 Can I get access to the testing project (at least temporarily) and what
 would I need to setup to run the publish target that is shown on cwiki?

 Thanks,
 Thomas


 On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang 
 wrote:

> Hi Thomas
>
> I haven't uploaded any snapshot images yet. Here is how you can create
> one from head.
> > cd [...]/beam/
> # For Python
> > ./gradlew :sdks:python:container:py{version}:docker *where version
> is {2,35,36,37}*
> # For Java
> > ./gradlew -p sdks/java/container docker
> # For Go
> > ./gradlew -p sdks/go/container docker
>
> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
> from head.
>
> Please let me know if you have any questions.
> Hannah
>
> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise  wrote:
>
>> I actually found something in [1], but it is 2.15 unfortunately.
>>
>> [1]
>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>
>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise  wrote:
>>
>>> Thanks for working on this. Do you happen to have publicly
>>> accessible snapshots published for your testing currently (even when the
>>> final location isn't sorted out)?
>>>
>>> I would like to use a 2.16 based Python SDK image for working on my
>>> downstream project, but could not find anything in
>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>
>>> Thanks,
>>> Thomas
>>>
>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
 hannahji...@google.com> wrote:

> Hi team
>
> I am working on improving docker container support for Beam. We
> would like to publish prebuilt containers for each release version and
> daily snapshot. Current work focuses on release images only and it 
> would be
> part of the release process.
>
> The release images will be pushed to GCR which is publicly
> accessible(pullable). We will use the following locations.
> *Repository*: gcr.io/beam
> *Project*: apache-beam-testing
> More details, including naming and tagging scheme, can be found at
> wiki
> 
>  which
> is written by several contributors.
>
> I would like to discuss these two questions.
> *1. How many tests do we need to run before pushing images to gcr*
> 

Re: Possible Python SDK performance regression

[Ahmet Altay]

There are a few in this dashboard [1], but not very useful in this case
because they do not go back more than a month and not very comprehensive. I
do not see a jump there. Thomas, would it be possible to bisect to find
what commit caused the regression?

+Pablo Estrada  do we have any python on flink
benchmarks for chicago example?
+Alan Myrvold  +Yifan Zou  It
would be good to have alerts on benchmarks. Do we have such an ability
today?

[1] https://apache-beam-testing.appspot.com/dashboard-admin

On Thu, Sep 5, 2019 at 3:15 PM Thomas Weise  wrote:

> Hi,
>
> Are there any performance tests run for the Python SDK as part of release
> verification (or otherwise as well)?
>
> I see what appears to be a regression in master (compared to 2.14) with
> our in-house application (~ 25% jump in cpu utilization and corresponds
> drop in throughput).
>
> I wanted to see if there is anything available to verify that within Beam.
>
> Thanks,
> Thomas
>
>

Possible Python SDK performance regression

[Thomas Weise]

Hi,

Are there any performance tests run for the Python SDK as part of release
verification (or otherwise as well)?

I see what appears to be a regression in master (compared to 2.14) with our
in-house application (~ 25% jump in cpu utilization and corresponds drop in
throughput).

I wanted to see if there is anything available to verify that within Beam.

Thanks,
Thomas

Re: Hackathon @BeamSummit @ApacheCon

[Chad Dombrova]

Has a date and time been picked for this?  I'll be there for part of the
week and would love to join.

On Tue, Sep 3, 2019 at 11:31 AM Brian Hulette  wrote:

> I will be around all week as well and would love to help with a Beam
> hackathon in any way :)
>
> On Thu, Aug 29, 2019 at 9:46 AM Maximilian Michels  wrote:
>
>> Hey,
>>
>> I'm in as well! Austin and I recently talked about how we could organize
>> the hackathon. Likely it will be an hour per day for exchanging ideas
>> and learning about Beam. For example, there has been interest from the
>> Apache Streams project to discuss points for collaboration.
>>
>> We will soon announce the exact hours.
>>
>> Cheers,
>> Max
>>
>> On 23.08.19 05:06, Kenneth Knowles wrote:
>> > I will be at Beam Summit / ApacheCon NA and would love to drop by a
>> > hackathon room if one is arranged. Really excited for both my first
>> > ApacheCon and Beam Summit (finally!)
>> >
>> > Kenn
>> >
>> > On Thu, Aug 22, 2019 at 10:18 AM Austin Bennett
>> > mailto:whatwouldausti...@gmail.com>>
>> wrote:
>> >
>> > And, for clarity, especially focused on Hackathon times on Monday
>> > and/or Tuesday of ApacheCon, to not conflict with BeamSummit
>> sessions.
>> >
>> > On Thu, Aug 22, 2019 at 9:47 AM Austin Bennett
>> > mailto:whatwouldausti...@gmail.com>>
>> > wrote:
>> >
>> > Less than 3 weeks till Beam Summit @ApacheCon!
>> >
>> > We are to be in Vegas for BeamSummit and ApacheCon in a few
>> weeks.
>> >
>> > Likely to reserve space in the Hackathon Room to accomplish some
>> > tasks:
>> > * Help Users
>> > * Build Beam
>> > * Collaborate with other projects
>> > * etc
>> >
>> > If you're to be around (or not) let us know how you'd like to be
>> > involved.  Also, please share and surface anything that would be
>> > good for us to look at (and, esp. any beginner tasks, in case we
>> > can entice some new contributors).
>> >
>> >
>> > P.S.  See BeamSummit.org, if you're thinking of attending -
>> > there's a discount code.
>> >
>>
>

installing Apache Beam on Pycharm with Python 3.7

[Priti Badami]

Hi Dev Team,

I am trying to install Apache Beam. I have pip 19.2.3 but I am facing
issues while installing Beam.

please advice,

Thanks,
Priti Badami

Re: [VOTE] Vendored Dependencies Release

[Lukasz Cwik]

LGTM

On Wed, Sep 4, 2019 at 4:24 PM Rui Wang  wrote:

> Thanks Pablo for jumping in for help.
>
> Now the sources are moved to [1]. Please let me know if it is ok.
>
> [1]:
> https://dist.apache.org/repos/dist/release/beam/vendor/calcite/1_20_0/
>
> -Rui
>
> On Wed, Sep 4, 2019 at 4:15 PM Pablo Estrada  wrote:
>
>> I can help.
>>
>> On Wed, Sep 4, 2019 at 1:09 PM Rui Wang  wrote:
>>
>>> There is a step of releasing requires PMC permission:
>>>
>>> """
>>>
>>> Copy the source release from the dev repository to the release
>>> repository at dist.apache.org using Subversion.
>>> Move last release artifacts from dist.apache.org to archive.apache.org
>>> using Subversion. """
>>>
>>> Is there a PMC member could help on this operation to move [1] to
>>> "release" repo?
>>>
>>> [1]: [1]
>>> https://dist.apache.org/repos/dist/dev/beam/vendor/calcite/1_20_0
>>>
>>> -Rui
>>>
>>> On Wed, Sep 4, 2019 at 10:16 AM Rui Wang  wrote:
>>>
 I'm happy to announce that we have unanimously approved this release.

 There are 5 approving votes, 3 of which are binding:

 * Lukasz Cwik

 * Kenneth Knowles

 * Ahmet Altay

 There are no disapproving votes.

 Thanks everyone!

 On Tue, Sep 3, 2019 at 1:29 PM Lukasz Cwik  wrote:

> +1
>
> On Tue, Sep 3, 2019 at 1:22 PM Kenneth Knowles 
> wrote:
>
>> +1
>>
>> On Tue, Sep 3, 2019 at 11:00 AM Ahmet Altay  wrote:
>>
>>> +1
>>>
>>> On Tue, Sep 3, 2019 at 10:52 AM Andrew Pilloud 
>>> wrote:
>>>
 +1

 Inspected the jar it looked reasonable.

 Andrew

 On Tue, Sep 3, 2019 at 9:06 AM Rui Wang  wrote:

> Friendly ping.
>
>
> -Rui
>
> On Thu, Aug 29, 2019 at 9:50 AM Rui Wang 
> wrote:
>
>> Thanks Kai and Andrew. Now prgapachebeam-1083 is publicly exposed.
>>
>> I also found a useful link[1] to explain staging repos in Apache
>> Nexus
>>
>>
>> [1]:
>> https://help.sonatype.com/repomanager2/staging-releases/managing-staging-repositories#ManagingStagingRepositories-ClosinganOpenRepository
>>
>> -Rui
>>
>> On Wed, Aug 28, 2019 at 9:19 PM Andrew Pilloud <
>> apill...@google.com> wrote:
>>
>>> You need to close the release for it to be published to the
>>> staging server. I can help if you still have questions.
>>>
>>> Andrew
>>>
>>> On Wed, Aug 28, 2019, 8:48 PM Rui Wang 
>>> wrote:
>>>
 I can see prgapachebeam-1083 is in open status in staging
 repository. I am not sure why it is not public exposed. I probably 
 need
 some guidance on it.


 -Rui

 On Wed, Aug 28, 2019 at 3:50 PM Kai Jiang 
 wrote:

> Hi Rui,
>
> For accessing artifacts [1] in Maven Central Repository, is
> this intent to be not public exposed?
>
> Best,
> Kai
>
> [1]
> https://repository.apache.org/content/repositories/orgapachebeam-1083/
>
> On Wed, Aug 28, 2019 at 11:57 AM Kai Jiang 
> wrote:
>
>> +1 (non-binding)Thanks Rui!
>>
>> On Tue, Aug 27, 2019 at 10:46 PM Rui Wang 
>> wrote:
>>
>>> Please review the release of the following artifacts that we
>>> vendor:
>>>
>>>  * beam-vendor-calcite-1_20_0
>>>
>>> Hi everyone,
>>>
>>> Please review and vote on the release candidate #1 for the
>>> org.apache.beam:beam-vendor-calcite-1_20_0:0.1, as follows:
>>>
>>> [ ] +1, Approve the release
>>>
>>> [ ] -1, Do not approve the release (please provide specific
>>> comments)
>>>
>>>
>>> The complete staging area is available for your review,
>>> which includes:
>>>
>>> * the official Apache source release to be deployed to
>>> dist.apache.org [1], which is signed with the key with
>>> fingerprint 0D7BE1A252DBCEE89F6491BBDFA64862B703F5C8 [2],
>>>
>>> * all artifacts to be deployed to the Maven Central
>>> Repository [3],
>>>
>>> * commit hash "664e25019fc1977e7041e4b834e8d9628b912473" [4],
>>>
>>> The vote will be open for at least 72 hours. It is adopted
>>> by majority approval, with at least 3 PMC affirmative votes.
>>>
>>> Thanks,
>>>
>>> Rui
>>>

Re: [PROPOSAL] Storing, displaying and detecting anomalies in test results

[Kamil Wasilewski]

We've just started working on this and here comes the first PR:
https://github.com/apache/beam/pull/9482.

Feel free to share your thoughts and ideas.

Kamil

On Tue, Aug 27, 2019 at 1:19 AM Pablo Estrada  wrote:

> Thanks Kamil for bringing this!
> +Manisha Bhardwaj  +Mark Liu  have
> worked on internal benchmarking at Google - would you take a look please?
>
> On Fri, Aug 23, 2019 at 3:22 AM Kamil Wasilewski <
> kamil.wasilew...@polidea.com> wrote:
>
>> Hi all,
>>
>> Recently we did some research on how to visualize IO performance tests,
>> Nexmark and Load test results better and how to detect regressions
>> automatically in an easy way using tools dedicated for the job.
>>
>> We'd like to share a proposal with you:
>> 
>> https://s.apache.org/test-metrics-storage
>>
>>
>>
>> Any comments are highly appreciated.
>>
>> Thanks,
>>
>> Kamil
>>
>

Re: Jira - Contributor

[Jean-Baptiste Onofré]

Done:

you are in the contributor list and the Jira is assigned to you.

Regards
JB

On 05/09/2019 10:31, Matthew Darwin wrote:
> Hi,
> 
> Could I please be added as a contributor on Jira so I can assign
> https://issues.apache.org/jira/browse/BEAM-8153 to me?
> 
> Kind regards
> 
> Matthew

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Jira - Contributor

[Matthew Darwin]

Hi,

Could I please be added as a contributor on Jira so I can assign 
https://issues.apache.org/jira/browse/BEAM-8153 to me?

Kind regards

Matthew

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.