Re: Companies using Beam?

[Jean-Baptiste Onofre]

Hi,

We already have some testimonials on Beam home page (I did the one about Beam 
use at Talend).

It makes sense to have a dedicated section as it gives ideas about use case and 
production system running with Beam.

Regards
JB

> Le 28 avr. 2020 à 23:42, Austin Bennett  a écrit 
> :
> 
> Hi All,
> 
> Have we considered getting onto our website or our our GitHub repo the 
> ability for individuals to share that their company is using Beam?  Seeing - 
> what I believe to be a reasonable list of - companies productively using Beam 
> would be helpful to point others to.  For instance, a common question I get 
> is whether anyone or who is using?  I'm not sure that's the best metric or 
> datapoint in many cases for adoption, but a heuristic that some rely upon.  
> 
> Naturally, we could ask for a roll-call, esp. via user list, but imagining  a 
> persistent web-list would be of interest.   
> 
> Cheers,
> Austin
> 
> 
> P.S.  If putting such a list into our repo, that would also get some people 
> to submit PRs (so more contributors!) :-)
> 
> 

Re: sdks:java:container:generateThirdPartyLicenses effect on build time / stability

[Robert Bradshaw]

On Tue, Apr 28, 2020 at 4:46 PM Hannah Jiang  wrote:

> I guess I assumed there was some reason we needed "lightweight images" in
>> our tests (because licenses take up a lot of space IIRC), but maybe not.
>> Can you elaborate on the purpose of this option Hannah?
>
> Reducing image size is a good reason to have the pull option. There were
> user requests to create lightweight images. Now I think users can use skip
> option to create lightweight iamges. pull option is not needed.
>
> Then the discussion becomes which one should be the default mode?
> According to feedback, skip should be the default mode, and change it to
> add mode when running Jenkins test. And the docker-pull-licenses tag is
> binary again.
> It seems like there is an easy way to pass the tag to all Jenkins test,
> which is adding *context.switches("-Pdocker-pull-licenses"**)* to
> CommonJobProperties.groovy
> .
> I'm not very familiar with Jenkins, does this would work as expected?
> If it sounds like a bad idea to pass this tag to ALL tests, I would ask
> help from community to identify those tests which create docker images
> (Java and Python for now) and pass the tag to the tests.
>

That could work.

The other question is, how easy would it be for a release manager to
accidentally push non-compliant images?

We have definitely not explored how cheap we can make doing the "real
thing" can be, which would IMHO be best.


>
>
> On Tue, Apr 28, 2020 at 3:50 PM Kyle Weaver  wrote:
>
>> > To overwrite, we need to pass the tag for ALL Jenkins tests that create
>> docker images, which is quite a bunch of work.
>>
>> If it comes to that, I'd rather we do the work of passing tags instead of
>> users.
>>
>> > Does anyone know if there is an easy way to pass the tag to many tests?
>> or overwrite the default mode, which will be defined at gradle.properties,
>> to pull ONLY with Jenkins tests?
>>
>> There is precedence for the latter option:
>> https://github.com/apache/beam/blob/1905dbde5ca0858fce89f65c761e88511840a384/build.gradle#L45
>>
>> > (On that note, however, pull seems bad for both.)
>>
>> I guess I assumed there was some reason we needed "lightweight images" in
>> our tests (because licenses take up a lot of space IIRC), but maybe not.
>> Can you elaborate on the purpose of this option Hannah?
>>
>> On Tue, Apr 28, 2020 at 6:40 PM Robert Bradshaw 
>> wrote:
>>
>>> Fundamentally having license checking off by default is dangerous for
>>> releases, but having it on by default is annoying for developers. (On that
>>> note, however, pull seems bad for both.) Is it possible to make it a gradle
>>> target that only runs when something (specifically dependencies, or the
>>> files declaring them) have changed, and would leverage the gradle cache
>>> (and hence be cheap) otherwise?
>>>
>>> Alternatively, I think we should invest in URL caching. These urls
>>> should be immutable; let's only download them once, ever.
>>>
>>> On Tue, Apr 28, 2020 at 3:31 PM Hannah Jiang 
>>> wrote:
>>>
 Do you mean set the default mode to skip and overwrite it to pull mode
 with Jenkins test? To overwrite, we need to pass the tag for ALL Jenkins
 tests that create docker images, which is quite a bunch of work.
 Does anyone know if there is an easy way to pass the tag to many tests?
 or overwrite the default mode, which will be defined at gradle.properties,
 to pull ONLY with Jenkins tests? We can add a step to do this after cloning
 a git branch to Jenkins machine. But it seems easier to change the local
 gradle.properties for local development purpose..?



 On Tue, Apr 28, 2020 at 3:09 PM Thomas Weise  wrote:

> Can this be solved by enabling the license magic for the Jenkins jobs?
>
> I also think the default should be off for better local development
> experience.
>
> On Tue, Apr 28, 2020 at 3:05 PM Hannah Jiang 
> wrote:
>
>> We need to make sure the release images contains licenses/notices in
>> order to avoid potential legal issues. The purpose of setting the default
>> to pull is checking PRs which introduce new dependencies include thier
>> licenses as well, in a way of auto pulling or tool pulling, to make sure
>> all licenses are included when we create release images. If setting 
>> default
>> to skip, we only check licenses when we create release images and may see
>> many issues with the licenses, and the release would be delayed, maybe a
>> lot.
>>
>>
>> On Tue, Apr 28, 2020 at 2:50 PM Kyle Weaver 
>> wrote:
>>
>>> The different modes make sense. One disagreement: I think the
>>> default should be skip. I imagine few users would want to put licenses 
>>> in
>>> their own images.
>>>
>>> On Tue, Apr 28, 2020 at 5:36 PM Hannah Jiang 
>>> wrote:
>>>
 The above 1,2,3,4 are merged

Re: Companies using Beam?

[Kyle Weaver]

> One danger I've seen is that such pages can grow stale/feel dated if not
regularly updated/added to, so we should have a plan there.

Definitely. Case in point: one of our existing testimonials is from data
artisans, which isn't even that company's name anymore.

On Tue, Apr 28, 2020 at 7:35 PM Robert Bradshaw  wrote:

> I think this is a great idea, as long as we can get critical mass. One
> danger I've seen is that such pages can grow stale/feel dated if not
> regularly updated/added to, so we should have a plan there.
>
> On Tue, Apr 28, 2020 at 4:21 PM Aizhamal Nurmamat kyzy <
> aizha...@apache.org> wrote:
>
>> +1 on adding testimonials/snippets from users either or both on GitHub
>> and website.
>>
>> As part of the Beam website redesign proposal [1], we are suggesting to
>> add a few pages like, "Powered by" or "Use Cases" which users can refer to
>> and find information on who is using Beam and why. The proposal hasn't been
>> finalized yet, we are expecting to continue to work on it after we finish
>> migrating the website to Hugo.
>>
>> It would be awesome if we started collecting those testimonials, so that
>> we can share them on the Beam site once there's a Powered By / Use cases
>> section. Austin, is that something that you're interested in helping with?
>>
>> [1]
>> https://docs.google.com/document/d/1btXMkQGqYaU9pUjYh0iCNrbXf4pAHe3tBFsLQ_cLYHg/edit
>>
>>
>> On Tue, Apr 28, 2020 at 3:06 PM Kyle Weaver  wrote:
>>
>>> I agree it'd be beneficial to the community to highlight users better.
>>> Rather than just a list, though, perhaps we can get some additional user
>>> testimonials (either by requesting them or linking to existing blog posts,
>>> etc). Testimonials are more effort to write, but I think it's more enticing
>>> and informative to potential new users to show, in addition to _who_ is
>>> using Beam, _how_ they are using it. I see there are three already on
>>> Beam's homepage, maybe we could add more there, or add it to a different
>>> page, such as the overview page (
>>> https://beam.apache.org/get-started/beam-overview/). For example, I
>>> visited GRPC's website by chance earlier today, and I noticed they had
>>> testimonials from 8 different organizations, that go into detail about why
>>> they use the software: https://grpc.io/about/#whos-using-grpc-and-why
>>>
>>> On Tue, Apr 28, 2020 at 5:43 PM Austin Bennett <
>>> whatwouldausti...@gmail.com> wrote:
>>>
 Hi All,

 Have we considered getting onto our website or our our GitHub repo the
 ability for individuals to share that their company is using Beam?  Seeing
 - what I believe to be a reasonable list of - companies productively using
 Beam would be helpful to point others to.  For instance, a common question
 I get is whether anyone or who is using?  I'm not sure that's the best
 metric or datapoint in many cases for adoption, but a heuristic that some
 rely upon.

 Naturally, we could ask for a roll-call, esp. via user list, but
 imagining  a persistent web-list would be of interest.

 Cheers,
 Austin


 P.S.  If putting such a list into our repo, that would also get some
 people to submit PRs (so more contributors!) :-)

Jenkins test triggers

[Udi Meiri]

Hi,
Has anyone noticed any changes today (since ~6h ago) to how tests are
triggered on PRs?

Are they triggering always, some of the time, not at all?
Are phrase comment triggers working?

Thanks.
(background: https://issues.apache.org/jira/browse/INFRA-19836)


smime.p7s
Description: S/MIME Cryptographic Signature

Re: sdks:java:container:generateThirdPartyLicenses effect on build time / stability

[Hannah Jiang]

>
> I guess I assumed there was some reason we needed "lightweight images" in
> our tests (because licenses take up a lot of space IIRC), but maybe not.
> Can you elaborate on the purpose of this option Hannah?

Reducing image size is a good reason to have the pull option. There were
user requests to create lightweight images. Now I think users can use skip
option to create lightweight iamges. pull option is not needed.

Then the discussion becomes which one should be the default mode? According
to feedback, skip should be the default mode, and change it to add mode
when running Jenkins test. And the docker-pull-licenses tag is binary again.
It seems like there is an easy way to pass the tag to all Jenkins test,
which is adding *context.switches("-Pdocker-pull-licenses"**)* to
CommonJobProperties.groovy
.
I'm not very familiar with Jenkins, does this would work as expected?
If it sounds like a bad idea to pass this tag to ALL tests, I would ask
help from community to identify those tests which create docker images
(Java and Python for now) and pass the tag to the tests.


On Tue, Apr 28, 2020 at 3:50 PM Kyle Weaver  wrote:

> > To overwrite, we need to pass the tag for ALL Jenkins tests that create
> docker images, which is quite a bunch of work.
>
> If it comes to that, I'd rather we do the work of passing tags instead of
> users.
>
> > Does anyone know if there is an easy way to pass the tag to many tests?
> or overwrite the default mode, which will be defined at gradle.properties,
> to pull ONLY with Jenkins tests?
>
> There is precedence for the latter option:
> https://github.com/apache/beam/blob/1905dbde5ca0858fce89f65c761e88511840a384/build.gradle#L45
>
> > (On that note, however, pull seems bad for both.)
>
> I guess I assumed there was some reason we needed "lightweight images" in
> our tests (because licenses take up a lot of space IIRC), but maybe not.
> Can you elaborate on the purpose of this option Hannah?
>
> On Tue, Apr 28, 2020 at 6:40 PM Robert Bradshaw 
> wrote:
>
>> Fundamentally having license checking off by default is dangerous for
>> releases, but having it on by default is annoying for developers. (On that
>> note, however, pull seems bad for both.) Is it possible to make it a gradle
>> target that only runs when something (specifically dependencies, or the
>> files declaring them) have changed, and would leverage the gradle cache
>> (and hence be cheap) otherwise?
>>
>> Alternatively, I think we should invest in URL caching. These urls should
>> be immutable; let's only download them once, ever.
>>
>> On Tue, Apr 28, 2020 at 3:31 PM Hannah Jiang 
>> wrote:
>>
>>> Do you mean set the default mode to skip and overwrite it to pull mode
>>> with Jenkins test? To overwrite, we need to pass the tag for ALL Jenkins
>>> tests that create docker images, which is quite a bunch of work.
>>> Does anyone know if there is an easy way to pass the tag to many tests?
>>> or overwrite the default mode, which will be defined at gradle.properties,
>>> to pull ONLY with Jenkins tests? We can add a step to do this after cloning
>>> a git branch to Jenkins machine. But it seems easier to change the local
>>> gradle.properties for local development purpose..?
>>>
>>>
>>>
>>> On Tue, Apr 28, 2020 at 3:09 PM Thomas Weise  wrote:
>>>
 Can this be solved by enabling the license magic for the Jenkins jobs?

 I also think the default should be off for better local development
 experience.

 On Tue, Apr 28, 2020 at 3:05 PM Hannah Jiang 
 wrote:

> We need to make sure the release images contains licenses/notices in
> order to avoid potential legal issues. The purpose of setting the default
> to pull is checking PRs which introduce new dependencies include thier
> licenses as well, in a way of auto pulling or tool pulling, to make sure
> all licenses are included when we create release images. If setting 
> default
> to skip, we only check licenses when we create release images and may see
> many issues with the licenses, and the release would be delayed, maybe a
> lot.
>
>
> On Tue, Apr 28, 2020 at 2:50 PM Kyle Weaver 
> wrote:
>
>> The different modes make sense. One disagreement: I think the default
>> should be skip. I imagine few users would want to put licenses in their 
>> own
>> images.
>>
>> On Tue, Apr 28, 2020 at 5:36 PM Hannah Jiang 
>> wrote:
>>
>>> The above 1,2,3,4 are merged to master.
>>> Here I would like to change behaviors with *docker-pull-licenses*
>>> tag and would like to collect feedbacks from community before 
>>> finalizing my
>>> PR.
>>>
>>> docker-pull-licenses can be set to one of ['add', 'pull', 'skip'].
>>>
>>>- *docker-pull-licenses=add* pulls licenses/notices and the
>>>files are added to docker images. This tag is used

Re: sdks:java:container:generateThirdPartyLicenses effect on build time / stability

[Valentyn Tymofieiev]

How about we pull/add dependencies when either:
- The container build target is executed for a release branch [1].
- We are running the build target on Jenkins:
  -  some environmental variable tells us that we are running on Jenkins
  - path to repo starts with /home/jenkins
  - perhaps there is a better way to check this.

Would this mitigate concerns for local development mentioned here?

[1]
https://github.com/apache/beam/blob/74a6565c8b64d9fadf256370df47a4c5dadafb55/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L296


On Tue, Apr 28, 2020 at 3:50 PM Kyle Weaver  wrote:

> > To overwrite, we need to pass the tag for ALL Jenkins tests that create
> docker images, which is quite a bunch of work.
>
> If it comes to that, I'd rather we do the work of passing tags instead of
> users.
>
> > Does anyone know if there is an easy way to pass the tag to many tests?
> or overwrite the default mode, which will be defined at gradle.properties,
> to pull ONLY with Jenkins tests?
>
> There is precedence for the latter option:
> https://github.com/apache/beam/blob/1905dbde5ca0858fce89f65c761e88511840a384/build.gradle#L45
>
> > (On that note, however, pull seems bad for both.)
>
> I guess I assumed there was some reason we needed "lightweight images" in
> our tests (because licenses take up a lot of space IIRC), but maybe not.
> Can you elaborate on the purpose of this option Hannah?
>
> On Tue, Apr 28, 2020 at 6:40 PM Robert Bradshaw 
> wrote:
>
>> Fundamentally having license checking off by default is dangerous for
>> releases, but having it on by default is annoying for developers. (On that
>> note, however, pull seems bad for both.) Is it possible to make it a gradle
>> target that only runs when something (specifically dependencies, or the
>> files declaring them) have changed, and would leverage the gradle cache
>> (and hence be cheap) otherwise?
>>
>> Alternatively, I think we should invest in URL caching. These urls should
>> be immutable; let's only download them once, ever.
>>
>> On Tue, Apr 28, 2020 at 3:31 PM Hannah Jiang 
>> wrote:
>>
>>> Do you mean set the default mode to skip and overwrite it to pull mode
>>> with Jenkins test? To overwrite, we need to pass the tag for ALL Jenkins
>>> tests that create docker images, which is quite a bunch of work.
>>> Does anyone know if there is an easy way to pass the tag to many tests?
>>> or overwrite the default mode, which will be defined at gradle.properties,
>>> to pull ONLY with Jenkins tests? We can add a step to do this after cloning
>>> a git branch to Jenkins machine. But it seems easier to change the local
>>> gradle.properties for local development purpose..?
>>>
>>>
>>>
>>> On Tue, Apr 28, 2020 at 3:09 PM Thomas Weise  wrote:
>>>
 Can this be solved by enabling the license magic for the Jenkins jobs?

 I also think the default should be off for better local development
 experience.

 On Tue, Apr 28, 2020 at 3:05 PM Hannah Jiang 
 wrote:

> We need to make sure the release images contains licenses/notices in
> order to avoid potential legal issues. The purpose of setting the default
> to pull is checking PRs which introduce new dependencies include thier
> licenses as well, in a way of auto pulling or tool pulling, to make sure
> all licenses are included when we create release images. If setting 
> default
> to skip, we only check licenses when we create release images and may see
> many issues with the licenses, and the release would be delayed, maybe a
> lot.
>
>
> On Tue, Apr 28, 2020 at 2:50 PM Kyle Weaver 
> wrote:
>
>> The different modes make sense. One disagreement: I think the default
>> should be skip. I imagine few users would want to put licenses in their 
>> own
>> images.
>>
>> On Tue, Apr 28, 2020 at 5:36 PM Hannah Jiang 
>> wrote:
>>
>>> The above 1,2,3,4 are merged to master.
>>> Here I would like to change behaviors with *docker-pull-licenses*
>>> tag and would like to collect feedbacks from community before 
>>> finalizing my
>>> PR.
>>>
>>> docker-pull-licenses can be set to one of ['add', 'pull', 'skip'].
>>>
>>>- *docker-pull-licenses=add* pulls licenses/notices and the
>>>files are added to docker images. This tag is used to create release 
>>> docker
>>>images.
>>>- *docker-pull-licenses=pull* pulls licenses/notices but do not
>>>add the files to docker images so that create lightweight images. 
>>> This is
>>>the default mode, and docker images created by Jenkins test use this 
>>> mode.
>>>This will make sure the files are all pullable and detect issues 
>>> with the
>>>tool, except the ADD part at DockerFile, which is rarely likely to 
>>> be an
>>>issue. It is better than the checking URLs approach mentioned above,
>>>because it is a more similar proce

Re: Companies using Beam?

[Robert Bradshaw]

I think this is a great idea, as long as we can get critical mass. One
danger I've seen is that such pages can grow stale/feel dated if not
regularly updated/added to, so we should have a plan there.

On Tue, Apr 28, 2020 at 4:21 PM Aizhamal Nurmamat kyzy 
wrote:

> +1 on adding testimonials/snippets from users either or both on GitHub and
> website.
>
> As part of the Beam website redesign proposal [1], we are suggesting to
> add a few pages like, "Powered by" or "Use Cases" which users can refer to
> and find information on who is using Beam and why. The proposal hasn't been
> finalized yet, we are expecting to continue to work on it after we finish
> migrating the website to Hugo.
>
> It would be awesome if we started collecting those testimonials, so that
> we can share them on the Beam site once there's a Powered By / Use cases
> section. Austin, is that something that you're interested in helping with?
>
> [1]
> https://docs.google.com/document/d/1btXMkQGqYaU9pUjYh0iCNrbXf4pAHe3tBFsLQ_cLYHg/edit
>
>
> On Tue, Apr 28, 2020 at 3:06 PM Kyle Weaver  wrote:
>
>> I agree it'd be beneficial to the community to highlight users better.
>> Rather than just a list, though, perhaps we can get some additional user
>> testimonials (either by requesting them or linking to existing blog posts,
>> etc). Testimonials are more effort to write, but I think it's more enticing
>> and informative to potential new users to show, in addition to _who_ is
>> using Beam, _how_ they are using it. I see there are three already on
>> Beam's homepage, maybe we could add more there, or add it to a different
>> page, such as the overview page (
>> https://beam.apache.org/get-started/beam-overview/). For example, I
>> visited GRPC's website by chance earlier today, and I noticed they had
>> testimonials from 8 different organizations, that go into detail about why
>> they use the software: https://grpc.io/about/#whos-using-grpc-and-why
>>
>> On Tue, Apr 28, 2020 at 5:43 PM Austin Bennett <
>> whatwouldausti...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> Have we considered getting onto our website or our our GitHub repo the
>>> ability for individuals to share that their company is using Beam?  Seeing
>>> - what I believe to be a reasonable list of - companies productively using
>>> Beam would be helpful to point others to.  For instance, a common question
>>> I get is whether anyone or who is using?  I'm not sure that's the best
>>> metric or datapoint in many cases for adoption, but a heuristic that some
>>> rely upon.
>>>
>>> Naturally, we could ask for a roll-call, esp. via user list, but
>>> imagining  a persistent web-list would be of interest.
>>>
>>> Cheers,
>>> Austin
>>>
>>>
>>> P.S.  If putting such a list into our repo, that would also get some
>>> people to submit PRs (so more contributors!) :-)
>>>
>>>
>>>

Re: Companies using Beam?

[Aizhamal Nurmamat kyzy]

+1 on adding testimonials/snippets from users either or both on GitHub and
website.

As part of the Beam website redesign proposal [1], we are suggesting to add
a few pages like, "Powered by" or "Use Cases" which users can refer to and
find information on who is using Beam and why. The proposal hasn't been
finalized yet, we are expecting to continue to work on it after we finish
migrating the website to Hugo.

It would be awesome if we started collecting those testimonials, so that we
can share them on the Beam site once there's a Powered By / Use cases
section. Austin, is that something that you're interested in helping with?

[1]
https://docs.google.com/document/d/1btXMkQGqYaU9pUjYh0iCNrbXf4pAHe3tBFsLQ_cLYHg/edit


On Tue, Apr 28, 2020 at 3:06 PM Kyle Weaver  wrote:

> I agree it'd be beneficial to the community to highlight users better.
> Rather than just a list, though, perhaps we can get some additional user
> testimonials (either by requesting them or linking to existing blog posts,
> etc). Testimonials are more effort to write, but I think it's more enticing
> and informative to potential new users to show, in addition to _who_ is
> using Beam, _how_ they are using it. I see there are three already on
> Beam's homepage, maybe we could add more there, or add it to a different
> page, such as the overview page (
> https://beam.apache.org/get-started/beam-overview/). For example, I
> visited GRPC's website by chance earlier today, and I noticed they had
> testimonials from 8 different organizations, that go into detail about why
> they use the software: https://grpc.io/about/#whos-using-grpc-and-why
>
> On Tue, Apr 28, 2020 at 5:43 PM Austin Bennett <
> whatwouldausti...@gmail.com> wrote:
>
>> Hi All,
>>
>> Have we considered getting onto our website or our our GitHub repo the
>> ability for individuals to share that their company is using Beam?  Seeing
>> - what I believe to be a reasonable list of - companies productively using
>> Beam would be helpful to point others to.  For instance, a common question
>> I get is whether anyone or who is using?  I'm not sure that's the best
>> metric or datapoint in many cases for adoption, but a heuristic that some
>> rely upon.
>>
>> Naturally, we could ask for a roll-call, esp. via user list, but
>> imagining  a persistent web-list would be of interest.
>>
>> Cheers,
>> Austin
>>
>>
>> P.S.  If putting such a list into our repo, that would also get some
>> people to submit PRs (so more contributors!) :-)
>>
>>
>>

Re: sdks:java:container:generateThirdPartyLicenses effect on build time / stability

[Kyle Weaver]

> To overwrite, we need to pass the tag for ALL Jenkins tests that create
docker images, which is quite a bunch of work.

If it comes to that, I'd rather we do the work of passing tags instead of
users.

> Does anyone know if there is an easy way to pass the tag to many tests?
or overwrite the default mode, which will be defined at gradle.properties,
to pull ONLY with Jenkins tests?

There is precedence for the latter option:
https://github.com/apache/beam/blob/1905dbde5ca0858fce89f65c761e88511840a384/build.gradle#L45

> (On that note, however, pull seems bad for both.)

I guess I assumed there was some reason we needed "lightweight images" in
our tests (because licenses take up a lot of space IIRC), but maybe not.
Can you elaborate on the purpose of this option Hannah?

On Tue, Apr 28, 2020 at 6:40 PM Robert Bradshaw  wrote:

> Fundamentally having license checking off by default is dangerous for
> releases, but having it on by default is annoying for developers. (On that
> note, however, pull seems bad for both.) Is it possible to make it a gradle
> target that only runs when something (specifically dependencies, or the
> files declaring them) have changed, and would leverage the gradle cache
> (and hence be cheap) otherwise?
>
> Alternatively, I think we should invest in URL caching. These urls should
> be immutable; let's only download them once, ever.
>
> On Tue, Apr 28, 2020 at 3:31 PM Hannah Jiang 
> wrote:
>
>> Do you mean set the default mode to skip and overwrite it to pull mode
>> with Jenkins test? To overwrite, we need to pass the tag for ALL Jenkins
>> tests that create docker images, which is quite a bunch of work.
>> Does anyone know if there is an easy way to pass the tag to many tests?
>> or overwrite the default mode, which will be defined at gradle.properties,
>> to pull ONLY with Jenkins tests? We can add a step to do this after cloning
>> a git branch to Jenkins machine. But it seems easier to change the local
>> gradle.properties for local development purpose..?
>>
>>
>>
>> On Tue, Apr 28, 2020 at 3:09 PM Thomas Weise  wrote:
>>
>>> Can this be solved by enabling the license magic for the Jenkins jobs?
>>>
>>> I also think the default should be off for better local development
>>> experience.
>>>
>>> On Tue, Apr 28, 2020 at 3:05 PM Hannah Jiang 
>>> wrote:
>>>
 We need to make sure the release images contains licenses/notices in
 order to avoid potential legal issues. The purpose of setting the default
 to pull is checking PRs which introduce new dependencies include thier
 licenses as well, in a way of auto pulling or tool pulling, to make sure
 all licenses are included when we create release images. If setting default
 to skip, we only check licenses when we create release images and may see
 many issues with the licenses, and the release would be delayed, maybe a
 lot.


 On Tue, Apr 28, 2020 at 2:50 PM Kyle Weaver 
 wrote:

> The different modes make sense. One disagreement: I think the default
> should be skip. I imagine few users would want to put licenses in their 
> own
> images.
>
> On Tue, Apr 28, 2020 at 5:36 PM Hannah Jiang 
> wrote:
>
>> The above 1,2,3,4 are merged to master.
>> Here I would like to change behaviors with *docker-pull-licenses*
>> tag and would like to collect feedbacks from community before finalizing 
>> my
>> PR.
>>
>> docker-pull-licenses can be set to one of ['add', 'pull', 'skip'].
>>
>>- *docker-pull-licenses=add* pulls licenses/notices and the files
>>are added to docker images. This tag is used to create release docker
>>images.
>>- *docker-pull-licenses=pull* pulls licenses/notices but do not
>>add the files to docker images so that create lightweight images. 
>> This is
>>the default mode, and docker images created by Jenkins test use this 
>> mode.
>>This will make sure the files are all pullable and detect issues with 
>> the
>>tool, except the ADD part at DockerFile, which is rarely likely to be 
>> an
>>issue. It is better than the checking URLs approach mentioned above,
>>because it is a more similar process like creating release images and 
>> code
>>can be simplified so that make it easier to maintain.
>>- *docker-pull-licenses=skip* skips all license pulling related
>>tasks. This tag is provided for users who customize their images with
>>DockerFile provided by us and would not like to deal with license 
>> stuff.
>>Without this tag, the users should change the .gradle file to get rid 
>> of
>>it. It also can be used for local development when developers want to 
>> solve
>>license related issues later.
>>
>> I would like to see this tag is adopted by all docker images released
>> by Beam, including Flink and Spark job server images, and share 

Re: sdks:java:container:generateThirdPartyLicenses effect on build time / stability

[Robert Bradshaw]

Fundamentally having license checking off by default is dangerous for
releases, but having it on by default is annoying for developers. (On that
note, however, pull seems bad for both.) Is it possible to make it a gradle
target that only runs when something (specifically dependencies, or the
files declaring them) have changed, and would leverage the gradle cache
(and hence be cheap) otherwise?

Alternatively, I think we should invest in URL caching. These urls should
be immutable; let's only download them once, ever.

On Tue, Apr 28, 2020 at 3:31 PM Hannah Jiang  wrote:

> Do you mean set the default mode to skip and overwrite it to pull mode
> with Jenkins test? To overwrite, we need to pass the tag for ALL Jenkins
> tests that create docker images, which is quite a bunch of work.
> Does anyone know if there is an easy way to pass the tag to many tests? or
> overwrite the default mode, which will be defined at gradle.properties, to
> pull ONLY with Jenkins tests? We can add a step to do this after cloning a
> git branch to Jenkins machine. But it seems easier to change the local
> gradle.properties for local development purpose..?
>
>
>
> On Tue, Apr 28, 2020 at 3:09 PM Thomas Weise  wrote:
>
>> Can this be solved by enabling the license magic for the Jenkins jobs?
>>
>> I also think the default should be off for better local development
>> experience.
>>
>> On Tue, Apr 28, 2020 at 3:05 PM Hannah Jiang 
>> wrote:
>>
>>> We need to make sure the release images contains licenses/notices in
>>> order to avoid potential legal issues. The purpose of setting the default
>>> to pull is checking PRs which introduce new dependencies include thier
>>> licenses as well, in a way of auto pulling or tool pulling, to make sure
>>> all licenses are included when we create release images. If setting default
>>> to skip, we only check licenses when we create release images and may see
>>> many issues with the licenses, and the release would be delayed, maybe a
>>> lot.
>>>
>>>
>>> On Tue, Apr 28, 2020 at 2:50 PM Kyle Weaver  wrote:
>>>
 The different modes make sense. One disagreement: I think the default
 should be skip. I imagine few users would want to put licenses in their own
 images.

 On Tue, Apr 28, 2020 at 5:36 PM Hannah Jiang 
 wrote:

> The above 1,2,3,4 are merged to master.
> Here I would like to change behaviors with *docker-pull-licenses* tag
> and would like to collect feedbacks from community before finalizing my 
> PR.
>
> docker-pull-licenses can be set to one of ['add', 'pull', 'skip'].
>
>- *docker-pull-licenses=add* pulls licenses/notices and the files
>are added to docker images. This tag is used to create release docker
>images.
>- *docker-pull-licenses=pull* pulls licenses/notices but do not
>add the files to docker images so that create lightweight images. This 
> is
>the default mode, and docker images created by Jenkins test use this 
> mode.
>This will make sure the files are all pullable and detect issues with 
> the
>tool, except the ADD part at DockerFile, which is rarely likely to be 
> an
>issue. It is better than the checking URLs approach mentioned above,
>because it is a more similar process like creating release images and 
> code
>can be simplified so that make it easier to maintain.
>- *docker-pull-licenses=skip* skips all license pulling related
>tasks. This tag is provided for users who customize their images with
>DockerFile provided by us and would not like to deal with license 
> stuff.
>Without this tag, the users should change the .gradle file to get rid 
> of
>it. It also can be used for local development when developers want to 
> solve
>license related issues later.
>
> I would like to see this tag is adopted by all docker images released
> by Beam, including Flink and Spark job server images, and share the same
> default mode.
> Does this sound good? Are there any concerns?
>
> Please let me know what you think.
> Hannah
>
>
>
> On Thu, Apr 16, 2020 at 11:46 AM Hannah Jiang 
> wrote:
>
>> Ok, so the fianl approach is
>> 1. Using multi threading with 16 threads.
>> 2. Introduce docker-pull-license tag, by default it is disabled.
>> 3. By default, only check if urls return 200, not actually pull the
>> files to reduce image size. Licenses add 85MB of image size as of today.
>> 4. Add retries to avoid flakiness. Initially it was added to download
>> part only, assuming urlib is stable when validates urls, but it is not 
>> true.
>>
>> For caching, it is definitely useful. It will mitigate flakiness
>> further, like when github is down (many urls point to github). It also
>> reduces unnecessary internet traffic.
>> Let's reconsider it after the curre

Re: sdks:java:container:generateThirdPartyLicenses effect on build time / stability

[Hannah Jiang]

Do you mean set the default mode to skip and overwrite it to pull mode with
Jenkins test? To overwrite, we need to pass the tag for ALL Jenkins tests
that create docker images, which is quite a bunch of work.
Does anyone know if there is an easy way to pass the tag to many tests? or
overwrite the default mode, which will be defined at gradle.properties, to
pull ONLY with Jenkins tests? We can add a step to do this after cloning a
git branch to Jenkins machine. But it seems easier to change the local
gradle.properties for local development purpose..?



On Tue, Apr 28, 2020 at 3:09 PM Thomas Weise  wrote:

> Can this be solved by enabling the license magic for the Jenkins jobs?
>
> I also think the default should be off for better local development
> experience.
>
> On Tue, Apr 28, 2020 at 3:05 PM Hannah Jiang 
> wrote:
>
>> We need to make sure the release images contains licenses/notices in
>> order to avoid potential legal issues. The purpose of setting the default
>> to pull is checking PRs which introduce new dependencies include thier
>> licenses as well, in a way of auto pulling or tool pulling, to make sure
>> all licenses are included when we create release images. If setting default
>> to skip, we only check licenses when we create release images and may see
>> many issues with the licenses, and the release would be delayed, maybe a
>> lot.
>>
>>
>> On Tue, Apr 28, 2020 at 2:50 PM Kyle Weaver  wrote:
>>
>>> The different modes make sense. One disagreement: I think the default
>>> should be skip. I imagine few users would want to put licenses in their own
>>> images.
>>>
>>> On Tue, Apr 28, 2020 at 5:36 PM Hannah Jiang 
>>> wrote:
>>>
 The above 1,2,3,4 are merged to master.
 Here I would like to change behaviors with *docker-pull-licenses* tag
 and would like to collect feedbacks from community before finalizing my PR.

 docker-pull-licenses can be set to one of ['add', 'pull', 'skip'].

- *docker-pull-licenses=add* pulls licenses/notices and the files
are added to docker images. This tag is used to create release docker
images.
- *docker-pull-licenses=pull* pulls licenses/notices but do not add
the files to docker images so that create lightweight images. This is 
 the
default mode, and docker images created by Jenkins test use this mode. 
 This
will make sure the files are all pullable and detect issues with the 
 tool,
except the ADD part at DockerFile, which is rarely likely to be an 
 issue.
It is better than the checking URLs approach mentioned above, because 
 it is
a more similar process like creating release images and code can be
simplified so that make it easier to maintain.
- *docker-pull-licenses=skip* skips all license pulling related
tasks. This tag is provided for users who customize their images with
DockerFile provided by us and would not like to deal with license stuff.
Without this tag, the users should change the .gradle file to get rid of
it. It also can be used for local development when developers want to 
 solve
license related issues later.

 I would like to see this tag is adopted by all docker images released
 by Beam, including Flink and Spark job server images, and share the same
 default mode.
 Does this sound good? Are there any concerns?

 Please let me know what you think.
 Hannah



 On Thu, Apr 16, 2020 at 11:46 AM Hannah Jiang 
 wrote:

> Ok, so the fianl approach is
> 1. Using multi threading with 16 threads.
> 2. Introduce docker-pull-license tag, by default it is disabled.
> 3. By default, only check if urls return 200, not actually pull the
> files to reduce image size. Licenses add 85MB of image size as of today.
> 4. Add retries to avoid flakiness. Initially it was added to download
> part only, assuming urlib is stable when validates urls, but it is not 
> true.
>
> For caching, it is definitely useful. It will mitigate flakiness
> further, like when github is down (many urls point to github). It also
> reduces unnecessary internet traffic.
> Let's reconsider it after the current work is merged to 2.21.0.
>
> Thank you all for providing feedback and ideas.
>
>
> On Thu, Apr 16, 2020 at 10:24 AM Kyle Weaver 
> wrote:
>
>> > I tried with multi processing and it improved the performance a lot.
>>
>> Great! Though it won't help with flakes, so as you said we should
>> still look into caching as well.
>>
>> > You could try using a threadpool
>>
>> +1
>>
>> > However, we would like to include this work as part of 2.21.0
>>
>> I had marked the jira as a blocker for 2.21.0 because I was afraid
>> something was broken, but now it looks like the failures were just 
>> flakes.
>> So BEAM-9764

Re: sdks:java:container:generateThirdPartyLicenses effect on build time / stability

[Thomas Weise]

Can this be solved by enabling the license magic for the Jenkins jobs?

I also think the default should be off for better local development
experience.

On Tue, Apr 28, 2020 at 3:05 PM Hannah Jiang  wrote:

> We need to make sure the release images contains licenses/notices in order
> to avoid potential legal issues. The purpose of setting the default to pull
> is checking PRs which introduce new dependencies include thier licenses as
> well, in a way of auto pulling or tool pulling, to make sure all licenses
> are included when we create release images. If setting default to skip, we
> only check licenses when we create release images and may see many issues
> with the licenses, and the release would be delayed, maybe a lot.
>
>
> On Tue, Apr 28, 2020 at 2:50 PM Kyle Weaver  wrote:
>
>> The different modes make sense. One disagreement: I think the default
>> should be skip. I imagine few users would want to put licenses in their own
>> images.
>>
>> On Tue, Apr 28, 2020 at 5:36 PM Hannah Jiang 
>> wrote:
>>
>>> The above 1,2,3,4 are merged to master.
>>> Here I would like to change behaviors with *docker-pull-licenses* tag
>>> and would like to collect feedbacks from community before finalizing my PR.
>>>
>>> docker-pull-licenses can be set to one of ['add', 'pull', 'skip'].
>>>
>>>- *docker-pull-licenses=add* pulls licenses/notices and the files
>>>are added to docker images. This tag is used to create release docker
>>>images.
>>>- *docker-pull-licenses=pull* pulls licenses/notices but do not add
>>>the files to docker images so that create lightweight images. This is the
>>>default mode, and docker images created by Jenkins test use this mode. 
>>> This
>>>will make sure the files are all pullable and detect issues with the 
>>> tool,
>>>except the ADD part at DockerFile, which is rarely likely to be an issue.
>>>It is better than the checking URLs approach mentioned above, because it 
>>> is
>>>a more similar process like creating release images and code can be
>>>simplified so that make it easier to maintain.
>>>- *docker-pull-licenses=skip* skips all license pulling related
>>>tasks. This tag is provided for users who customize their images with
>>>DockerFile provided by us and would not like to deal with license stuff.
>>>Without this tag, the users should change the .gradle file to get rid of
>>>it. It also can be used for local development when developers want to 
>>> solve
>>>license related issues later.
>>>
>>> I would like to see this tag is adopted by all docker images released by
>>> Beam, including Flink and Spark job server images, and share the same
>>> default mode.
>>> Does this sound good? Are there any concerns?
>>>
>>> Please let me know what you think.
>>> Hannah
>>>
>>>
>>>
>>> On Thu, Apr 16, 2020 at 11:46 AM Hannah Jiang 
>>> wrote:
>>>
 Ok, so the fianl approach is
 1. Using multi threading with 16 threads.
 2. Introduce docker-pull-license tag, by default it is disabled.
 3. By default, only check if urls return 200, not actually pull the
 files to reduce image size. Licenses add 85MB of image size as of today.
 4. Add retries to avoid flakiness. Initially it was added to download
 part only, assuming urlib is stable when validates urls, but it is not 
 true.

 For caching, it is definitely useful. It will mitigate flakiness
 further, like when github is down (many urls point to github). It also
 reduces unnecessary internet traffic.
 Let's reconsider it after the current work is merged to 2.21.0.

 Thank you all for providing feedback and ideas.


 On Thu, Apr 16, 2020 at 10:24 AM Kyle Weaver 
 wrote:

> > I tried with multi processing and it improved the performance a lot.
>
> Great! Though it won't help with flakes, so as you said we should
> still look into caching as well.
>
> > You could try using a threadpool
>
> +1
>
> > However, we would like to include this work as part of 2.21.0
>
> I had marked the jira as a blocker for 2.21.0 because I was afraid
> something was broken, but now it looks like the failures were just flakes.
> So BEAM-9764  should
> not be a release blocker.
>
> On Thu, Apr 16, 2020 at 1:00 PM Thomas Weise  wrote:
>
>> Hi Hannah,
>>
>> Thanks for investigating!
>>
>> I think it would be great to eliminate the overhead for local builds
>> (by default turn off the license assembly) and make it as lightweight
>> as possible in the frequent CI path.
>>
>> Thomas
>>
>>
>> On Thu, Apr 16, 2020 at 1:37 AM Hannah Jiang 
>> wrote:
>>
>>> I tried to check if urls are valid instead of pulling the files and
>>> it reduced only 1 min of running time. So, it's not an option here.
>>>
>>> I tried with multi processing and it impro

Re: Companies using Beam?

[Kyle Weaver]

I agree it'd be beneficial to the community to highlight users better.
Rather than just a list, though, perhaps we can get some additional user
testimonials (either by requesting them or linking to existing blog posts,
etc). Testimonials are more effort to write, but I think it's more enticing
and informative to potential new users to show, in addition to _who_ is
using Beam, _how_ they are using it. I see there are three already on
Beam's homepage, maybe we could add more there, or add it to a different
page, such as the overview page (
https://beam.apache.org/get-started/beam-overview/). For example, I visited
GRPC's website by chance earlier today, and I noticed they had testimonials
from 8 different organizations, that go into detail about why they use the
software: https://grpc.io/about/#whos-using-grpc-and-why

On Tue, Apr 28, 2020 at 5:43 PM Austin Bennett 
wrote:

> Hi All,
>
> Have we considered getting onto our website or our our GitHub repo the
> ability for individuals to share that their company is using Beam?  Seeing
> - what I believe to be a reasonable list of - companies productively using
> Beam would be helpful to point others to.  For instance, a common question
> I get is whether anyone or who is using?  I'm not sure that's the best
> metric or datapoint in many cases for adoption, but a heuristic that some
> rely upon.
>
> Naturally, we could ask for a roll-call, esp. via user list, but imagining
>  a persistent web-list would be of interest.
>
> Cheers,
> Austin
>
>
> P.S.  If putting such a list into our repo, that would also get some
> people to submit PRs (so more contributors!) :-)
>
>
>

Re: Greetings from Tyson

[Hannah Jiang]

Welcome to the community!


On Tue, Apr 28, 2020 at 2:45 PM Tyson Hamilton  wrote:

> Hello Beam Community,
>
> This is just a simple 'Hello' to introduce myself. I'm a Software Engineer
> at Google and have worked with data processing languages and runtime
> systems on and off during my career. I now have the pleasure of dedicating
> more time towards working with you lovely folks on Beam and I'm really
> excited!
>
> I hope you're all doing well and staying safe in these difficult times.
>
> -Tyson
>
>
>
>

Re: sdks:java:container:generateThirdPartyLicenses effect on build time / stability

[Hannah Jiang]

We need to make sure the release images contains licenses/notices in order
to avoid potential legal issues. The purpose of setting the default to pull
is checking PRs which introduce new dependencies include thier licenses as
well, in a way of auto pulling or tool pulling, to make sure all licenses
are included when we create release images. If setting default to skip, we
only check licenses when we create release images and may see many issues
with the licenses, and the release would be delayed, maybe a lot.


On Tue, Apr 28, 2020 at 2:50 PM Kyle Weaver  wrote:

> The different modes make sense. One disagreement: I think the default
> should be skip. I imagine few users would want to put licenses in their own
> images.
>
> On Tue, Apr 28, 2020 at 5:36 PM Hannah Jiang 
> wrote:
>
>> The above 1,2,3,4 are merged to master.
>> Here I would like to change behaviors with *docker-pull-licenses* tag
>> and would like to collect feedbacks from community before finalizing my PR.
>>
>> docker-pull-licenses can be set to one of ['add', 'pull', 'skip'].
>>
>>- *docker-pull-licenses=add* pulls licenses/notices and the files are
>>added to docker images. This tag is used to create release docker images.
>>- *docker-pull-licenses=pull* pulls licenses/notices but do not add
>>the files to docker images so that create lightweight images. This is the
>>default mode, and docker images created by Jenkins test use this mode. 
>> This
>>will make sure the files are all pullable and detect issues with the tool,
>>except the ADD part at DockerFile, which is rarely likely to be an issue.
>>It is better than the checking URLs approach mentioned above, because it 
>> is
>>a more similar process like creating release images and code can be
>>simplified so that make it easier to maintain.
>>- *docker-pull-licenses=skip* skips all license pulling related
>>tasks. This tag is provided for users who customize their images with
>>DockerFile provided by us and would not like to deal with license stuff.
>>Without this tag, the users should change the .gradle file to get rid of
>>it. It also can be used for local development when developers want to 
>> solve
>>license related issues later.
>>
>> I would like to see this tag is adopted by all docker images released by
>> Beam, including Flink and Spark job server images, and share the same
>> default mode.
>> Does this sound good? Are there any concerns?
>>
>> Please let me know what you think.
>> Hannah
>>
>>
>>
>> On Thu, Apr 16, 2020 at 11:46 AM Hannah Jiang 
>> wrote:
>>
>>> Ok, so the fianl approach is
>>> 1. Using multi threading with 16 threads.
>>> 2. Introduce docker-pull-license tag, by default it is disabled.
>>> 3. By default, only check if urls return 200, not actually pull the
>>> files to reduce image size. Licenses add 85MB of image size as of today.
>>> 4. Add retries to avoid flakiness. Initially it was added to download
>>> part only, assuming urlib is stable when validates urls, but it is not true.
>>>
>>> For caching, it is definitely useful. It will mitigate flakiness
>>> further, like when github is down (many urls point to github). It also
>>> reduces unnecessary internet traffic.
>>> Let's reconsider it after the current work is merged to 2.21.0.
>>>
>>> Thank you all for providing feedback and ideas.
>>>
>>>
>>> On Thu, Apr 16, 2020 at 10:24 AM Kyle Weaver 
>>> wrote:
>>>
 > I tried with multi processing and it improved the performance a lot.

 Great! Though it won't help with flakes, so as you said we should still
 look into caching as well.

 > You could try using a threadpool

 +1

 > However, we would like to include this work as part of 2.21.0

 I had marked the jira as a blocker for 2.21.0 because I was afraid
 something was broken, but now it looks like the failures were just flakes.
 So BEAM-9764  should
 not be a release blocker.

 On Thu, Apr 16, 2020 at 1:00 PM Thomas Weise  wrote:

> Hi Hannah,
>
> Thanks for investigating!
>
> I think it would be great to eliminate the overhead for local builds
> (by default turn off the license assembly) and make it as lightweight
> as possible in the frequent CI path.
>
> Thomas
>
>
> On Thu, Apr 16, 2020 at 1:37 AM Hannah Jiang 
> wrote:
>
>> I tried to check if urls are valid instead of pulling the files and
>> it reduced only 1 min of running time. So, it's not an option here.
>>
>> I tried with multi processing and it improved the performance a lot.
>> With 12 subprocesses, the running time reduced to 49 seconds, and
>> with 16 cores, it reduced to 18 seconds.
>> The number of subprocesses is defined by the number of cores, and
>> Jenkins machine has 16 cores.
>> FYI: with my local machine (12 cores) and home network, it takes 1min
>> 40 

Greetings from Tyson

[Tyson Hamilton]

Hello Beam Community,

This is just a simple 'Hello' to introduce myself. I'm a Software Engineer at 
Google and have worked with data processing languages and runtime systems on 
and off during my career. I now have the pleasure of dedicating more time 
towards working with you lovely folks on Beam and I'm really excited!

I hope you're all doing well and staying safe in these difficult times.

-Tyson

Re: sdks:java:container:generateThirdPartyLicenses effect on build time / stability

[Kyle Weaver]

The different modes make sense. One disagreement: I think the default
should be skip. I imagine few users would want to put licenses in their own
images.

On Tue, Apr 28, 2020 at 5:36 PM Hannah Jiang  wrote:

> The above 1,2,3,4 are merged to master.
> Here I would like to change behaviors with *docker-pull-licenses* tag and
> would like to collect feedbacks from community before finalizing my PR.
>
> docker-pull-licenses can be set to one of ['add', 'pull', 'skip'].
>
>- *docker-pull-licenses=add* pulls licenses/notices and the files are
>added to docker images. This tag is used to create release docker images.
>- *docker-pull-licenses=pull* pulls licenses/notices but do not add
>the files to docker images so that create lightweight images. This is the
>default mode, and docker images created by Jenkins test use this mode. This
>will make sure the files are all pullable and detect issues with the tool,
>except the ADD part at DockerFile, which is rarely likely to be an issue.
>It is better than the checking URLs approach mentioned above, because it is
>a more similar process like creating release images and code can be
>simplified so that make it easier to maintain.
>- *docker-pull-licenses=skip* skips all license pulling related tasks.
>This tag is provided for users who customize their images with DockerFile
>provided by us and would not like to deal with license stuff. Without this
>tag, the users should change the .gradle file to get rid of it. It also can
>be used for local development when developers want to solve license related
>issues later.
>
> I would like to see this tag is adopted by all docker images released by
> Beam, including Flink and Spark job server images, and share the same
> default mode.
> Does this sound good? Are there any concerns?
>
> Please let me know what you think.
> Hannah
>
>
>
> On Thu, Apr 16, 2020 at 11:46 AM Hannah Jiang 
> wrote:
>
>> Ok, so the fianl approach is
>> 1. Using multi threading with 16 threads.
>> 2. Introduce docker-pull-license tag, by default it is disabled.
>> 3. By default, only check if urls return 200, not actually pull the files
>> to reduce image size. Licenses add 85MB of image size as of today.
>> 4. Add retries to avoid flakiness. Initially it was added to download
>> part only, assuming urlib is stable when validates urls, but it is not true.
>>
>> For caching, it is definitely useful. It will mitigate flakiness further,
>> like when github is down (many urls point to github). It also reduces
>> unnecessary internet traffic.
>> Let's reconsider it after the current work is merged to 2.21.0.
>>
>> Thank you all for providing feedback and ideas.
>>
>>
>> On Thu, Apr 16, 2020 at 10:24 AM Kyle Weaver  wrote:
>>
>>> > I tried with multi processing and it improved the performance a lot.
>>>
>>> Great! Though it won't help with flakes, so as you said we should still
>>> look into caching as well.
>>>
>>> > You could try using a threadpool
>>>
>>> +1
>>>
>>> > However, we would like to include this work as part of 2.21.0
>>>
>>> I had marked the jira as a blocker for 2.21.0 because I was afraid
>>> something was broken, but now it looks like the failures were just flakes.
>>> So BEAM-9764  should
>>> not be a release blocker.
>>>
>>> On Thu, Apr 16, 2020 at 1:00 PM Thomas Weise  wrote:
>>>
 Hi Hannah,

 Thanks for investigating!

 I think it would be great to eliminate the overhead for local builds
 (by default turn off the license assembly) and make it as lightweight
 as possible in the frequent CI path.

 Thomas


 On Thu, Apr 16, 2020 at 1:37 AM Hannah Jiang 
 wrote:

> I tried to check if urls are valid instead of pulling the files and it
> reduced only 1 min of running time. So, it's not an option here.
>
> I tried with multi processing and it improved the performance a lot.
> With 12 subprocesses, the running time reduced to 49 seconds, and with
> 16 cores, it reduced to 18 seconds.
> The number of subprocesses is defined by the number of cores, and
> Jenkins machine has 16 cores.
> FYI: with my local machine (12 cores) and home network, it takes 1min
> 40 secs to create a Java docker image.
>
> The caching approach mentioned by Robert brings many benefits, not
> only to this use case.
> However, we would like to include this work as part of 2.21.0, so I
> will move with the multi processing approach this time.
>
> Please let me know if you have objections.
>
>
> On Wed, Apr 15, 2020 at 4:01 PM Robert Bradshaw 
> wrote:
>
>> Is the cost primarily in pulling these remote licenses/sources? I'd
>> guess that 99.9% of the URLs remain the same from run to run. Would a
>> simple cache, or caching proxy, be sufficient?
>>
>> Otherwise, a tag to check that licenses can be pulled

Companies using Beam?

[Austin Bennett]

Hi All,

Have we considered getting onto our website or our our GitHub repo the
ability for individuals to share that their company is using Beam?  Seeing
- what I believe to be a reasonable list of - companies productively using
Beam would be helpful to point others to.  For instance, a common question
I get is whether anyone or who is using?  I'm not sure that's the best
metric or datapoint in many cases for adoption, but a heuristic that some
rely upon.

Naturally, we could ask for a roll-call, esp. via user list, but imagining
 a persistent web-list would be of interest.

Cheers,
Austin


P.S.  If putting such a list into our repo, that would also get some people
to submit PRs (so more contributors!) :-)

Re: sdks:java:container:generateThirdPartyLicenses effect on build time / stability

[Hannah Jiang]

The above 1,2,3,4 are merged to master.
Here I would like to change behaviors with *docker-pull-licenses* tag and
would like to collect feedbacks from community before finalizing my PR.

docker-pull-licenses can be set to one of ['add', 'pull', 'skip'].

   - *docker-pull-licenses=add* pulls licenses/notices and the files are
   added to docker images. This tag is used to create release docker images.
   - *docker-pull-licenses=pull* pulls licenses/notices but do not add the
   files to docker images so that create lightweight images. This is the
   default mode, and docker images created by Jenkins test use this mode. This
   will make sure the files are all pullable and detect issues with the tool,
   except the ADD part at DockerFile, which is rarely likely to be an issue.
   It is better than the checking URLs approach mentioned above, because it is
   a more similar process like creating release images and code can be
   simplified so that make it easier to maintain.
   - *docker-pull-licenses=skip* skips all license pulling related tasks.
   This tag is provided for users who customize their images with DockerFile
   provided by us and would not like to deal with license stuff. Without this
   tag, the users should change the .gradle file to get rid of it. It also can
   be used for local development when developers want to solve license related
   issues later.

I would like to see this tag is adopted by all docker images released by
Beam, including Flink and Spark job server images, and share the same
default mode.
Does this sound good? Are there any concerns?

Please let me know what you think.
Hannah



On Thu, Apr 16, 2020 at 11:46 AM Hannah Jiang 
wrote:

> Ok, so the fianl approach is
> 1. Using multi threading with 16 threads.
> 2. Introduce docker-pull-license tag, by default it is disabled.
> 3. By default, only check if urls return 200, not actually pull the files
> to reduce image size. Licenses add 85MB of image size as of today.
> 4. Add retries to avoid flakiness. Initially it was added to download part
> only, assuming urlib is stable when validates urls, but it is not true.
>
> For caching, it is definitely useful. It will mitigate flakiness further,
> like when github is down (many urls point to github). It also reduces
> unnecessary internet traffic.
> Let's reconsider it after the current work is merged to 2.21.0.
>
> Thank you all for providing feedback and ideas.
>
>
> On Thu, Apr 16, 2020 at 10:24 AM Kyle Weaver  wrote:
>
>> > I tried with multi processing and it improved the performance a lot.
>>
>> Great! Though it won't help with flakes, so as you said we should still
>> look into caching as well.
>>
>> > You could try using a threadpool
>>
>> +1
>>
>> > However, we would like to include this work as part of 2.21.0
>>
>> I had marked the jira as a blocker for 2.21.0 because I was afraid
>> something was broken, but now it looks like the failures were just flakes.
>> So BEAM-9764  should
>> not be a release blocker.
>>
>> On Thu, Apr 16, 2020 at 1:00 PM Thomas Weise  wrote:
>>
>>> Hi Hannah,
>>>
>>> Thanks for investigating!
>>>
>>> I think it would be great to eliminate the overhead for local builds (by
>>> default turn off the license assembly) and make it as lightweight
>>> as possible in the frequent CI path.
>>>
>>> Thomas
>>>
>>>
>>> On Thu, Apr 16, 2020 at 1:37 AM Hannah Jiang 
>>> wrote:
>>>
 I tried to check if urls are valid instead of pulling the files and it
 reduced only 1 min of running time. So, it's not an option here.

 I tried with multi processing and it improved the performance a lot.
 With 12 subprocesses, the running time reduced to 49 seconds, and with
 16 cores, it reduced to 18 seconds.
 The number of subprocesses is defined by the number of cores, and
 Jenkins machine has 16 cores.
 FYI: with my local machine (12 cores) and home network, it takes 1min
 40 secs to create a Java docker image.

 The caching approach mentioned by Robert brings many benefits, not only
 to this use case.
 However, we would like to include this work as part of 2.21.0, so I
 will move with the multi processing approach this time.

 Please let me know if you have objections.


 On Wed, Apr 15, 2020 at 4:01 PM Robert Bradshaw 
 wrote:

> Is the cost primarily in pulling these remote licenses/sources? I'd
> guess that 99.9% of the URLs remain the same from run to run. Would a
> simple cache, or caching proxy, be sufficient?
>
> Otherwise, a tag to check that licenses can be pulled, but not really
> pull them, might be sufficient. (Making sure the default is cheap but we
> don't accidentally omit them when it matters is the tricky bit I see 
> here.)
>
>
> On Wed, Apr 15, 2020 at 3:38 PM Hannah Jiang 
> wrote:
>
>> Thanks for providing feedback.
>>
>> Here is what happending now an

Re: Jacek's new Apache Beam Internals Project

[Ismaël Mejía]

The tweet URL for ref in case someone wants to like/RT

https://twitter.com/jaceklaskowski/status/1255046717277376512?s=19

On Tue, Apr 28, 2020, 8:04 PM Holden Karau  wrote:

> Hi Folks,
>
> I just saw Jacek's tweet about his new Beam Internals project (he's done a
> great job on his Spark Internals documentation and blog posts) and I
> figured I'd share the link
> https://leanpub.com/the-internals-of-apache-beam in case folks are
> interested :)
>
> Cheers,
>
> Holden :)
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: [REVIEW][please pause website changes] Migrated the Beam website to Hugo

[Aizhamal Nurmamat kyzy]

Adding +Nam Bui  and +Karolina Rosół
 to follow up on questions.

On Tue, Apr 28, 2020 at 11:34 AM Ahmet Altay  wrote:

> I am having trouble reviewing the staged version. What is the best way to
> review this change?
>
> Do we expect any changes to markdown files, beyond some metadata?
>
> On Tue, Apr 28, 2020 at 10:45 AM Robert Bradshaw 
> wrote:
>
>> Thanks. It'll be great to better support more languages.
>>
>> I looked at the PR and there seems to be no provenance/history. E.g. all
>> the content seems to be entirely new files rather than diffs from the old.
>> (There also seems to be a huge amount of auto-generated js code as well.)
>>
>
> I agree. This makes it very hard to review. I also see a bunch of deleted
> markdown files. Are they not getting migrated?
>
>
>>
>> On Tue, Apr 28, 2020 at 10:23 AM Aizhamal Nurmamat kyzy <
>> aizha...@apache.org> wrote:
>>
>>> Hello everybody,
>>>
>>> We are almost done migrating the Apache Beam website from Jekyll to
>>> Hugo. You can see the PR in [1], and we'd love to hear your
>>> feedback/comments on the PR. It includes  detailed guidelines on
>>> contributing to the new Hugo-based website and adding translations to pages
>>> [2]. For those who are curious about adding new languages, we will provide
>>> a proof of concept in the next couple of days in this thread.
>>>
>>> Since we want to move forward with the PR, I would like to ask the
>>> community to hold off changes to the current Beam website for a week, until
>>> we are able to review and merge the PR. Is this acceptable to everyone?
>>>
>>> In case anyone missed my previous email with the background for the
>>> website migration, you can find more context here [3].
>>>
>>> Thanks,
>>> Aizhamal
>>>
>>> [1] https://github.com/apache/beam/pull/11554
>>> [2]
>>> https://github.com/apache/beam/blob/256b7042bf504b94f161ca03b388a2ba247918d9/website/CONTRIBUTE.md
>>> [3]
>>> https://lists.apache.org/thread.html/r7fa6d710c0a1959cce5108e460d71c306ce5756cf96af818b41cb7ca%40%3Cdev.beam.apache.org%3E
>>>
>>

Re: [REVIEW][please pause website changes] Migrated the Beam website to Hugo

[Ahmet Altay]

I am having trouble reviewing the staged version. What is the best way to
review this change?

Do we expect any changes to markdown files, beyond some metadata?

On Tue, Apr 28, 2020 at 10:45 AM Robert Bradshaw 
wrote:

> Thanks. It'll be great to better support more languages.
>
> I looked at the PR and there seems to be no provenance/history. E.g. all
> the content seems to be entirely new files rather than diffs from the old.
> (There also seems to be a huge amount of auto-generated js code as well.)
>

I agree. This makes it very hard to review. I also see a bunch of deleted
markdown files. Are they not getting migrated?


>
> On Tue, Apr 28, 2020 at 10:23 AM Aizhamal Nurmamat kyzy <
> aizha...@apache.org> wrote:
>
>> Hello everybody,
>>
>> We are almost done migrating the Apache Beam website from Jekyll to Hugo.
>> You can see the PR in [1], and we'd love to hear your feedback/comments on
>> the PR. It includes  detailed guidelines on contributing to the new
>> Hugo-based website and adding translations to pages [2]. For those who are
>> curious about adding new languages, we will provide a proof of concept in
>> the next couple of days in this thread.
>>
>> Since we want to move forward with the PR, I would like to ask the
>> community to hold off changes to the current Beam website for a week, until
>> we are able to review and merge the PR. Is this acceptable to everyone?
>>
>> In case anyone missed my previous email with the background for the
>> website migration, you can find more context here [3].
>>
>> Thanks,
>> Aizhamal
>>
>> [1] https://github.com/apache/beam/pull/11554
>> [2]
>> https://github.com/apache/beam/blob/256b7042bf504b94f161ca03b388a2ba247918d9/website/CONTRIBUTE.md
>> [3]
>> https://lists.apache.org/thread.html/r7fa6d710c0a1959cce5108e460d71c306ce5756cf96af818b41cb7ca%40%3Cdev.beam.apache.org%3E
>>
>

Re: How to submit PRs for dependant changes?

[Ankur Goenka]

I would prefer c.
I suppose you already have 4 separate changes in 4 separate local branches,
Each of a,b and c will require merging locally. So there shouldn't be much
of a difference in effort in these 3 options.
c would be easiest to review and will preserve information in the most
meaningful way.

On Tue, Apr 28, 2020 at 10:48 AM Robert Bradshaw 
wrote:

> I prefer (c) as well, rebasing as things get merged. I would do (a) if
> they're really prerequisites for one another.
>
> On Tue, Apr 28, 2020 at 10:40 AM Udi Meiri  wrote:
>
>> (a) or (c) should work. (c) is preferred if you want faster reviews.
>>
>> For multiple JIRAs, I've seen both [BEAM-123,BEAM-456] and
>> [BEAM-123][BEAM-456] formats. One of them works but I'm not sure which. :D
>> You can always manually add a PR to a JIRA.
>>
>>
>>
>> On Sun, Apr 26, 2020 at 2:49 PM Reuven Lax  wrote:
>>
>>> For c), I don't think you need merge resolutions. You can submit each
>>> commit in a separate PR, and rebase your branch after each one.
>>>
>>> On Sun, Apr 26, 2020 at 10:25 AM Niel Markwick  wrote:
>>>

 Hey Beam devs...

 I have 4 changes to submit as PRs to fix 4 independent issues in the
 io.gcp.SpannerIO class.

 The PRs are notionally independent, but will cause merge conflicts if
 submitted separately, as the fix for each issue will change code related to
 the fix for some of the others.

 How do you prefer the PRs to be submitted?

 a) one single PR with 4 sequential commits within it
 b) one single PR with all changes squashed.
 c) 4 separate conflicting PRs which will have to be merged separately,
 and a merge conflict resolution after each one.

 a) is how it is in my repo.
 b) would be easy, but less clear what the changes were for.
 c) I guess would be clearest in the Beam changelog.

 If the answer is a) or b), how would I specify multiple JIRA tickets in
 the PR title?

 Thanks!

 --
 
 * •  **Niel Markwick*
 * •  *Cloud Solutions Architect
 
 * •  *Google Belgium
 * •  *ni...@google.com
 * •  *+32 2 894 6771


 Google Belgium NV/SA, Steenweg op Etterbeek 180, 1040 Brussel, Belgie. 
 RPR: 0878.065.378

 If you have received this communication by mistake, please don't
 forward it to anyone else (it may contain confidential or privileged
 information), please erase all copies of it, including all attachments, and
 please let the sender know it went to the wrong person. Thanks

>>>

Jacek's new Apache Beam Internals Project

[Holden Karau]

Hi Folks,

I just saw Jacek's tweet about his new Beam Internals project (he's done a
great job on his Spark Internals documentation and blog posts) and I
figured I'd share the link https://leanpub.com/the-internals-of-apache-beam in
case folks are interested :)

Cheers,

Holden :)

-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: How to submit PRs for dependant changes?

[Robert Bradshaw]

I prefer (c) as well, rebasing as things get merged. I would do (a) if
they're really prerequisites for one another.

On Tue, Apr 28, 2020 at 10:40 AM Udi Meiri  wrote:

> (a) or (c) should work. (c) is preferred if you want faster reviews.
>
> For multiple JIRAs, I've seen both [BEAM-123,BEAM-456] and
> [BEAM-123][BEAM-456] formats. One of them works but I'm not sure which. :D
> You can always manually add a PR to a JIRA.
>
>
>
> On Sun, Apr 26, 2020 at 2:49 PM Reuven Lax  wrote:
>
>> For c), I don't think you need merge resolutions. You can submit each
>> commit in a separate PR, and rebase your branch after each one.
>>
>> On Sun, Apr 26, 2020 at 10:25 AM Niel Markwick  wrote:
>>
>>>
>>> Hey Beam devs...
>>>
>>> I have 4 changes to submit as PRs to fix 4 independent issues in the
>>> io.gcp.SpannerIO class.
>>>
>>> The PRs are notionally independent, but will cause merge conflicts if
>>> submitted separately, as the fix for each issue will change code related to
>>> the fix for some of the others.
>>>
>>> How do you prefer the PRs to be submitted?
>>>
>>> a) one single PR with 4 sequential commits within it
>>> b) one single PR with all changes squashed.
>>> c) 4 separate conflicting PRs which will have to be merged separately,
>>> and a merge conflict resolution after each one.
>>>
>>> a) is how it is in my repo.
>>> b) would be easy, but less clear what the changes were for.
>>> c) I guess would be clearest in the Beam changelog.
>>>
>>> If the answer is a) or b), how would I specify multiple JIRA tickets in
>>> the PR title?
>>>
>>> Thanks!
>>>
>>> --
>>> 
>>> * •  **Niel Markwick*
>>> * •  *Cloud Solutions Architect
>>> 
>>> * •  *Google Belgium
>>> * •  *ni...@google.com
>>> * •  *+32 2 894 6771
>>>
>>>
>>> Google Belgium NV/SA, Steenweg op Etterbeek 180, 1040 Brussel, Belgie. RPR: 
>>> 0878.065.378
>>>
>>> If you have received this communication by mistake, please don't forward
>>> it to anyone else (it may contain confidential or privileged information),
>>> please erase all copies of it, including all attachments, and please let
>>> the sender know it went to the wrong person. Thanks
>>>
>>

Re: [REVIEW][please pause website changes] Migrated the Beam website to Hugo

[Robert Bradshaw]

Thanks. It'll be great to better support more languages.

I looked at the PR and there seems to be no provenance/history. E.g. all
the content seems to be entirely new files rather than diffs from the old.
(There also seems to be a huge amount of auto-generated js code as well.)

On Tue, Apr 28, 2020 at 10:23 AM Aizhamal Nurmamat kyzy 
wrote:

> Hello everybody,
>
> We are almost done migrating the Apache Beam website from Jekyll to Hugo.
> You can see the PR in [1], and we'd love to hear your feedback/comments on
> the PR. It includes  detailed guidelines on contributing to the new
> Hugo-based website and adding translations to pages [2]. For those who are
> curious about adding new languages, we will provide a proof of concept in
> the next couple of days in this thread.
>
> Since we want to move forward with the PR, I would like to ask the
> community to hold off changes to the current Beam website for a week, until
> we are able to review and merge the PR. Is this acceptable to everyone?
>
> In case anyone missed my previous email with the background for the
> website migration, you can find more context here [3].
>
> Thanks,
> Aizhamal
>
> [1] https://github.com/apache/beam/pull/11554
> [2]
> https://github.com/apache/beam/blob/256b7042bf504b94f161ca03b388a2ba247918d9/website/CONTRIBUTE.md
> [3]
> https://lists.apache.org/thread.html/r7fa6d710c0a1959cce5108e460d71c306ce5756cf96af818b41cb7ca%40%3Cdev.beam.apache.org%3E
>

Re: [RESULT][VOTE] Accept the Firefly design donation as Beam Mascot - Deadline Mon April 6

[Pablo Estrada]

I'll be happy to as well!

On Sun, Apr 26, 2020 at 4:18 AM Maximilian Michels  wrote:

> Hey Maria,
>
> I can testify :)
>
> Cheers,
> Max
>
> On 23.04.20 20:49, María Cruz wrote:
> > Hi everyone!
> > It is amazing to see how this process developed to collaboratively
> > create Apache Beam's mascot. Thank you to everyone who got involved!
> > I would like to write a blogpost for the Beam website, and I wanted to
> > ask you: would anyone like to offer their testimony about the process of
> > creating the Beam mascot, and what this means to you? Everyone's
> > testimony is welcome! If you witnessed the development of a mascot for
> > another open source project, even better =)
> >
> > Please feel free to express interest on this thread, and I'll reach out
> > to you off-list.
> >
> > Thanks,
> >
> > María
> >
> > On Fri, Apr 17, 2020 at 6:19 AM Jeff Klukas  > > wrote:
> >
> > I personally like the sound of "Datum" as a name. I also like the
> > idea of not assigning them a gender.
> >
> > As a counterpoint on the naming side, one of the slide decks
> > provided while iterating on the design mentions:
> >
> > > Mascot can change colors when it is “full of data” or has a “batch
> > of data” to process.  Yellow is supercharged and ready to process!
> >
> > Based on that, I'd argue that the mascot maps to the concept of a
> > bundle in the beam execution model and we should consider a name
> > that's a play on "bundle" or perhaps a play on "checkpoint".
> >
> > On Thu, Apr 16, 2020 at 3:44 PM Julian Bruno  > > wrote:
> >
> > Hi all,
> >
> > While working on the design of our Mascot
> > Some ideas showed up and I wish to share them.
> > In regard to Alex Van Boxel's question about the name of our
> Mascot.
> >
> > I was thinking about this yesterday night and feel it could be a
> > great idea to name the Mascot "*Data*" or "*Datum*". Both names
> > sound cute and make sense to me. I prefer the later. Datum means
> > a single piece of information. The Mascot is the first piece of
> > information and its job is to collect batches of data and
> > process it. Datum is in charge of linking information together.
> >
> > In addition, our Mascot should have no gender. Rendering it
> > accessible to all users.
> >
> > Beam as a name for the mascot is pretty straight forward but I
> > think there are many things carrying that same name already.
> >
> > What do you think?
> >
> > Looking forward to hearing your feedback. Names are important
> > and I feel it can expand the personality and create a cool
> > background for our Mascot.
> >
> > Cheers!
> >
> > Julian
> >
> > On Mon, Apr 13, 2020, 3:40 PM Kyle Weaver  > > wrote:
> >
> > Beam Firefly is fine with me (I guess people tend to forget
> > mascot names anyway). But if anyone comes up with something
> > particularly cute/clever we can consider it.
> >
> > On Mon, Apr 13, 2020 at 6:33 PM Aizhamal Nurmamat kyzy
> > mailto:aizha...@apache.org>> wrote:
> >
> > @Alex, Beam Firefly?
> >
> > On Thu, Apr 9, 2020 at 10:57 PM Alex Van Boxel
> > mailto:a...@vanboxel.be>> wrote:
> >
> > We forgot something
> >
> > ...
> >
> > ...
> >
> > it/she/he needs a *name*!
> >
> >
> >  _/
> > _/ Alex Van Boxel
> >
> >
> > On Fri, Apr 10, 2020 at 6:19 AM Kenneth Knowles
> > mailto:k...@apache.org>> wrote:
> >
> > Looking forward to the guide. I enjoy doing
> > (bad) drawings as a way to relax. And I want
> > them to be properly on brand :-)
> >
> > Kenn
> >
> > On Thu, Apr 9, 2020 at 10:35 AM Maximilian
> > Michels mailto:m...@apache.org>>
> > wrote:
> >
> > Awesome. What a milestone! The mascot is a
> > real eye catcher. Thank you
> > Julian and Aizhamal for making it happen.
> >
> > On 06.04.20 22:05, Aizhamal Nurmamat kyzy
> wrote:
> > > I am happy to announce that this vote has
> > passed, with 13 approving +1
> > > votes, 5 of which are binding PMC votes.
> > >
> > > We have the final design for the Beam
> > Firefly! Yahoo!
> > >
> >   

Re: How to submit PRs for dependant changes?

[Udi Meiri]

(a) or (c) should work. (c) is preferred if you want faster reviews.

For multiple JIRAs, I've seen both [BEAM-123,BEAM-456] and
[BEAM-123][BEAM-456] formats. One of them works but I'm not sure which. :D
You can always manually add a PR to a JIRA.



On Sun, Apr 26, 2020 at 2:49 PM Reuven Lax  wrote:

> For c), I don't think you need merge resolutions. You can submit each
> commit in a separate PR, and rebase your branch after each one.
>
> On Sun, Apr 26, 2020 at 10:25 AM Niel Markwick  wrote:
>
>>
>> Hey Beam devs...
>>
>> I have 4 changes to submit as PRs to fix 4 independent issues in the
>> io.gcp.SpannerIO class.
>>
>> The PRs are notionally independent, but will cause merge conflicts if
>> submitted separately, as the fix for each issue will change code related to
>> the fix for some of the others.
>>
>> How do you prefer the PRs to be submitted?
>>
>> a) one single PR with 4 sequential commits within it
>> b) one single PR with all changes squashed.
>> c) 4 separate conflicting PRs which will have to be merged separately,
>> and a merge conflict resolution after each one.
>>
>> a) is how it is in my repo.
>> b) would be easy, but less clear what the changes were for.
>> c) I guess would be clearest in the Beam changelog.
>>
>> If the answer is a) or b), how would I specify multiple JIRA tickets in
>> the PR title?
>>
>> Thanks!
>>
>> --
>> 
>> * •  **Niel Markwick*
>> * •  *Cloud Solutions Architect 
>> * •  *Google Belgium
>> * •  *ni...@google.com
>> * •  *+32 2 894 6771
>>
>>
>> Google Belgium NV/SA, Steenweg op Etterbeek 180, 1040 Brussel, Belgie. RPR: 
>> 0878.065.378
>>
>> If you have received this communication by mistake, please don't forward
>> it to anyone else (it may contain confidential or privileged information),
>> please erase all copies of it, including all attachments, and please let
>> the sender know it went to the wrong person. Thanks
>>
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Jenkins jobs not running for my PR 10438

[Alexey Romanenko]

Thanks Udi! I'll track for updates on this.

> On 28 Apr 2020, at 19:16, Udi Meiri  wrote:
> 
> Alexey, what you're doing should be working (commits should trigger tests, as 
> should "retest this please" and other phrases).
> 
> https://issues.apache.org/jira/browse/INFRA-19836 
>  tracks this issue
> 
> On Tue, Apr 28, 2020 at 10:04 AM Alexey Romanenko  > wrote:
> Does anyone know the “golden rule” how to trigger Jenkins tests?
> 
> For example: 
> https://github.com/apache/beam/pull/11341 
> 
> I tried several times and it’s still not triggered. 
> 
>> On 28 Apr 2020, at 13:33, Ismaël Mejía > > wrote:
>> 
>> done
>> 
>> On Tue, Apr 28, 2020 at 12:47 PM Shoaib Zafar > > wrote:
>> Hello Beam Committers,
>> 
>> I would appreciate if you could trigger precommit checks for the PR: 
>> https://github.com/apache/beam/pull/11210 
>>  along with the python 
>> post-commit check (Run Python 3.5 PostCommit).
>> 
>> Thanks and Regards.
>> Shoaib Zafar
>> Software Engineering Lead
>> Mobile: +92 333 274 6242
>> Skype: live:shoaibzafar_1
>> 
>>  
>> 
>> On Wed, Apr 22, 2020 at 9:40 PM Rehman Murad Ali 
>> mailto:rehman.murad...@venturedive.com>> 
>> wrote:
>> Hello Beam Committers.
>> 
>> Would you please trigger basic tests as well as all validatesRunner test on 
>> this PR: 
>> https://github.com/apache/beam/pull/11154  
>> 
>> Thanks & Regards
>> 
>> 
>> 
>> 
>> 
>> Rehman Murad Ali
>> Software Engineer
>> Mobile: +92 3452076766 
>> Skype: rehman.muradali
>> 
>> 
>> 
>> On Wed, Apr 22, 2020 at 9:25 PM Yoshiki Obata > > wrote:
>> Hello Beam Committers,
>> 
>> I would appreciate if you could trigger precommit checks for these PRs;
>> https://github.com/apache/beam/pull/11493 
>> 
>> https://github.com/apache/beam/pull/11494 
>> 
>> 
>> Regards
>> yoshiki
>> 
>> 2020年4月21日(火) 1:11 Luke Cwik mailto:lc...@google.com>>:
>> The precommits started and I provided the comments for the postcommits as 
>> you have requested but they have yet to start.
>> 
>> On Mon, Apr 20, 2020 at 8:31 AM Shoaib Zafar > > wrote:
>> Hello Beam Committers.
>> 
>> Would you please trigger the pre-commit checks on the PR: 
>> https://github.com/apache/beam/pull/11210 
>>  along with the python 
>> post-commit checks (Run Python PostCommit, Run Python 3.5 PostCommit)?
>> 
>> Thanks! Regards,
>> Shoaib Zafar
>> Software Engineering Lead
>> Mobile: +92 333 274 6242
>> Skype: live:shoaibzafar_1
>> 
>>  
>> 
>> On Fri, Apr 17, 2020 at 1:19 PM Ismaël Mejía > > wrote:
>> done
>> 
>> On Thu, Apr 16, 2020 at 4:32 PM Rehman Murad Ali 
>> mailto:rehman.murad...@venturedive.com>> 
>> wrote:
>> Hello Beam Committers.
>> 
>> Would you please trigger basic tests as well as validatesRunner test on this 
>> PR: 
>> 
>>  https://github.com/apache/beam/pull/11350 
>> 
>> 
>> Thanks & Regards
>> 
>> 
>> 
>> 
>> 
>> Rehman Murad Ali
>> Software Engineer
>> Mobile: +92 3452076766 
>> Skype: rehman.muradali
>> 
>> 
>> 
>> On Mon, Apr 13, 2020 at 10:16 PM Ahmet Altay > > wrote:
>> Done.
>> 
>> On Mon, Apr 13, 2020 at 8:52 AM Shoaib Zafar > > wrote:
>> Hello Beam Committers.
>> 
>> Would you please trigger the pre-commit checks on the PR: 
>> https://github.com/apache/beam/pull/11210 
>>  along with the python 
>> post-commit checks (Run Python PostCommit, Run Python 3.5 PostCommit)?
>> 
>> Thanks!
>> Shoaib Zafar
>> Software Engineering Lead
>> Mobile: +92 333 274 6242
>> Skype: live:shoaibzafar_1
>> 
>>  
>> 
>> On Mon, Apr 13, 2020 at 4:00 PM Ismaël Mejía > > wrote:
>> done
>> 
>> On Mon, Apr 13, 2020 at 12:42 PM Rehman Murad Ali
>> mailto:rehman.murad...@venturedive.com>> 
>> wrote:
>> >
>> > Hi Beam Committers!
>> >
>> > Thanks( Ismael )
>> >
>> > I appreciate if someone could trigger these tests on this PR 
>> > https://github.com/apache/beam/pull/11154 
>> > 
>> >
>> > run dataflow validatesrunner
>> > run flink validatesrunner
>> > Run Java Flink PortableValidatesRunner Streaming
>> >
>> > Thanks
>> >
>> >
>> >
>> > Rehman Murad Ali
>> > Software Engineer
>> > Mobile: +92 3452076766 
>> > Skype: rehman.muradali
>> >
>> >
>> >
>> > On Wed, Apr 1, 2020 at 1:19 PM Ismaël Mejía > > > wrote:
>> >>
>> >> done
>> >>
>

[REVIEW][please pause website changes] Migrated the Beam website to Hugo

[Aizhamal Nurmamat kyzy]

Hello everybody,

We are almost done migrating the Apache Beam website from Jekyll to Hugo.
You can see the PR in [1], and we'd love to hear your feedback/comments on
the PR. It includes  detailed guidelines on contributing to the new
Hugo-based website and adding translations to pages [2]. For those who are
curious about adding new languages, we will provide a proof of concept in
the next couple of days in this thread.

Since we want to move forward with the PR, I would like to ask the
community to hold off changes to the current Beam website for a week, until
we are able to review and merge the PR. Is this acceptable to everyone?

In case anyone missed my previous email with the background for the website
migration, you can find more context here [3].

Thanks,
Aizhamal

[1] https://github.com/apache/beam/pull/11554
[2]
https://github.com/apache/beam/blob/256b7042bf504b94f161ca03b388a2ba247918d9/website/CONTRIBUTE.md
[3]
https://lists.apache.org/thread.html/r7fa6d710c0a1959cce5108e460d71c306ce5756cf96af818b41cb7ca%40%3Cdev.beam.apache.org%3E

Re: Jenkins jobs not running for my PR 10438

[Udi Meiri]

Alexey, what you're doing should be working (commits should trigger tests,
as should "retest this please" and other phrases).

https://issues.apache.org/jira/browse/INFRA-19836 tracks this issue

On Tue, Apr 28, 2020 at 10:04 AM Alexey Romanenko 
wrote:

> Does anyone know the “golden rule” how to trigger Jenkins tests?
>
> For example:
> https://github.com/apache/beam/pull/11341
> I tried several times and it’s still not triggered.
>
> On 28 Apr 2020, at 13:33, Ismaël Mejía  wrote:
>
> done
>
> On Tue, Apr 28, 2020 at 12:47 PM Shoaib Zafar <
> shoaib.za...@venturedive.com> wrote:
>
>> Hello Beam Committers,
>>
>> I would appreciate if you could trigger precommit checks for the PR:
>> https://github.com/apache/beam/pull/11210 along with the python
>> post-commit check (Run Python 3.5 PostCommit).
>>
>> Thanks and Regards.
>>
>> *Shoaib Zafar*
>> Software Engineering Lead
>> Mobile: +92 333 274 6242
>> Skype: live:shoaibzafar_1
>>
>> 
>>
>>
>> On Wed, Apr 22, 2020 at 9:40 PM Rehman Murad Ali <
>> rehman.murad...@venturedive.com> wrote:
>>
>>> Hello Beam Committers.
>>>
>>> Would you please trigger basic tests as well as all *validatesRunner*
>>> test on this PR:
>>> https://github.com/apache/beam/pull/11154 
>>> 
>>>
>>>
>>> *Thanks & Regards*
>>>
>>>
>>>
>>> *Rehman Murad Ali*
>>> Software Engineer
>>> Mobile: +92 3452076766 <+92%20345%202076766>
>>> Skype: rehman.muradali
>>>
>>>
>>> On Wed, Apr 22, 2020 at 9:25 PM Yoshiki Obata 
>>> wrote:
>>>
 Hello Beam Committers,

 I would appreciate if you could trigger precommit checks for these PRs;
 https://github.com/apache/beam/pull/11493
 https://github.com/apache/beam/pull/11494

 Regards
 yoshiki

 2020年4月21日(火) 1:11 Luke Cwik :

> The precommits started and I provided the comments for the postcommits
> as you have requested but they have yet to start.
>
> On Mon, Apr 20, 2020 at 8:31 AM Shoaib Zafar <
> shoaib.za...@venturedive.com> wrote:
>
>> Hello Beam Committers.
>>
>> Would you please trigger the pre-commit checks on the PR:
>> https://github.com/apache/beam/pull/11210 along with the python
>> post-commit checks (Run Python PostCommit, Run Python 3.5 PostCommit)?
>>
>> Thanks! Regards,
>>
>> *Shoaib Zafar*
>> Software Engineering Lead
>> Mobile: +92 333 274 6242
>> Skype: live:shoaibzafar_1
>>
>> 
>>
>>
>> On Fri, Apr 17, 2020 at 1:19 PM Ismaël Mejía 
>> wrote:
>>
>>> done
>>>
>>> On Thu, Apr 16, 2020 at 4:32 PM Rehman Murad Ali <
>>> rehman.murad...@venturedive.com> wrote:
>>>
 Hello Beam Committers.

 Would you please trigger basic tests as well as validatesRunner
 test on this PR:

 
 https://github.com/apache/beam/pull/11350


 *Thanks & Regards*



 *Rehman Murad Ali*
 Software Engineer
 Mobile: +92 3452076766 <+92%20345%202076766>
 Skype: rehman.muradali


 On Mon, Apr 13, 2020 at 10:16 PM Ahmet Altay 
 wrote:

> Done.
>
> On Mon, Apr 13, 2020 at 8:52 AM Shoaib Zafar <
> shoaib.za...@venturedive.com> wrote:
>
>> Hello Beam Committers.
>>
>> Would you please trigger the pre-commit checks on the PR:
>> https://github.com/apache/beam/pull/11210 along with the python
>> post-commit checks (Run Python PostCommit, Run Python 3.5 
>> PostCommit)?
>>
>> Thanks!
>>
>> *Shoaib Zafar*
>> Software Engineering Lead
>> Mobile: +92 333 274 6242
>> Skype: live:shoaibzafar_1
>>
>> 
>>
>>
>> On Mon, Apr 13, 2020 at 4:00 PM Ismaël Mejía 
>> wrote:
>>
>>> done
>>>
>>> On Mon, Apr 13, 2020 at 12:42 PM Rehman Murad Ali
>>>  wrote:
>>> >
>>> > Hi Beam Committers!
>>> >
>>> > Thanks( Ismael )
>>> >
>>> > I appreciate if someone could trigger these tests on this PR
>>> https://github.com/apache/beam/pull/11154
>>> >
>>> > run dataflow validatesrunner
>>> > run flink validatesrunner
>>> > Run Java Flink PortableValidatesRunner Streaming
>>> >
>>> > Thanks
>>> >
>>> >
>>> >
>>> > Rehman Murad Ali
>>> > Software Engineer
>>> > Mobile: +92 3452076766 <+92%20345%202076766>
>>> > Skype: rehman.muradali
>>> >
>>> >
>>> >
>>> > On Wed, Apr 1, 2020 at 1:19 PM Ismaël Mejía 
>>> wrote:
>>> >>
>>> >> done
>>> >>
>>> >> On Wed, Ap

Re: Jenkins jobs not running for my PR 10438

[Alexey Romanenko]

Does anyone know the “golden rule” how to trigger Jenkins tests?

For example: 
https://github.com/apache/beam/pull/11341 

I tried several times and it’s still not triggered. 

> On 28 Apr 2020, at 13:33, Ismaël Mejía  wrote:
> 
> done
> 
> On Tue, Apr 28, 2020 at 12:47 PM Shoaib Zafar  > wrote:
> Hello Beam Committers,
> 
> I would appreciate if you could trigger precommit checks for the PR: 
> https://github.com/apache/beam/pull/11210 
>  along with the python post-commit 
> check (Run Python 3.5 PostCommit).
> 
> Thanks and Regards.
> Shoaib Zafar
> Software Engineering Lead
> Mobile: +92 333 274 6242
> Skype: live:shoaibzafar_1
> 
>  
> 
> On Wed, Apr 22, 2020 at 9:40 PM Rehman Murad Ali 
> mailto:rehman.murad...@venturedive.com>> 
> wrote:
> Hello Beam Committers.
> 
> Would you please trigger basic tests as well as all validatesRunner test on 
> this PR: 
> https://github.com/apache/beam/pull/11154  
> 
> Thanks & Regards
> 
> 
> 
> 
> 
> Rehman Murad Ali
> Software Engineer
> Mobile: +92 3452076766
> Skype: rehman.muradali
> 
> 
> 
> On Wed, Apr 22, 2020 at 9:25 PM Yoshiki Obata  > wrote:
> Hello Beam Committers,
> 
> I would appreciate if you could trigger precommit checks for these PRs;
> https://github.com/apache/beam/pull/11493 
> 
> https://github.com/apache/beam/pull/11494 
> 
> 
> Regards
> yoshiki
> 
> 2020年4月21日(火) 1:11 Luke Cwik mailto:lc...@google.com>>:
> The precommits started and I provided the comments for the postcommits as you 
> have requested but they have yet to start.
> 
> On Mon, Apr 20, 2020 at 8:31 AM Shoaib Zafar  > wrote:
> Hello Beam Committers.
> 
> Would you please trigger the pre-commit checks on the PR: 
> https://github.com/apache/beam/pull/11210 
>  along with the python post-commit 
> checks (Run Python PostCommit, Run Python 3.5 PostCommit)?
> 
> Thanks! Regards,
> Shoaib Zafar
> Software Engineering Lead
> Mobile: +92 333 274 6242
> Skype: live:shoaibzafar_1
> 
>  
> 
> On Fri, Apr 17, 2020 at 1:19 PM Ismaël Mejía  > wrote:
> done
> 
> On Thu, Apr 16, 2020 at 4:32 PM Rehman Murad Ali 
> mailto:rehman.murad...@venturedive.com>> 
> wrote:
> Hello Beam Committers.
> 
> Would you please trigger basic tests as well as validatesRunner test on this 
> PR: 
> 
>  https://github.com/apache/beam/pull/11350 
> 
> 
> Thanks & Regards
> 
> 
> 
> 
> 
> Rehman Murad Ali
> Software Engineer
> Mobile: +92 3452076766 
> Skype: rehman.muradali
> 
> 
> 
> On Mon, Apr 13, 2020 at 10:16 PM Ahmet Altay  > wrote:
> Done.
> 
> On Mon, Apr 13, 2020 at 8:52 AM Shoaib Zafar  > wrote:
> Hello Beam Committers.
> 
> Would you please trigger the pre-commit checks on the PR: 
> https://github.com/apache/beam/pull/11210 
>  along with the python post-commit 
> checks (Run Python PostCommit, Run Python 3.5 PostCommit)?
> 
> Thanks!
> Shoaib Zafar
> Software Engineering Lead
> Mobile: +92 333 274 6242
> Skype: live:shoaibzafar_1
> 
>  
> 
> On Mon, Apr 13, 2020 at 4:00 PM Ismaël Mejía  > wrote:
> done
> 
> On Mon, Apr 13, 2020 at 12:42 PM Rehman Murad Ali
> mailto:rehman.murad...@venturedive.com>> 
> wrote:
> >
> > Hi Beam Committers!
> >
> > Thanks( Ismael )
> >
> > I appreciate if someone could trigger these tests on this PR 
> > https://github.com/apache/beam/pull/11154 
> > 
> >
> > run dataflow validatesrunner
> > run flink validatesrunner
> > Run Java Flink PortableValidatesRunner Streaming
> >
> > Thanks
> >
> >
> >
> > Rehman Murad Ali
> > Software Engineer
> > Mobile: +92 3452076766 
> > Skype: rehman.muradali
> >
> >
> >
> > On Wed, Apr 1, 2020 at 1:19 PM Ismaël Mejía  > > wrote:
> >>
> >> done
> >>
> >> On Wed, Apr 1, 2020 at 9:15 AM Rehman Murad Ali
> >> mailto:rehman.murad...@venturedive.com>> 
> >> wrote:
> >> >
> >> > Hi Beam Committers!
> >> >
> >> > I appreciate if someone could trigger the tests on this PR 
> >> > https://github.com/apache/beam/pull/11154 
> >> > 
> >> >
> >> > Thanks
> >> >
> >> >
> >> >
> >> > Rehman Murad Ali
> >> > Software Engineer
> >> > Mobile: +92 3452076766 
> >> > Skype: rehman.muradali
> >> >
> >> >
> >> >
> >> > On Mon, Mar 30, 2020 at 8:09 PM Kamil Wasilewski 
> >> > mailto:kamil.wasilew...@polidea.com>> 
> >> > wrote:
> >> >>
> >> >> Done.
> >> >>
> >> >> On Mon, M

Re: Support for Avro 1.9.X

[Brian Hulette]

There's no way to do this now. There was an attempt to upgrade to 1.9.1
here [1]. The hangup was just that the default datetime type was switched
to java time (from joda time). My memory is we could get that PR working
just by setting some configuration.. somewhere.. to make sure we use joda
time for Avro in Beam.

That approach limits us to 1.9 though as joda support is dropped in 1.10
[2]. Also you note that 1.9.x handles date fields much better - is that
*because* it uses java time by default?

Brian

[1] https://github.com/apache/beam/pull/9779
[2] https://github.com/apache/beam/pull/9779#discussion_r334400709

On Tue, Apr 28, 2020 at 9:34 AM Parth Desai  wrote:

> Hi,
>
> The latest version of beam currently only supports Avro 1.8.2, however, my
> customer (Charles Schwab) is heavily using Avro 1.9.x which handles date
> fields much better. Is there any way they can upgrade from 1.8.2 to 1.9.x
> in Dataflow/beam?
>
> Thanks,
>
>
>
> Parth Desai
>
> Customer Engineer, Data Analytics Specialist - Google Cloud
>
> 500 W 2nd St, Austin, TX 78701
>
> Mobile: 404-637-6674 <(404)%20637-6674>
>
>
>
>

Support for Avro 1.9.X

[Parth Desai]

Hi,

The latest version of beam currently only supports Avro 1.8.2, however, my
customer (Charles Schwab) is heavily using Avro 1.9.x which handles date
fields much better. Is there any way they can upgrade from 1.8.2 to 1.9.x
in Dataflow/beam?

Thanks,



Parth Desai

Customer Engineer, Data Analytics Specialist - Google Cloud

500 W 2nd St, Austin, TX 78701

Mobile: 404-637-6674

Re: Jenkins jobs not running for my PR 10438

[Ismaël Mejía]

done

On Tue, Apr 28, 2020 at 12:47 PM Shoaib Zafar 
wrote:

> Hello Beam Committers,
>
> I would appreciate if you could trigger precommit checks for the PR:
> https://github.com/apache/beam/pull/11210 along with the python
> post-commit check (Run Python 3.5 PostCommit).
>
> Thanks and Regards.
>
> *Shoaib Zafar*
> Software Engineering Lead
> Mobile: +92 333 274 6242
> Skype: live:shoaibzafar_1
>
> 
>
>
> On Wed, Apr 22, 2020 at 9:40 PM Rehman Murad Ali <
> rehman.murad...@venturedive.com> wrote:
>
>> Hello Beam Committers.
>>
>> Would you please trigger basic tests as well as all *validatesRunner*
>> test on this PR:
>> https://github.com/apache/beam/pull/11154 
>> 
>>
>>
>> *Thanks & Regards*
>>
>>
>>
>> *Rehman Murad Ali*
>> Software Engineer
>> Mobile: +92 3452076766
>> Skype: rehman.muradali
>>
>>
>> On Wed, Apr 22, 2020 at 9:25 PM Yoshiki Obata 
>> wrote:
>>
>>> Hello Beam Committers,
>>>
>>> I would appreciate if you could trigger precommit checks for these PRs;
>>> https://github.com/apache/beam/pull/11493
>>> https://github.com/apache/beam/pull/11494
>>>
>>> Regards
>>> yoshiki
>>>
>>> 2020年4月21日(火) 1:11 Luke Cwik :
>>>
 The precommits started and I provided the comments for the postcommits
 as you have requested but they have yet to start.

 On Mon, Apr 20, 2020 at 8:31 AM Shoaib Zafar <
 shoaib.za...@venturedive.com> wrote:

> Hello Beam Committers.
>
> Would you please trigger the pre-commit checks on the PR:
> https://github.com/apache/beam/pull/11210 along with the python
> post-commit checks (Run Python PostCommit, Run Python 3.5 PostCommit)?
>
> Thanks! Regards,
>
> *Shoaib Zafar*
> Software Engineering Lead
> Mobile: +92 333 274 6242
> Skype: live:shoaibzafar_1
>
> 
>
>
> On Fri, Apr 17, 2020 at 1:19 PM Ismaël Mejía 
> wrote:
>
>> done
>>
>> On Thu, Apr 16, 2020 at 4:32 PM Rehman Murad Ali <
>> rehman.murad...@venturedive.com> wrote:
>>
>>> Hello Beam Committers.
>>>
>>> Would you please trigger basic tests as well as validatesRunner test
>>> on this PR:
>>>
>>> 
>>> https://github.com/apache/beam/pull/11350
>>>
>>>
>>> *Thanks & Regards*
>>>
>>>
>>>
>>> *Rehman Murad Ali*
>>> Software Engineer
>>> Mobile: +92 3452076766 <+92%20345%202076766>
>>> Skype: rehman.muradali
>>>
>>>
>>> On Mon, Apr 13, 2020 at 10:16 PM Ahmet Altay 
>>> wrote:
>>>
 Done.

 On Mon, Apr 13, 2020 at 8:52 AM Shoaib Zafar <
 shoaib.za...@venturedive.com> wrote:

> Hello Beam Committers.
>
> Would you please trigger the pre-commit checks on the PR:
> https://github.com/apache/beam/pull/11210 along with the python
> post-commit checks (Run Python PostCommit, Run Python 3.5 PostCommit)?
>
> Thanks!
>
> *Shoaib Zafar*
> Software Engineering Lead
> Mobile: +92 333 274 6242
> Skype: live:shoaibzafar_1
>
> 
>
>
> On Mon, Apr 13, 2020 at 4:00 PM Ismaël Mejía 
> wrote:
>
>> done
>>
>> On Mon, Apr 13, 2020 at 12:42 PM Rehman Murad Ali
>>  wrote:
>> >
>> > Hi Beam Committers!
>> >
>> > Thanks( Ismael )
>> >
>> > I appreciate if someone could trigger these tests on this PR
>> https://github.com/apache/beam/pull/11154
>> >
>> > run dataflow validatesrunner
>> > run flink validatesrunner
>> > Run Java Flink PortableValidatesRunner Streaming
>> >
>> > Thanks
>> >
>> >
>> >
>> > Rehman Murad Ali
>> > Software Engineer
>> > Mobile: +92 3452076766 <+92%20345%202076766>
>> > Skype: rehman.muradali
>> >
>> >
>> >
>> > On Wed, Apr 1, 2020 at 1:19 PM Ismaël Mejía 
>> wrote:
>> >>
>> >> done
>> >>
>> >> On Wed, Apr 1, 2020 at 9:15 AM Rehman Murad Ali
>> >>  wrote:
>> >> >
>> >> > Hi Beam Committers!
>> >> >
>> >> > I appreciate if someone could trigger the tests on this PR
>> https://github.com/apache/beam/pull/11154
>> >> >
>> >> > Thanks
>> >> >
>> >> >
>> >> >
>> >> > Rehman Murad Ali
>> >> > Software Engineer
>> >> > Mobile: +92 3452076766 <+92%20345%202076766>
>> >> > Skype: rehman.muradali
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Mar 30, 2020 at 8:09 PM Kamil Wasilewski <
>> kamil.wasilew...@polidea.com> wrote:
>> >> >>
>> >> >> Done.
>> >

Re: Jenkins jobs not running for my PR 10438

[Shoaib Zafar]

Hello Beam Committers,

I would appreciate if you could trigger precommit checks for the PR:
https://github.com/apache/beam/pull/11210 along with the python post-commit
check (Run Python 3.5 PostCommit).

Thanks and Regards.

*Shoaib Zafar*
Software Engineering Lead
Mobile: +92 333 274 6242
Skype: live:shoaibzafar_1




On Wed, Apr 22, 2020 at 9:40 PM Rehman Murad Ali <
rehman.murad...@venturedive.com> wrote:

> Hello Beam Committers.
>
> Would you please trigger basic tests as well as all *validatesRunner*
> test on this PR:
> https://github.com/apache/beam/pull/11154 
> 
>
>
> *Thanks & Regards*
>
>
>
> *Rehman Murad Ali*
> Software Engineer
> Mobile: +92 3452076766
> Skype: rehman.muradali
>
>
> On Wed, Apr 22, 2020 at 9:25 PM Yoshiki Obata 
> wrote:
>
>> Hello Beam Committers,
>>
>> I would appreciate if you could trigger precommit checks for these PRs;
>> https://github.com/apache/beam/pull/11493
>> https://github.com/apache/beam/pull/11494
>>
>> Regards
>> yoshiki
>>
>> 2020年4月21日(火) 1:11 Luke Cwik :
>>
>>> The precommits started and I provided the comments for the postcommits
>>> as you have requested but they have yet to start.
>>>
>>> On Mon, Apr 20, 2020 at 8:31 AM Shoaib Zafar <
>>> shoaib.za...@venturedive.com> wrote:
>>>
 Hello Beam Committers.

 Would you please trigger the pre-commit checks on the PR:
 https://github.com/apache/beam/pull/11210 along with the python
 post-commit checks (Run Python PostCommit, Run Python 3.5 PostCommit)?

 Thanks! Regards,

 *Shoaib Zafar*
 Software Engineering Lead
 Mobile: +92 333 274 6242
 Skype: live:shoaibzafar_1

 


 On Fri, Apr 17, 2020 at 1:19 PM Ismaël Mejía  wrote:

> done
>
> On Thu, Apr 16, 2020 at 4:32 PM Rehman Murad Ali <
> rehman.murad...@venturedive.com> wrote:
>
>> Hello Beam Committers.
>>
>> Would you please trigger basic tests as well as validatesRunner test
>> on this PR:
>>
>> 
>> https://github.com/apache/beam/pull/11350
>>
>>
>> *Thanks & Regards*
>>
>>
>>
>> *Rehman Murad Ali*
>> Software Engineer
>> Mobile: +92 3452076766 <+92%20345%202076766>
>> Skype: rehman.muradali
>>
>>
>> On Mon, Apr 13, 2020 at 10:16 PM Ahmet Altay 
>> wrote:
>>
>>> Done.
>>>
>>> On Mon, Apr 13, 2020 at 8:52 AM Shoaib Zafar <
>>> shoaib.za...@venturedive.com> wrote:
>>>
 Hello Beam Committers.

 Would you please trigger the pre-commit checks on the PR:
 https://github.com/apache/beam/pull/11210 along with the python
 post-commit checks (Run Python PostCommit, Run Python 3.5 PostCommit)?

 Thanks!

 *Shoaib Zafar*
 Software Engineering Lead
 Mobile: +92 333 274 6242
 Skype: live:shoaibzafar_1

 


 On Mon, Apr 13, 2020 at 4:00 PM Ismaël Mejía 
 wrote:

> done
>
> On Mon, Apr 13, 2020 at 12:42 PM Rehman Murad Ali
>  wrote:
> >
> > Hi Beam Committers!
> >
> > Thanks( Ismael )
> >
> > I appreciate if someone could trigger these tests on this PR
> https://github.com/apache/beam/pull/11154
> >
> > run dataflow validatesrunner
> > run flink validatesrunner
> > Run Java Flink PortableValidatesRunner Streaming
> >
> > Thanks
> >
> >
> >
> > Rehman Murad Ali
> > Software Engineer
> > Mobile: +92 3452076766 <+92%20345%202076766>
> > Skype: rehman.muradali
> >
> >
> >
> > On Wed, Apr 1, 2020 at 1:19 PM Ismaël Mejía 
> wrote:
> >>
> >> done
> >>
> >> On Wed, Apr 1, 2020 at 9:15 AM Rehman Murad Ali
> >>  wrote:
> >> >
> >> > Hi Beam Committers!
> >> >
> >> > I appreciate if someone could trigger the tests on this PR
> https://github.com/apache/beam/pull/11154
> >> >
> >> > Thanks
> >> >
> >> >
> >> >
> >> > Rehman Murad Ali
> >> > Software Engineer
> >> > Mobile: +92 3452076766 <+92%20345%202076766>
> >> > Skype: rehman.muradali
> >> >
> >> >
> >> >
> >> > On Mon, Mar 30, 2020 at 8:09 PM Kamil Wasilewski <
> kamil.wasilew...@polidea.com> wrote:
> >> >>
> >> >> Done.
> >> >>
> >> >> On Mon, Mar 30, 2020 at 4:58 PM Tomo Suzuki <
> suzt...@google.com> wrote:
> >> >>>
> >> >>> Hi Beam committers,
> >> >>> (Thanks Ahmet)
> >> >>>
> >> >>> Would you retrigger the following 5 j

Re: JIRA Committer Permissions

[Maximilian Michels]

FWIW committer and contributor roles in Beam's JIRA have practically
identical permissions[1]. The only different is the "Set Issue Security"
and "Manage Watchers" permissions, both of which I have never used before.

Thanks for updating the wiki page!

-Max

[1]
https://jira.apache.org/jira/plugins/servlet/project-config/BEAM/permissions

On 28.04.20 06:09, Kenneth Knowles wrote:
> I think it would be very valuable to have a committer onboarding guide,
> with info for both the committer and steps for PMC to take. I think the
> wiki is the right place for it...
> 
> (two seconds of checking later)
> 
> It
> exists! 
> https://cwiki.apache.org/confluence/display/BEAM/Committer+onboarding+guide
> 
> By the time you read this, I hope to have added the links that Luke
> suggested. We do need to remember to send this guide to new committers.
> And potentially announce changes that existing committers may not have
> followed.
> 
> Kenn
> 
> On Mon, Apr 27, 2020 at 7:23 PM Luke Cwik  > wrote:
> 
> The Beam committer guide is about reviewing code and the "become a
> committer" is more about what we look for and not the process.
> 
> Since this is common for all ASF projects, I suspect an ASF page may
> have this documented or should be updated to have this covered as
> well but didn't see it on the ASF new committers resources page[1]
> or on the developers & contributors overview[2].
> 
> 1: https://www.apache.org/dev/new-committers-guide.html
> 2: https://www.apache.org/dev/index.html
> 
> On Mon, Apr 27, 2020 at 3:32 PM Udi Meiri  > wrote:
> 
> Should this step be added to our new committer guide?
> 
> On Fri, Apr 24, 2020 at 6:21 PM Luke Cwik  > wrote:
> 
> I noticed that several committers only had contributor level
> permissions and I went and updated your account permissions
> for the Beam project to be committer level. Feel free to let
> me know If you run into any issues.
> 
> There were about ~25 accounts like this.
>