Gradle status

[Romain Manni-Bucau]

Hi guys,

Do you plan to solve gradle issue (even dropping gradle from beam source),
issue I hit is:

1. gradle build is not equivalent to maven one - met some shadowed
dependencies which shouldnt be shaded like junit in some modules
2. gradle build doesn't use the same output directory than maven so it is
not really smooth to have both and have to maintain both
3. There are a lot of PR to flush before being able to switch

Any action plan taken to fix that and remove this ambiguity? Once again, I
would indeed prefer to stay on Maven but I'm fine to move to gradle,
however I'd like to stop having both builds and not really know what to use
or loose time with some differences between both.

Would the plan be to either do a PR merge effort and then gradle effort or
drop it - that are the 2 options I see to exit that state?

wdyt?

Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn 

Re: Gradle status

[Kenneth Knowles]

The phase we are in is: "Once Gradle is able to replace Maven in a specific
process (or portion thereof), Maven will no longer be maintained for said
process (or portion thereof) and will be removed."

Once the Gradle presubmit can replace a particular Maven presubmit, we can
remove the Maven version.

Can you file JIRAs for the things you suspect are shaded incorrectly by our
gradle configurations?

Kenn

On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau 
wrote:

> Hi guys,
>
> Do you plan to solve gradle issue (even dropping gradle from beam source),
> issue I hit is:
>
> 1. gradle build is not equivalent to maven one - met some shadowed
> dependencies which shouldnt be shaded like junit in some modules
> 2. gradle build doesn't use the same output directory than maven so it is
> not really smooth to have both and have to maintain both
>
3. There are a lot of PR to flush before being able to switch
>
> Any action plan taken to fix that and remove this ambiguity? Once again, I
> would indeed prefer to stay on Maven but I'm fine to move to gradle,
> however I'd like to stop having both builds and not really know what to use
> or loose time with some differences between both.
>
> Would the plan be to either do a PR merge effort and then gradle effort or
> drop it - that are the 2 options I see to exit that state?
>
> wdyt?
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
> 
>

Re: Gradle status

[Lukasz Cwik]

The top level JIRA is here: https://issues.apache.org/jira/browse/BEAM-3249

On Thu, Jan 11, 2018 at 9:56 AM, Kenneth Knowles  wrote:

> The phase we are in is: "Once Gradle is able to replace Maven in a
> specific process (or portion thereof), Maven will no longer be maintained
> for said process (or portion thereof) and will be removed."
>
> Once the Gradle presubmit can replace a particular Maven presubmit, we can
> remove the Maven version.
>
> Can you file JIRAs for the things you suspect are shaded incorrectly by
> our gradle configurations?
>
> Kenn
>
> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau  > wrote:
>
>> Hi guys,
>>
>> Do you plan to solve gradle issue (even dropping gradle from beam
>> source), issue I hit is:
>>
>> 1. gradle build is not equivalent to maven one - met some shadowed
>> dependencies which shouldnt be shaded like junit in some modules
>> 2. gradle build doesn't use the same output directory than maven so it is
>> not really smooth to have both and have to maintain both
>>
> 3. There are a lot of PR to flush before being able to switch
>>
>> Any action plan taken to fix that and remove this ambiguity? Once again,
>> I would indeed prefer to stay on Maven but I'm fine to move to gradle,
>> however I'd like to stop having both builds and not really know what to use
>> or loose time with some differences between both.
>>
>> Would the plan be to either do a PR merge effort and then gradle effort
>> or drop it - that are the 2 options I see to exit that state?
>>
>> wdyt?
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>> 
>>
>
>

Re: Gradle status

[Jean-Baptiste Onofré]

Hi,

I think we are still in the "polishing" part where Gradle is not yet fully 
equivalent to Maven. When Maven and Gradle will be aligned, then, we will be 
able to remove one.


I agree that today Gradle is not yet stable/equivalent to Maven. However, that's 
why we have to create the corresponding Jira when we identify issues/gaps.


You are right that we have to move forward on this in a more decent and 
transparent way. I'm suggesting to start a kind of "build campaign" to clearly 
identify the differences and create the corresponding Jira.


Thanks for bringing this on the table !
Regards
JB

On 01/11/2018 05:43 PM, Romain Manni-Bucau wrote:

Hi guys,

Do you plan to solve gradle issue (even dropping gradle from beam source), issue 
I hit is:


1. gradle build is not equivalent to maven one - met some shadowed dependencies 
which shouldnt be shaded like junit in some modules
2. gradle build doesn't use the same output directory than maven so it is not 
really smooth to have both and have to maintain both

3. There are a lot of PR to flush before being able to switch

Any action plan taken to fix that and remove this ambiguity? Once again, I would 
indeed prefer to stay on Maven but I'm fine to move to gradle, however I'd like 
to stop having both builds and not really know what to use or loose time with 
some differences between both.


Would the plan be to either do a PR merge effort and then gradle effort or drop 
it - that are the 2 options I see to exit that state?


wdyt?
 
Romain Manni-Bucau
@rmannibucau  | Blog 
 | Old Blog 
 | Github  | 
LinkedIn 


--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Gradle status

[Jean-Baptiste Onofré]

Hi Luke,

Great: let's eventually add new sub-tasks in this Jira !

Thanks !
Regards
JB

On 01/11/2018 07:02 PM, Lukasz Cwik wrote:

The top level JIRA is here: https://issues.apache.org/jira/browse/BEAM-3249

On Thu, Jan 11, 2018 at 9:56 AM, Kenneth Knowles > wrote:


The phase we are in is: "Once Gradle is able to replace Maven in a specific
process (or portion thereof), Maven will no longer be maintained for said
process (or portion thereof) and will be removed."

Once the Gradle presubmit can replace a particular Maven presubmit, we can
remove the Maven version.

Can you file JIRAs for the things you suspect are shaded incorrectly by our
gradle configurations?

Kenn

On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau mailto:rmannibu...@gmail.com>> wrote:

Hi guys,

Do you plan to solve gradle issue (even dropping gradle from beam
source), issue I hit is:

1. gradle build is not equivalent to maven one - met some shadowed
dependencies which shouldnt be shaded like junit in some modules
2. gradle build doesn't use the same output directory than maven so it
is not really smooth to have both and have to maintain both

3. There are a lot of PR to flush before being able to switch

Any action plan taken to fix that and remove this ambiguity? Once again,
I would indeed prefer to stay on Maven but I'm fine to move to gradle,
however I'd like to stop having both builds and not really know what to
use or loose time with some differences between both.

Would the plan be to either do a PR merge effort and then gradle effort
or drop it - that are the 2 options I see to exit that state?

wdyt?

Romain Manni-Bucau
@rmannibucau  | Blog
 | Old Blog
 | Github
 | LinkedIn






--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Gradle status

[Romain Manni-Bucau]

Ok

Will try to create task from now on when seeing them

Can we also put a kind of timeout for the switch to avoid an in between
state lasting 6 months? Something like april?

Le 11 janv. 2018 19:05, "Jean-Baptiste Onofré"  a écrit :

> Hi Luke,
>
> Great: let's eventually add new sub-tasks in this Jira !
>
> Thanks !
> Regards
> JB
>
> On 01/11/2018 07:02 PM, Lukasz Cwik wrote:
>
>> The top level JIRA is here: https://issues.apache.org/jira
>> /browse/BEAM-3249
>>
>> On Thu, Jan 11, 2018 at 9:56 AM, Kenneth Knowles > k...@google.com>> wrote:
>>
>> The phase we are in is: "Once Gradle is able to replace Maven in a
>> specific
>> process (or portion thereof), Maven will no longer be maintained for
>> said
>> process (or portion thereof) and will be removed."
>>
>> Once the Gradle presubmit can replace a particular Maven presubmit,
>> we can
>> remove the Maven version.
>>
>> Can you file JIRAs for the things you suspect are shaded incorrectly
>> by our
>> gradle configurations?
>>
>> Kenn
>>
>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com
>> > wrote:
>>
>> Hi guys,
>>
>> Do you plan to solve gradle issue (even dropping gradle from beam
>> source), issue I hit is:
>>
>> 1. gradle build is not equivalent to maven one - met some shadowed
>> dependencies which shouldnt be shaded like junit in some modules
>> 2. gradle build doesn't use the same output directory than maven
>> so it
>> is not really smooth to have both and have to maintain both
>>
>> 3. There are a lot of PR to flush before being able to switch
>>
>> Any action plan taken to fix that and remove this ambiguity? Once
>> again,
>> I would indeed prefer to stay on Maven but I'm fine to move to
>> gradle,
>> however I'd like to stop having both builds and not really know
>> what to
>> use or loose time with some differences between both.
>>
>> Would the plan be to either do a PR merge effort and then gradle
>> effort
>> or drop it - that are the 2 options I see to exit that state?
>>
>> wdyt?
>>
>> Romain Manni-Bucau
>> @rmannibucau  | Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>> 
>>
>>
>>
>>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Gradle status

[Lukasz Cwik]

A process vote can be happen if the in-between state is too painful to
maintain.

On Thu, Jan 11, 2018 at 12:11 PM, Romain Manni-Bucau 
wrote:

> Ok
>
> Will try to create task from now on when seeing them
>
> Can we also put a kind of timeout for the switch to avoid an in between
> state lasting 6 months? Something like april?
>
> Le 11 janv. 2018 19:05, "Jean-Baptiste Onofré"  a écrit :
>
>> Hi Luke,
>>
>> Great: let's eventually add new sub-tasks in this Jira !
>>
>> Thanks !
>> Regards
>> JB
>>
>> On 01/11/2018 07:02 PM, Lukasz Cwik wrote:
>>
>>> The top level JIRA is here: https://issues.apache.org/jira
>>> /browse/BEAM-3249
>>>
>>> On Thu, Jan 11, 2018 at 9:56 AM, Kenneth Knowles >> > wrote:
>>>
>>> The phase we are in is: "Once Gradle is able to replace Maven in a
>>> specific
>>> process (or portion thereof), Maven will no longer be maintained for
>>> said
>>> process (or portion thereof) and will be removed."
>>>
>>> Once the Gradle presubmit can replace a particular Maven presubmit,
>>> we can
>>> remove the Maven version.
>>>
>>> Can you file JIRAs for the things you suspect are shaded incorrectly
>>> by our
>>> gradle configurations?
>>>
>>> Kenn
>>>
>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com
>>> > wrote:
>>>
>>> Hi guys,
>>>
>>> Do you plan to solve gradle issue (even dropping gradle from beam
>>> source), issue I hit is:
>>>
>>> 1. gradle build is not equivalent to maven one - met some
>>> shadowed
>>> dependencies which shouldnt be shaded like junit in some modules
>>> 2. gradle build doesn't use the same output directory than maven
>>> so it
>>> is not really smooth to have both and have to maintain both
>>>
>>> 3. There are a lot of PR to flush before being able to switch
>>>
>>> Any action plan taken to fix that and remove this ambiguity?
>>> Once again,
>>> I would indeed prefer to stay on Maven but I'm fine to move to
>>> gradle,
>>> however I'd like to stop having both builds and not really know
>>> what to
>>> use or loose time with some differences between both.
>>>
>>> Would the plan be to either do a PR merge effort and then gradle
>>> effort
>>> or drop it - that are the 2 options I see to exit that state?
>>>
>>> wdyt?
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau  | Blog
>>>  | Old Blog
>>>  | Github
>>>  | LinkedIn
>>> 
>>>
>>>
>>>
>>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>

Re: Gradle status

[Kenneth Knowles]

On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau 
wrote:

> 2. gradle build doesn't use the same output directory than maven so it is
> not really smooth to have both and have to maintain both
>

I also have an opinion on this. It is useful and reasonable to be able to
build even when the source is on a read-only filesystem. Maven's defaults
are undesirable and require workarounds. We shouldn't mimic the behavior,
but actually should set gradle up to build to a directory outside the
source tree always, if we can.

Kenn

Re: Gradle status

[Romain Manni-Bucau]

Le 11 janv. 2018 23:13, "Kenneth Knowles"  a écrit :

On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau 
wrote:

> 2. gradle build doesn't use the same output directory than maven so it is
> not really smooth to have both and have to maintain both
>

I also have an opinion on this. It is useful and reasonable to be able to
build even when the source is on a read-only filesystem. Maven's defaults
are undesirable and require workarounds. We shouldn't mimic the behavior,
but actually should set gradle up to build to a directory outside the
source tree always, if we can.


Hmm, which is something you can do with maven as well so not sure I get it.

Also note the thread is no more about the technical points but more the
sources maintenance and consistency.


Kenn

Re: Gradle status

[Romain Manni-Bucau]

Up,

We discussed to have a strong switch to gradle or rollback to maven around
april to not be blocked by the build tool. I noticed gradle build rarely
passes on PR and kind of blurry our vision - not sure why exactly. Also, PR
don't always contain the gradle updates - generally dependencies+plugins
are added in pom.xml AFAIK, so it seems the adoption is very slow to not
say rejected.

What do we do about that? When do we drop the double build maintenance -
whatever is picked?



Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book


2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau :

>
>
> Le 11 janv. 2018 23:13, "Kenneth Knowles"  a écrit :
>
> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau  > wrote:
>
>> 2. gradle build doesn't use the same output directory than maven so it is
>> not really smooth to have both and have to maintain both
>>
>
> I also have an opinion on this. It is useful and reasonable to be able to
> build even when the source is on a read-only filesystem. Maven's defaults
> are undesirable and require workarounds. We shouldn't mimic the behavior,
> but actually should set gradle up to build to a directory outside the
> source tree always, if we can.
>
>
> Hmm, which is something you can do with maven as well so not sure I get
> it.
>
> Also note the thread is no more about the technical points but more the
> sources maintenance and consistency.
>
>
> Kenn
>
>
>

Re: Gradle status

[Lukasz Cwik]

Thanks for bringing this up Romain but I believe your data points on pass
rates are only partially correct.

In the past week the Java Gradle precommit passed 46.34% of the time
compared to the Java Maven precommit which passed 46.15% of the time. When
I looked at these numbers in mid January they were around 37% so there has
been some improvement. Regardless of the build tool it seems that our pass
rates aren't stellar for the Java build and are causing the community to
not follow best practices (wait for precommits to be green before merging).
I know that on the website we used the mergebot to ensure that things
passed before they were merged, should we institute this on the master
branch or are their any other ideas?

As a side note we had achieved the goals we set out to not need to maintain
the Maven precommit and have authored the first PR to drop the Maven
precommit:  https://github.com/apache/beam/pull/4814


On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau 
wrote:

> Up,
>
> We discussed to have a strong switch to gradle or rollback to maven around
> april to not be blocked by the build tool. I noticed gradle build rarely
> passes on PR and kind of blurry our vision - not sure why exactly. Also, PR
> don't always contain the gradle updates - generally dependencies+plugins
> are added in pom.xml AFAIK, so it seems the adoption is very slow to not
> say rejected.
>
> What do we do about that? When do we drop the double build maintenance -
> whatever is picked?
>
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau :
>
>>
>>
>> Le 11 janv. 2018 23:13, "Kenneth Knowles"  a écrit :
>>
>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> 2. gradle build doesn't use the same output directory than maven so it
>>> is not really smooth to have both and have to maintain both
>>>
>>
>> I also have an opinion on this. It is useful and reasonable to be able to
>> build even when the source is on a read-only filesystem. Maven's defaults
>> are undesirable and require workarounds. We shouldn't mimic the behavior,
>> but actually should set gradle up to build to a directory outside the
>> source tree always, if we can.
>>
>>
>> Hmm, which is something you can do with maven as well so not sure I get
>> it.
>>
>> Also note the thread is no more about the technical points but more the
>> sources maintenance and consistency.
>>
>>
>> Kenn
>>
>>
>>
>

Re: Gradle status

[Romain Manni-Bucau]

Le 7 mars 2018 17:34, "Lukasz Cwik"  a écrit :

Thanks for bringing this up Romain but I believe your data points on pass
rates are only partially correct.


Sure sure, it is mainly about my own PR which a very small % of the whole
project ;).



In the past week the Java Gradle precommit passed 46.34% of the time
compared to the Java Maven precommit which passed 46.15% of the time. When
I looked at these numbers in mid January they were around 37% so there has
been some improvement. Regardless of the build tool it seems that our pass
rates aren't stellar for the Java build and are causing the community to
not follow best practices (wait for precommits to be green before merging).
I know that on the website we used the mergebot to ensure that things
passed before they were merged, should we institute this on the master
branch or are their any other ideas?

As a side note we had achieved the goals we set out to not need to maintain
the Maven precommit and have authored the first PR to drop the Maven
precommit:  https://github.com/apache/beam/pull/4814


Well, I'd be for a strong switch otherwise PR will keep using maven,
jenkins will not test the code and at the end we fail to deliver something
consistent. So whatever tool is selected I'm tempted to say drop other
build files and jenkins hooks references.

What about doing it after 2.4 vote?



On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau 
wrote:

> Up,
>
> We discussed to have a strong switch to gradle or rollback to maven around
> april to not be blocked by the build tool. I noticed gradle build rarely
> passes on PR and kind of blurry our vision - not sure why exactly. Also, PR
> don't always contain the gradle updates - generally dependencies+plugins
> are added in pom.xml AFAIK, so it seems the adoption is very slow to not
> say rejected.
>
> What do we do about that? When do we drop the double build maintenance -
> whatever is picked?
>
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau :
>
>>
>>
>> Le 11 janv. 2018 23:13, "Kenneth Knowles"  a écrit :
>>
>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> 2. gradle build doesn't use the same output directory than maven so it
>>> is not really smooth to have both and have to maintain both
>>>
>>
>> I also have an opinion on this. It is useful and reasonable to be able to
>> build even when the source is on a read-only filesystem. Maven's defaults
>> are undesirable and require workarounds. We shouldn't mimic the behavior,
>> but actually should set gradle up to build to a directory outside the
>> source tree always, if we can.
>>
>>
>> Hmm, which is something you can do with maven as well so not sure I get
>> it.
>>
>> Also note the thread is no more about the technical points but more the
>> sources maintenance and consistency.
>>
>>
>> Kenn
>>
>>
>>
>

Re: Gradle status

[Lukasz Cwik]

Note that Alan Myrvold has been making steady progress making the release
process via Gradle a reality:
1) Creating a jenkins job which can run the quickstart validation against
nightly snapshots and also can be used for release candidates (
https://github.com/apache/beam/pull/4252)
2) Building a release candidate using Gradle (
https://github.com/apache/beam/pull/4812)

Also, Gradle is the tool that has been selected already and there was a
community discussion about what was needed for the migration to occur with
a clear set of criteria. Romain, it doesn't seem like we should ignore that
criteria or are you suggesting we change that criteria (if yes, how)?


On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau 
wrote:

>
>
> Le 7 mars 2018 17:34, "Lukasz Cwik"  a écrit :
>
> Thanks for bringing this up Romain but I believe your data points on pass
> rates are only partially correct.
>
>
> Sure sure, it is mainly about my own PR which a very small % of the whole
> project ;).
>
>
>
> In the past week the Java Gradle precommit passed 46.34% of the time
> compared to the Java Maven precommit which passed 46.15% of the time. When
> I looked at these numbers in mid January they were around 37% so there has
> been some improvement. Regardless of the build tool it seems that our pass
> rates aren't stellar for the Java build and are causing the community to
> not follow best practices (wait for precommits to be green before merging).
> I know that on the website we used the mergebot to ensure that things
> passed before they were merged, should we institute this on the master
> branch or are their any other ideas?
>
> As a side note we had achieved the goals we set out to not need to
> maintain the Maven precommit and have authored the first PR to drop the
> Maven precommit:  https://github.com/apache/beam/pull/4814
>
>
> Well, I'd be for a strong switch otherwise PR will keep using maven,
> jenkins will not test the code and at the end we fail to deliver something
> consistent. So whatever tool is selected I'm tempted to say drop other
> build files and jenkins hooks references.
>
> What about doing it after 2.4 vote?
>
>
>
> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau 
> wrote:
>
>> Up,
>>
>> We discussed to have a strong switch to gradle or rollback to maven
>> around april to not be blocked by the build tool. I noticed gradle build
>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>> Also, PR don't always contain the gradle updates - generally
>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>> is very slow to not say rejected.
>>
>> What do we do about that? When do we drop the double build maintenance -
>> whatever is picked?
>>
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau :
>>
>>>
>>>
>>> Le 11 janv. 2018 23:13, "Kenneth Knowles"  a écrit :
>>>
>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 2. gradle build doesn't use the same output directory than maven so it
 is not really smooth to have both and have to maintain both

>>>
>>> I also have an opinion on this. It is useful and reasonable to be able
>>> to build even when the source is on a read-only filesystem. Maven's
>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>> behavior, but actually should set gradle up to build to a directory outside
>>> the source tree always, if we can.
>>>
>>>
>>> Hmm, which is something you can do with maven as well so not sure I get
>>> it.
>>>
>>> Also note the thread is no more about the technical points but more the
>>> sources maintenance and consistency.
>>>
>>>
>>> Kenn
>>>
>>>
>>>
>>
>
>

Re: Gradle status

[Kenneth Knowles]

Now that the NeedsRunner tests are running via Gradle I support removing
the Java maven precommit.

Lower latency, simpler config, and more available Jenkins workers FTW.

Now let's fix the --rerun-tasks bugs so we can do even better.


On Wed, Mar 7, 2018 at 11:21 AM Lukasz Cwik  wrote:

> Note that Alan Myrvold has been making steady progress making the release
> process via Gradle a reality:
> 1) Creating a jenkins job which can run the quickstart validation against
> nightly snapshots and also can be used for release candidates (
> https://github.com/apache/beam/pull/4252)
> 2) Building a release candidate using Gradle (
> https://github.com/apache/beam/pull/4812)
>
> Also, Gradle is the tool that has been selected already and there was a
> community discussion about what was needed for the migration to occur with
> a clear set of criteria. Romain, it doesn't seem like we should ignore that
> criteria or are you suggesting we change that criteria (if yes, how)?
>
>
> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau  > wrote:
>
>>
>>
>> Le 7 mars 2018 17:34, "Lukasz Cwik"  a écrit :
>>
>> Thanks for bringing this up Romain but I believe your data points on pass
>> rates are only partially correct.
>>
>>
>> Sure sure, it is mainly about my own PR which a very small % of the whole
>> project ;).
>>
>>
>>
>> In the past week the Java Gradle precommit passed 46.34% of the time
>> compared to the Java Maven precommit which passed 46.15% of the time. When
>> I looked at these numbers in mid January they were around 37% so there has
>> been some improvement. Regardless of the build tool it seems that our pass
>> rates aren't stellar for the Java build and are causing the community to
>> not follow best practices (wait for precommits to be green before merging).
>> I know that on the website we used the mergebot to ensure that things
>> passed before they were merged, should we institute this on the master
>> branch or are their any other ideas?
>>
>> As a side note we had achieved the goals we set out to not need to
>> maintain the Maven precommit and have authored the first PR to drop the
>> Maven precommit:  https://github.com/apache/beam/pull/4814
>>
>>
>> Well, I'd be for a strong switch otherwise PR will keep using maven,
>> jenkins will not test the code and at the end we fail to deliver something
>> consistent. So whatever tool is selected I'm tempted to say drop other
>> build files and jenkins hooks references.
>>
>> What about doing it after 2.4 vote?
>>
>>
>>
>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau > > wrote:
>>
>>> Up,
>>>
>>> We discussed to have a strong switch to gradle or rollback to maven
>>> around april to not be blocked by the build tool. I noticed gradle build
>>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>>> Also, PR don't always contain the gradle updates - generally
>>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>>> is very slow to not say rejected.
>>>
>>> What do we do about that? When do we drop the double build maintenance -
>>> whatever is picked?
>>>
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau  |  Blog
>>>  | Old Blog
>>>  | Github
>>>  | LinkedIn
>>>  | Book
>>> 
>>>
>>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau :
>>>


 Le 11 janv. 2018 23:13, "Kenneth Knowles"  a écrit :

 On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> 2. gradle build doesn't use the same output directory than maven so it
> is not really smooth to have both and have to maintain both
>

 I also have an opinion on this. It is useful and reasonable to be able
 to build even when the source is on a read-only filesystem. Maven's
 defaults are undesirable and require workarounds. We shouldn't mimic the
 behavior, but actually should set gradle up to build to a directory outside
 the source tree always, if we can.


 Hmm, which is something you can do with maven as well so not sure I get
 it.

 Also note the thread is no more about the technical points but more the
 sources maintenance and consistency.


 Kenn



>>>
>>
>>
>

Re: Gradle status

[Romain Manni-Bucau]

Le 7 mars 2018 20:21, "Lukasz Cwik"  a écrit :

Note that Alan Myrvold has been making steady progress making the release
process via Gradle a reality:
1) Creating a jenkins job which can run the quickstart validation against
nightly snapshots and also can be used for release candidates (
https://github.com/apache/beam/pull/4252)
2) Building a release candidate using Gradle (https://github.com/apache/
beam/pull/4812)

Also, Gradle is the tool that has been selected already and there was a
community discussion about what was needed for the migration to occur with
a clear set of criteria. Romain, it doesn't seem like we should ignore that
criteria or are you suggesting we change that criteria (if yes, how)?


No, no. My goal is just to quit this state.

Let s draft a plan:

1. 2.4 is released - i assume it is done with mvn here
2. We drop all poms and jenkins mvn config
3. We fix all build issues if so (let say in a week)
4. Pr can nees updates but no more mvn merge

April is gradle month :)

Wdyt?




On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau 
wrote:

>
>
> Le 7 mars 2018 17:34, "Lukasz Cwik"  a écrit :
>
> Thanks for bringing this up Romain but I believe your data points on pass
> rates are only partially correct.
>
>
> Sure sure, it is mainly about my own PR which a very small % of the whole
> project ;).
>
>
>
> In the past week the Java Gradle precommit passed 46.34% of the time
> compared to the Java Maven precommit which passed 46.15% of the time. When
> I looked at these numbers in mid January they were around 37% so there has
> been some improvement. Regardless of the build tool it seems that our pass
> rates aren't stellar for the Java build and are causing the community to
> not follow best practices (wait for precommits to be green before merging).
> I know that on the website we used the mergebot to ensure that things
> passed before they were merged, should we institute this on the master
> branch or are their any other ideas?
>
> As a side note we had achieved the goals we set out to not need to
> maintain the Maven precommit and have authored the first PR to drop the
> Maven precommit:  https://github.com/apache/beam/pull/4814
>
>
> Well, I'd be for a strong switch otherwise PR will keep using maven,
> jenkins will not test the code and at the end we fail to deliver something
> consistent. So whatever tool is selected I'm tempted to say drop other
> build files and jenkins hooks references.
>
> What about doing it after 2.4 vote?
>
>
>
> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau 
> wrote:
>
>> Up,
>>
>> We discussed to have a strong switch to gradle or rollback to maven
>> around april to not be blocked by the build tool. I noticed gradle build
>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>> Also, PR don't always contain the gradle updates - generally
>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>> is very slow to not say rejected.
>>
>> What do we do about that? When do we drop the double build maintenance -
>> whatever is picked?
>>
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau :
>>
>>>
>>>
>>> Le 11 janv. 2018 23:13, "Kenneth Knowles"  a écrit :
>>>
>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 2. gradle build doesn't use the same output directory than maven so it
 is not really smooth to have both and have to maintain both

>>>
>>> I also have an opinion on this. It is useful and reasonable to be able
>>> to build even when the source is on a read-only filesystem. Maven's
>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>> behavior, but actually should set gradle up to build to a directory outside
>>> the source tree always, if we can.
>>>
>>>
>>> Hmm, which is something you can do with maven as well so not sure I get
>>> it.
>>>
>>> Also note the thread is no more about the technical points but more the
>>> sources maintenance and consistency.
>>>
>>>
>>> Kenn
>>>
>>>
>>>
>>
>
>

Re: Gradle status

[Lukasz Cwik]

I am working on various projects and may not be able to pause my work for a
couple of weeks while the build/test process is migrated.

What is everyone thinking about Romain's suggestion because If I'm the only
person in such a situation, I would be willing to go along with the plan.

On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau 
wrote:

>
>
> Le 7 mars 2018 20:21, "Lukasz Cwik"  a écrit :
>
> Note that Alan Myrvold has been making steady progress making the release
> process via Gradle a reality:
> 1) Creating a jenkins job which can run the quickstart validation against
> nightly snapshots and also can be used for release candidates (
> https://github.com/apache/beam/pull/4252)
> 2) Building a release candidate using Gradle (
> https://github.com/apache/beam/pull/4812)
>
> Also, Gradle is the tool that has been selected already and there was a
> community discussion about what was needed for the migration to occur with
> a clear set of criteria. Romain, it doesn't seem like we should ignore that
> criteria or are you suggesting we change that criteria (if yes, how)?
>
>
> No, no. My goal is just to quit this state.
>
> Let s draft a plan:
>
> 1. 2.4 is released - i assume it is done with mvn here
> 2. We drop all poms and jenkins mvn config
> 3. We fix all build issues if so (let say in a week)
> 4. Pr can nees updates but no more mvn merge
>
> April is gradle month :)
>
> Wdyt?
>
>
>
>
> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau  > wrote:
>
>>
>>
>> Le 7 mars 2018 17:34, "Lukasz Cwik"  a écrit :
>>
>> Thanks for bringing this up Romain but I believe your data points on pass
>> rates are only partially correct.
>>
>>
>> Sure sure, it is mainly about my own PR which a very small % of the whole
>> project ;).
>>
>>
>>
>> In the past week the Java Gradle precommit passed 46.34% of the time
>> compared to the Java Maven precommit which passed 46.15% of the time. When
>> I looked at these numbers in mid January they were around 37% so there has
>> been some improvement. Regardless of the build tool it seems that our pass
>> rates aren't stellar for the Java build and are causing the community to
>> not follow best practices (wait for precommits to be green before merging).
>> I know that on the website we used the mergebot to ensure that things
>> passed before they were merged, should we institute this on the master
>> branch or are their any other ideas?
>>
>> As a side note we had achieved the goals we set out to not need to
>> maintain the Maven precommit and have authored the first PR to drop the
>> Maven precommit:  https://github.com/apache/beam/pull/4814
>>
>>
>> Well, I'd be for a strong switch otherwise PR will keep using maven,
>> jenkins will not test the code and at the end we fail to deliver something
>> consistent. So whatever tool is selected I'm tempted to say drop other
>> build files and jenkins hooks references.
>>
>> What about doing it after 2.4 vote?
>>
>>
>>
>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau > > wrote:
>>
>>> Up,
>>>
>>> We discussed to have a strong switch to gradle or rollback to maven
>>> around april to not be blocked by the build tool. I noticed gradle build
>>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>>> Also, PR don't always contain the gradle updates - generally
>>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>>> is very slow to not say rejected.
>>>
>>> What do we do about that? When do we drop the double build maintenance -
>>> whatever is picked?
>>>
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau  |  Blog
>>>  | Old Blog
>>>  | Github
>>>  | LinkedIn
>>>  | Book
>>> 
>>>
>>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau :
>>>


 Le 11 janv. 2018 23:13, "Kenneth Knowles"  a écrit :

 On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> 2. gradle build doesn't use the same output directory than maven so it
> is not really smooth to have both and have to maintain both
>

 I also have an opinion on this. It is useful and reasonable to be able
 to build even when the source is on a read-only filesystem. Maven's
 defaults are undesirable and require workarounds. We shouldn't mimic the
 behavior, but actually should set gradle up to build to a directory outside
 the source tree always, if we can.


 Hmm, which is something you can do with maven as well so not sure I get
 it.

 Also note the thread is no more about the technical points but more the
 sources maintenance and consistency.


 Kenn



>>>
>>
>>
>
>

Re: Gradle status

[Kenneth Knowles]

I also cannot drop everything to work on Gradle build, but maybe it isn't
that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner tests
and some progress on the release, is there any other known missing
functionality in the Gradle builds? Archetypes? Docker container images?


On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik  wrote:

> I am working on various projects and may not be able to pause my work for
> a couple of weeks while the build/test process is migrated.
>
> What is everyone thinking about Romain's suggestion because If I'm the
> only person in such a situation, I would be willing to go along with the
> plan.
>
> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau  > wrote:
>
>>
>>
>> Le 7 mars 2018 20:21, "Lukasz Cwik"  a écrit :
>>
>> Note that Alan Myrvold has been making steady progress making the release
>> process via Gradle a reality:
>> 1) Creating a jenkins job which can run the quickstart validation against
>> nightly snapshots and also can be used for release candidates (
>> https://github.com/apache/beam/pull/4252)
>> 2) Building a release candidate using Gradle (
>> https://github.com/apache/beam/pull/4812)
>>
>> Also, Gradle is the tool that has been selected already and there was a
>> community discussion about what was needed for the migration to occur with
>> a clear set of criteria. Romain, it doesn't seem like we should ignore that
>> criteria or are you suggesting we change that criteria (if yes, how)?
>>
>>
>> No, no. My goal is just to quit this state.
>>
>> Let s draft a plan:
>>
>> 1. 2.4 is released - i assume it is done with mvn here
>> 2. We drop all poms and jenkins mvn config
>> 3. We fix all build issues if so (let say in a week)
>> 4. Pr can nees updates but no more mvn merge
>>
>> April is gradle month :)
>>
>> Wdyt?
>>
>>
>>
>>
>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>>
>>>
>>> Le 7 mars 2018 17:34, "Lukasz Cwik"  a écrit :
>>>
>>> Thanks for bringing this up Romain but I believe your data points on
>>> pass rates are only partially correct.
>>>
>>>
>>> Sure sure, it is mainly about my own PR which a very small % of the
>>> whole project ;).
>>>
>>>
>>>
>>> In the past week the Java Gradle precommit passed 46.34% of the time
>>> compared to the Java Maven precommit which passed 46.15% of the time. When
>>> I looked at these numbers in mid January they were around 37% so there has
>>> been some improvement. Regardless of the build tool it seems that our pass
>>> rates aren't stellar for the Java build and are causing the community to
>>> not follow best practices (wait for precommits to be green before merging).
>>> I know that on the website we used the mergebot to ensure that things
>>> passed before they were merged, should we institute this on the master
>>> branch or are their any other ideas?
>>>
>>> As a side note we had achieved the goals we set out to not need to
>>> maintain the Maven precommit and have authored the first PR to drop the
>>> Maven precommit:  https://github.com/apache/beam/pull/4814
>>>
>>>
>>> Well, I'd be for a strong switch otherwise PR will keep using maven,
>>> jenkins will not test the code and at the end we fail to deliver something
>>> consistent. So whatever tool is selected I'm tempted to say drop other
>>> build files and jenkins hooks references.
>>>
>>> What about doing it after 2.4 vote?
>>>
>>>
>>>
>>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Up,

 We discussed to have a strong switch to gradle or rollback to maven
 around april to not be blocked by the build tool. I noticed gradle build
 rarely passes on PR and kind of blurry our vision - not sure why exactly.
 Also, PR don't always contain the gradle updates - generally
 dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
 is very slow to not say rejected.

 What do we do about that? When do we drop the double build maintenance
 - whatever is picked?



 Romain Manni-Bucau
 @rmannibucau  |  Blog
  | Old Blog
  | Github
  | LinkedIn
  | Book
 

 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau :

>
>
> Le 11 janv. 2018 23:13, "Kenneth Knowles"  a écrit :
>
> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> 2. gradle build doesn't use the same output directory than maven so
>> it is not really smooth to have both and have to maintain both
>>
>
> I also have an opinion on this. It is useful and reasonable to be able
> to build even when the source is on a read-only filesystem. Maven's
> defaults are undes

Re: Gradle status

[Reuven Lax]

I think Alan was making progress on the Gradle build.

What do people think of a "fixit" day for Gradle work? (or given that
people are distributed, maybe a fixit week, where everyone takes one day
from the week).


On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles  wrote:

> I also cannot drop everything to work on Gradle build, but maybe it isn't
> that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner tests
> and some progress on the release, is there any other known missing
> functionality in the Gradle builds? Archetypes? Docker container images?
>
>
> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik  wrote:
>
>> I am working on various projects and may not be able to pause my work for
>> a couple of weeks while the build/test process is migrated.
>>
>> What is everyone thinking about Romain's suggestion because If I'm the
>> only person in such a situation, I would be willing to go along with the
>> plan.
>>
>> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>>
>>>
>>> Le 7 mars 2018 20:21, "Lukasz Cwik"  a écrit :
>>>
>>> Note that Alan Myrvold has been making steady progress making the
>>> release process via Gradle a reality:
>>> 1) Creating a jenkins job which can run the quickstart validation
>>> against nightly snapshots and also can be used for release candidates (
>>> https://github.com/apache/beam/pull/4252)
>>> 2) Building a release candidate using Gradle (
>>> https://github.com/apache/beam/pull/4812)
>>>
>>> Also, Gradle is the tool that has been selected already and there was a
>>> community discussion about what was needed for the migration to occur with
>>> a clear set of criteria. Romain, it doesn't seem like we should ignore that
>>> criteria or are you suggesting we change that criteria (if yes, how)?
>>>
>>>
>>> No, no. My goal is just to quit this state.
>>>
>>> Let s draft a plan:
>>>
>>> 1. 2.4 is released - i assume it is done with mvn here
>>> 2. We drop all poms and jenkins mvn config
>>> 3. We fix all build issues if so (let say in a week)
>>> 4. Pr can nees updates but no more mvn merge
>>>
>>> April is gradle month :)
>>>
>>> Wdyt?
>>>
>>>
>>>
>>>
>>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>


 Le 7 mars 2018 17:34, "Lukasz Cwik"  a écrit :

 Thanks for bringing this up Romain but I believe your data points on
 pass rates are only partially correct.


 Sure sure, it is mainly about my own PR which a very small % of the
 whole project ;).



 In the past week the Java Gradle precommit passed 46.34% of the time
 compared to the Java Maven precommit which passed 46.15% of the time. When
 I looked at these numbers in mid January they were around 37% so there has
 been some improvement. Regardless of the build tool it seems that our pass
 rates aren't stellar for the Java build and are causing the community to
 not follow best practices (wait for precommits to be green before merging).
 I know that on the website we used the mergebot to ensure that things
 passed before they were merged, should we institute this on the master
 branch or are their any other ideas?

 As a side note we had achieved the goals we set out to not need to
 maintain the Maven precommit and have authored the first PR to drop the
 Maven precommit:  https://github.com/apache/beam/pull/4814


 Well, I'd be for a strong switch otherwise PR will keep using maven,
 jenkins will not test the code and at the end we fail to deliver something
 consistent. So whatever tool is selected I'm tempted to say drop other
 build files and jenkins hooks references.

 What about doing it after 2.4 vote?



 On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Up,
>
> We discussed to have a strong switch to gradle or rollback to maven
> around april to not be blocked by the build tool. I noticed gradle build
> rarely passes on PR and kind of blurry our vision - not sure why exactly.
> Also, PR don't always contain the gradle updates - generally
> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
> is very slow to not say rejected.
>
> What do we do about that? When do we drop the double build maintenance
> - whatever is picked?
>
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau :
>
>>
>>
>> Le 11 janv. 2018 23:13, "Kenneth Knowles"  a écrit :
>>
>> On T

Re: Gradle status

[Lukasz Cwik]

Largest outstanding areas are:
* Documentation relevant to the contributors guide/release process/testing
* Performance tests

There has been good progress towards:
* Release artifact validations and generation
* ValidatesRunner post commits
* Pre commits
* Container builds


On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax  wrote:

> I think Alan was making progress on the Gradle build.
>
> What do people think of a "fixit" day for Gradle work? (or given that
> people are distributed, maybe a fixit week, where everyone takes one day
> from the week).
>
>
> On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles  wrote:
>
>> I also cannot drop everything to work on Gradle build, but maybe it isn't
>> that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner tests
>> and some progress on the release, is there any other known missing
>> functionality in the Gradle builds? Archetypes? Docker container images?
>>
>>
>> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik  wrote:
>>
>>> I am working on various projects and may not be able to pause my work
>>> for a couple of weeks while the build/test process is migrated.
>>>
>>> What is everyone thinking about Romain's suggestion because If I'm the
>>> only person in such a situation, I would be willing to go along with the
>>> plan.
>>>
>>> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>


 Le 7 mars 2018 20:21, "Lukasz Cwik"  a écrit :

 Note that Alan Myrvold has been making steady progress making the
 release process via Gradle a reality:
 1) Creating a jenkins job which can run the quickstart validation
 against nightly snapshots and also can be used for release candidates (
 https://github.com/apache/beam/pull/4252)
 2) Building a release candidate using Gradle (
 https://github.com/apache/beam/pull/4812)

 Also, Gradle is the tool that has been selected already and there was a
 community discussion about what was needed for the migration to occur with
 a clear set of criteria. Romain, it doesn't seem like we should ignore that
 criteria or are you suggesting we change that criteria (if yes, how)?


 No, no. My goal is just to quit this state.

 Let s draft a plan:

 1. 2.4 is released - i assume it is done with mvn here
 2. We drop all poms and jenkins mvn config
 3. We fix all build issues if so (let say in a week)
 4. Pr can nees updates but no more mvn merge

 April is gradle month :)

 Wdyt?




 On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

>
>
> Le 7 mars 2018 17:34, "Lukasz Cwik"  a écrit :
>
> Thanks for bringing this up Romain but I believe your data points on
> pass rates are only partially correct.
>
>
> Sure sure, it is mainly about my own PR which a very small % of the
> whole project ;).
>
>
>
> In the past week the Java Gradle precommit passed 46.34% of the time
> compared to the Java Maven precommit which passed 46.15% of the time. When
> I looked at these numbers in mid January they were around 37% so there has
> been some improvement. Regardless of the build tool it seems that our pass
> rates aren't stellar for the Java build and are causing the community to
> not follow best practices (wait for precommits to be green before 
> merging).
> I know that on the website we used the mergebot to ensure that things
> passed before they were merged, should we institute this on the master
> branch or are their any other ideas?
>
> As a side note we had achieved the goals we set out to not need to
> maintain the Maven precommit and have authored the first PR to drop the
> Maven precommit:  https://github.com/apache/beam/pull/4814
>
>
> Well, I'd be for a strong switch otherwise PR will keep using maven,
> jenkins will not test the code and at the end we fail to deliver something
> consistent. So whatever tool is selected I'm tempted to say drop other
> build files and jenkins hooks references.
>
> What about doing it after 2.4 vote?
>
>
>
> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Up,
>>
>> We discussed to have a strong switch to gradle or rollback to maven
>> around april to not be blocked by the build tool. I noticed gradle build
>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>> Also, PR don't always contain the gradle updates - generally
>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>> is very slow to not say rejected.
>>
>> What do we do about that? When do we drop the double build
>> maintenance - whatever is picked?
>>
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau 

Re: Gradle status

[Kenneth Knowles]

SGTM. I should also say that every day is Gradle fixit day for me, as I
have been using only Gradle (with IntelliJ) for a while :-). If anyone is
hesitant, definitely it is ready to be used for normal dev.

Seems like changing the messaging in onboarding docs is the main thing to
fixit.

Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure spam
level the performance tests are mostly not healthy anyhow. So is there any
high level blocker to switching them or is it just someone sitting down
with each one?

Kenn


On Wed, Mar 7, 2018 at 1:22 PM Lukasz Cwik  wrote:

> Largest outstanding areas are:
> * Documentation relevant to the contributors guide/release process/testing
> * Performance tests
>
> There has been good progress towards:
> * Release artifact validations and generation
> * ValidatesRunner post commits
> * Pre commits
> * Container builds
>
>
> On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax  wrote:
>
>> I think Alan was making progress on the Gradle build.
>>
>> What do people think of a "fixit" day for Gradle work? (or given that
>> people are distributed, maybe a fixit week, where everyone takes one day
>> from the week).
>>
>>
>> On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles  wrote:
>>
>>> I also cannot drop everything to work on Gradle build, but maybe it
>>> isn't that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner
>>> tests and some progress on the release, is there any other known missing
>>> functionality in the Gradle builds? Archetypes? Docker container images?
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik  wrote:
>>>
 I am working on various projects and may not be able to pause my work
 for a couple of weeks while the build/test process is migrated.

 What is everyone thinking about Romain's suggestion because If I'm the
 only person in such a situation, I would be willing to go along with the
 plan.

 On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

>
>
> Le 7 mars 2018 20:21, "Lukasz Cwik"  a écrit :
>
> Note that Alan Myrvold has been making steady progress making the
> release process via Gradle a reality:
> 1) Creating a jenkins job which can run the quickstart validation
> against nightly snapshots and also can be used for release candidates (
> https://github.com/apache/beam/pull/4252)
> 2) Building a release candidate using Gradle (
> https://github.com/apache/beam/pull/4812)
>
> Also, Gradle is the tool that has been selected already and there was
> a community discussion about what was needed for the migration to occur
> with a clear set of criteria. Romain, it doesn't seem like we should 
> ignore
> that criteria or are you suggesting we change that criteria (if yes, how)?
>
>
> No, no. My goal is just to quit this state.
>
> Let s draft a plan:
>
> 1. 2.4 is released - i assume it is done with mvn here
> 2. We drop all poms and jenkins mvn config
> 3. We fix all build issues if so (let say in a week)
> 4. Pr can nees updates but no more mvn merge
>
> April is gradle month :)
>
> Wdyt?
>
>
>
>
> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>>
>>
>> Le 7 mars 2018 17:34, "Lukasz Cwik"  a écrit :
>>
>> Thanks for bringing this up Romain but I believe your data points on
>> pass rates are only partially correct.
>>
>>
>> Sure sure, it is mainly about my own PR which a very small % of the
>> whole project ;).
>>
>>
>>
>> In the past week the Java Gradle precommit passed 46.34% of the time
>> compared to the Java Maven precommit which passed 46.15% of the time. 
>> When
>> I looked at these numbers in mid January they were around 37% so there 
>> has
>> been some improvement. Regardless of the build tool it seems that our 
>> pass
>> rates aren't stellar for the Java build and are causing the community to
>> not follow best practices (wait for precommits to be green before 
>> merging).
>> I know that on the website we used the mergebot to ensure that things
>> passed before they were merged, should we institute this on the master
>> branch or are their any other ideas?
>>
>> As a side note we had achieved the goals we set out to not need to
>> maintain the Maven precommit and have authored the first PR to drop the
>> Maven precommit:  https://github.com/apache/beam/pull/4814
>>
>>
>> Well, I'd be for a strong switch otherwise PR will keep using maven,
>> jenkins will not test the code and at the end we fail to deliver 
>> something
>> consistent. So whatever tool is selected I'm tempted to say drop other
>> build files and jenkins hooks references.
>>
>> What about doing it after 2.4 vote?
>>
>>
>>
>>

Re: Gradle status

[Reuven Lax]

Are there instructions for how to do this? I would like too switch my
IntelliJ over to Gradle (it's still setup using Maven)


On Wed, Mar 7, 2018 at 1:29 PM Kenneth Knowles  wrote:

> SGTM. I should also say that every day is Gradle fixit day for me, as I
> have been using only Gradle (with IntelliJ) for a while :-). If anyone is
> hesitant, definitely it is ready to be used for normal dev.
>
> Seems like changing the messaging in onboarding docs is the main thing to
> fixit.
>
> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
> spam level the performance tests are mostly not healthy anyhow. So is there
> any high level blocker to switching them or is it just someone sitting down
> with each one?
>
> Kenn
>
>
> On Wed, Mar 7, 2018 at 1:22 PM Lukasz Cwik  wrote:
>
>> Largest outstanding areas are:
>> * Documentation relevant to the contributors guide/release process/testing
>> * Performance tests
>>
>> There has been good progress towards:
>> * Release artifact validations and generation
>> * ValidatesRunner post commits
>> * Pre commits
>> * Container builds
>>
>>
>> On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax  wrote:
>>
>>> I think Alan was making progress on the Gradle build.
>>>
>>> What do people think of a "fixit" day for Gradle work? (or given that
>>> people are distributed, maybe a fixit week, where everyone takes one day
>>> from the week).
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles  wrote:
>>>
 I also cannot drop everything to work on Gradle build, but maybe it
 isn't that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner
 tests and some progress on the release, is there any other known missing
 functionality in the Gradle builds? Archetypes? Docker container images?


 On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik  wrote:

> I am working on various projects and may not be able to pause my work
> for a couple of weeks while the build/test process is migrated.
>
> What is everyone thinking about Romain's suggestion because If I'm the
> only person in such a situation, I would be willing to go along with the
> plan.
>
> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>>
>>
>> Le 7 mars 2018 20:21, "Lukasz Cwik"  a écrit :
>>
>> Note that Alan Myrvold has been making steady progress making the
>> release process via Gradle a reality:
>> 1) Creating a jenkins job which can run the quickstart validation
>> against nightly snapshots and also can be used for release candidates (
>> https://github.com/apache/beam/pull/4252)
>> 2) Building a release candidate using Gradle (
>> https://github.com/apache/beam/pull/4812)
>>
>> Also, Gradle is the tool that has been selected already and there was
>> a community discussion about what was needed for the migration to occur
>> with a clear set of criteria. Romain, it doesn't seem like we should 
>> ignore
>> that criteria or are you suggesting we change that criteria (if yes, 
>> how)?
>>
>>
>> No, no. My goal is just to quit this state.
>>
>> Let s draft a plan:
>>
>> 1. 2.4 is released - i assume it is done with mvn here
>> 2. We drop all poms and jenkins mvn config
>> 3. We fix all build issues if so (let say in a week)
>> 4. Pr can nees updates but no more mvn merge
>>
>> April is gradle month :)
>>
>> Wdyt?
>>
>>
>>
>>
>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>>
>>>
>>> Le 7 mars 2018 17:34, "Lukasz Cwik"  a écrit :
>>>
>>> Thanks for bringing this up Romain but I believe your data points on
>>> pass rates are only partially correct.
>>>
>>>
>>> Sure sure, it is mainly about my own PR which a very small % of the
>>> whole project ;).
>>>
>>>
>>>
>>> In the past week the Java Gradle precommit passed 46.34% of the time
>>> compared to the Java Maven precommit which passed 46.15% of the time. 
>>> When
>>> I looked at these numbers in mid January they were around 37% so there 
>>> has
>>> been some improvement. Regardless of the build tool it seems that our 
>>> pass
>>> rates aren't stellar for the Java build and are causing the community to
>>> not follow best practices (wait for precommits to be green before 
>>> merging).
>>> I know that on the website we used the mergebot to ensure that things
>>> passed before they were merged, should we institute this on the master
>>> branch or are their any other ideas?
>>>
>>> As a side note we had achieved the goals we set out to not need to
>>> maintain the Maven precommit and have authored the first PR to drop the
>>> Maven precommit:  https://github.com/apache/beam/pull/4814
>>>
>>>
>>> Well, I'd be for a strong switch 

Re: Gradle status

[Henning Rohde]

+1 to a Gradle fixit day/week. I agree with Romain that we should make an
effort quit this dual state. The Go, Go PreCommit and various container
image Gradle builds, I think, are in a reasonable state (modulo some
documentation updates).

On Wed, Mar 7, 2018 at 1:29 PM, Kenneth Knowles  wrote:

> SGTM. I should also say that every day is Gradle fixit day for me, as I
> have been using only Gradle (with IntelliJ) for a while :-). If anyone is
> hesitant, definitely it is ready to be used for normal dev.
>
> Seems like changing the messaging in onboarding docs is the main thing to
> fixit.
>
> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
> spam level the performance tests are mostly not healthy anyhow. So is there
> any high level blocker to switching them or is it just someone sitting down
> with each one?
>
> Kenn
>
>
> On Wed, Mar 7, 2018 at 1:22 PM Lukasz Cwik  wrote:
>
>> Largest outstanding areas are:
>> * Documentation relevant to the contributors guide/release process/testing
>> * Performance tests
>>
>> There has been good progress towards:
>> * Release artifact validations and generation
>> * ValidatesRunner post commits
>> * Pre commits
>> * Container builds
>>
>>
>> On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax  wrote:
>>
>>> I think Alan was making progress on the Gradle build.
>>>
>>> What do people think of a "fixit" day for Gradle work? (or given that
>>> people are distributed, maybe a fixit week, where everyone takes one day
>>> from the week).
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles  wrote:
>>>
 I also cannot drop everything to work on Gradle build, but maybe it
 isn't that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner
 tests and some progress on the release, is there any other known missing
 functionality in the Gradle builds? Archetypes? Docker container images?


 On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik  wrote:

> I am working on various projects and may not be able to pause my work
> for a couple of weeks while the build/test process is migrated.
>
> What is everyone thinking about Romain's suggestion because If I'm the
> only person in such a situation, I would be willing to go along with the
> plan.
>
> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>>
>>
>> Le 7 mars 2018 20:21, "Lukasz Cwik"  a écrit :
>>
>> Note that Alan Myrvold has been making steady progress making the
>> release process via Gradle a reality:
>> 1) Creating a jenkins job which can run the quickstart validation
>> against nightly snapshots and also can be used for release candidates (
>> https://github.com/apache/beam/pull/4252)
>> 2) Building a release candidate using Gradle (
>> https://github.com/apache/beam/pull/4812)
>>
>> Also, Gradle is the tool that has been selected already and there was
>> a community discussion about what was needed for the migration to occur
>> with a clear set of criteria. Romain, it doesn't seem like we should 
>> ignore
>> that criteria or are you suggesting we change that criteria (if yes, 
>> how)?
>>
>>
>> No, no. My goal is just to quit this state.
>>
>> Let s draft a plan:
>>
>> 1. 2.4 is released - i assume it is done with mvn here
>> 2. We drop all poms and jenkins mvn config
>> 3. We fix all build issues if so (let say in a week)
>> 4. Pr can nees updates but no more mvn merge
>>
>> April is gradle month :)
>>
>> Wdyt?
>>
>>
>>
>>
>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>>
>>>
>>> Le 7 mars 2018 17:34, "Lukasz Cwik"  a écrit :
>>>
>>> Thanks for bringing this up Romain but I believe your data points on
>>> pass rates are only partially correct.
>>>
>>>
>>> Sure sure, it is mainly about my own PR which a very small % of the
>>> whole project ;).
>>>
>>>
>>>
>>> In the past week the Java Gradle precommit passed 46.34% of the time
>>> compared to the Java Maven precommit which passed 46.15% of the time. 
>>> When
>>> I looked at these numbers in mid January they were around 37% so there 
>>> has
>>> been some improvement. Regardless of the build tool it seems that our 
>>> pass
>>> rates aren't stellar for the Java build and are causing the community to
>>> not follow best practices (wait for precommits to be green before 
>>> merging).
>>> I know that on the website we used the mergebot to ensure that things
>>> passed before they were merged, should we institute this on the master
>>> branch or are their any other ideas?
>>>
>>> As a side note we had achieved the goals we set out to not need to
>>> maintain the Maven precommit and have authored the first PR to drop the
>>> Ma

Re: Gradle status

[Kenneth Knowles]

I'll write up the steps as part of the fixit :-)

I put IntelliJ hints into the build.gradle enough that you don't do any
specific tweaking. Here's short version:

1. Create new empty project and set up JDK or create new Java project. Put
it outside the source tree so you can `git clean -d -f -x`
2. Create new module from existing sources -> import module from external
model -> Gradle
- Use auto-import because our modules can be churny
- Create separate module per source set, necessary for smaller build
times IIRC
- Store generated project files externally or it is all screwy
- Use default gradle wrapper
3. In settings, delegate build actions to Gradle
4. Build project. This populates generated source folders so IntelliJ will
grok references to it.


On Wed, Mar 7, 2018 at 1:49 PM Henning Rohde  wrote:

> +1 to a Gradle fixit day/week. I agree with Romain that we should make an
> effort quit this dual state. The Go, Go PreCommit and various container
> image Gradle builds, I think, are in a reasonable state (modulo some
> documentation updates).
>
> On Wed, Mar 7, 2018 at 1:29 PM, Kenneth Knowles  wrote:
>
>> SGTM. I should also say that every day is Gradle fixit day for me, as I
>> have been using only Gradle (with IntelliJ) for a while :-). If anyone is
>> hesitant, definitely it is ready to be used for normal dev.
>>
>> Seems like changing the messaging in onboarding docs is the main thing to
>> fixit.
>>
>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
>> spam level the performance tests are mostly not healthy anyhow. So is there
>> any high level blocker to switching them or is it just someone sitting down
>> with each one?
>>
>> Kenn
>>
>>
>> On Wed, Mar 7, 2018 at 1:22 PM Lukasz Cwik  wrote:
>>
>>> Largest outstanding areas are:
>>> * Documentation relevant to the contributors guide/release
>>> process/testing
>>> * Performance tests
>>>
>>> There has been good progress towards:
>>> * Release artifact validations and generation
>>> * ValidatesRunner post commits
>>> * Pre commits
>>> * Container builds
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax  wrote:
>>>
 I think Alan was making progress on the Gradle build.

 What do people think of a "fixit" day for Gradle work? (or given that
 people are distributed, maybe a fixit week, where everyone takes one day
 from the week).


 On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles  wrote:

> I also cannot drop everything to work on Gradle build, but maybe it
> isn't that drastic anyhow. Now that we have ValidatesRunner and 
> NeedsRunner
> tests and some progress on the release, is there any other known missing
> functionality in the Gradle builds? Archetypes? Docker container images?
>
>
> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik  wrote:
>
>> I am working on various projects and may not be able to pause my work
>> for a couple of weeks while the build/test process is migrated.
>>
>> What is everyone thinking about Romain's suggestion because If I'm
>> the only person in such a situation, I would be willing to go along with
>> the plan.
>>
>> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>>
>>>
>>> Le 7 mars 2018 20:21, "Lukasz Cwik"  a écrit :
>>>
>>> Note that Alan Myrvold has been making steady progress making the
>>> release process via Gradle a reality:
>>> 1) Creating a jenkins job which can run the quickstart validation
>>> against nightly snapshots and also can be used for release candidates (
>>> https://github.com/apache/beam/pull/4252)
>>> 2) Building a release candidate using Gradle (
>>> https://github.com/apache/beam/pull/4812)
>>>
>>> Also, Gradle is the tool that has been selected already and there
>>> was a community discussion about what was needed for the migration to 
>>> occur
>>> with a clear set of criteria. Romain, it doesn't seem like we should 
>>> ignore
>>> that criteria or are you suggesting we change that criteria (if yes, 
>>> how)?
>>>
>>>
>>> No, no. My goal is just to quit this state.
>>>
>>> Let s draft a plan:
>>>
>>> 1. 2.4 is released - i assume it is done with mvn here
>>> 2. We drop all poms and jenkins mvn config
>>> 3. We fix all build issues if so (let say in a week)
>>> 4. Pr can nees updates but no more mvn merge
>>>
>>> April is gradle month :)
>>>
>>> Wdyt?
>>>
>>>
>>>
>>>
>>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>


 Le 7 mars 2018 17:34, "Lukasz Cwik"  a écrit :

 Thanks for bringing this up Romain but I believe your data points
 on pass rates are only partially correct.


 Sure sure, it is mainly about my own PR which a very

Re: Gradle status

[Robert Bradshaw]

+1 to a fixit day. I'd be happy to help out myself.


On Wed, Mar 7, 2018 at 1:49 PM Henning Rohde  wrote:

> +1 to a Gradle fixit day/week. I agree with Romain that we should make an
> effort quit this dual state. The Go, Go PreCommit and various container
> image Gradle builds, I think, are in a reasonable state (modulo some
> documentation updates).
>
> On Wed, Mar 7, 2018 at 1:29 PM, Kenneth Knowles  wrote:
>
>> SGTM. I should also say that every day is Gradle fixit day for me, as I
>> have been using only Gradle (with IntelliJ) for a while :-). If anyone is
>> hesitant, definitely it is ready to be used for normal dev.
>>
>> Seems like changing the messaging in onboarding docs is the main thing to
>> fixit.
>>
>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
>> spam level the performance tests are mostly not healthy anyhow. So is there
>> any high level blocker to switching them or is it just someone sitting down
>> with each one?
>>
>> Kenn
>>
>>
>> On Wed, Mar 7, 2018 at 1:22 PM Lukasz Cwik  wrote:
>>
>>> Largest outstanding areas are:
>>> * Documentation relevant to the contributors guide/release
>>> process/testing
>>> * Performance tests
>>>
>>> There has been good progress towards:
>>> * Release artifact validations and generation
>>> * ValidatesRunner post commits
>>> * Pre commits
>>> * Container builds
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax  wrote:
>>>
 I think Alan was making progress on the Gradle build.

 What do people think of a "fixit" day for Gradle work? (or given that
 people are distributed, maybe a fixit week, where everyone takes one day
 from the week).


 On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles  wrote:

> I also cannot drop everything to work on Gradle build, but maybe it
> isn't that drastic anyhow. Now that we have ValidatesRunner and 
> NeedsRunner
> tests and some progress on the release, is there any other known missing
> functionality in the Gradle builds? Archetypes? Docker container images?
>
>
> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik  wrote:
>
>> I am working on various projects and may not be able to pause my work
>> for a couple of weeks while the build/test process is migrated.
>>
>> What is everyone thinking about Romain's suggestion because If I'm
>> the only person in such a situation, I would be willing to go along with
>> the plan.
>>
>> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>>
>>>
>>> Le 7 mars 2018 20:21, "Lukasz Cwik"  a écrit :
>>>
>>> Note that Alan Myrvold has been making steady progress making the
>>> release process via Gradle a reality:
>>> 1) Creating a jenkins job which can run the quickstart validation
>>> against nightly snapshots and also can be used for release candidates (
>>> https://github.com/apache/beam/pull/4252)
>>> 2) Building a release candidate using Gradle (
>>> https://github.com/apache/beam/pull/4812)
>>>
>>> Also, Gradle is the tool that has been selected already and there
>>> was a community discussion about what was needed for the migration to 
>>> occur
>>> with a clear set of criteria. Romain, it doesn't seem like we should 
>>> ignore
>>> that criteria or are you suggesting we change that criteria (if yes, 
>>> how)?
>>>
>>>
>>> No, no. My goal is just to quit this state.
>>>
>>> Let s draft a plan:
>>>
>>> 1. 2.4 is released - i assume it is done with mvn here
>>> 2. We drop all poms and jenkins mvn config
>>> 3. We fix all build issues if so (let say in a week)
>>> 4. Pr can nees updates but no more mvn merge
>>>
>>> April is gradle month :)
>>>
>>> Wdyt?
>>>
>>>
>>>
>>>
>>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>


 Le 7 mars 2018 17:34, "Lukasz Cwik"  a écrit :

 Thanks for bringing this up Romain but I believe your data points
 on pass rates are only partially correct.


 Sure sure, it is mainly about my own PR which a very small % of the
 whole project ;).



 In the past week the Java Gradle precommit passed 46.34% of the
 time compared to the Java Maven precommit which passed 46.15% of the 
 time.
 When I looked at these numbers in mid January they were around 37% so 
 there
 has been some improvement. Regardless of the build tool it seems that 
 our
 pass rates aren't stellar for the Java build and are causing the 
 community
 to not follow best practices (wait for precommits to be green before
 merging). I know that on the website we used the mergebot to ensure 
 that
 things passed before they

Re: Gradle status

[Jason Kuster]

+1


On Wed, Mar 7, 2018 at 2:21 PM Robert Bradshaw  wrote:

> +1 to a fixit day. I'd be happy to help out myself.
>
>
> On Wed, Mar 7, 2018 at 1:49 PM Henning Rohde  wrote:
>
>> +1 to a Gradle fixit day/week. I agree with Romain that we should make an
>> effort quit this dual state. The Go, Go PreCommit and various container
>> image Gradle builds, I think, are in a reasonable state (modulo some
>> documentation updates).
>>
>> On Wed, Mar 7, 2018 at 1:29 PM, Kenneth Knowles  wrote:
>>
>>> SGTM. I should also say that every day is Gradle fixit day for me, as I
>>> have been using only Gradle (with IntelliJ) for a while :-). If anyone is
>>> hesitant, definitely it is ready to be used for normal dev.
>>>
>>> Seems like changing the messaging in onboarding docs is the main thing
>>> to fixit.
>>>
>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
>>> spam level the performance tests are mostly not healthy anyhow. So is there
>>> any high level blocker to switching them or is it just someone sitting down
>>> with each one?
>>>
>>> Kenn
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:22 PM Lukasz Cwik  wrote:
>>>
 Largest outstanding areas are:
 * Documentation relevant to the contributors guide/release
 process/testing
 * Performance tests

 There has been good progress towards:
 * Release artifact validations and generation
 * ValidatesRunner post commits
 * Pre commits
 * Container builds


 On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax  wrote:

> I think Alan was making progress on the Gradle build.
>
> What do people think of a "fixit" day for Gradle work? (or given that
> people are distributed, maybe a fixit week, where everyone takes one day
> from the week).
>
>
> On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles  wrote:
>
>> I also cannot drop everything to work on Gradle build, but maybe it
>> isn't that drastic anyhow. Now that we have ValidatesRunner and 
>> NeedsRunner
>> tests and some progress on the release, is there any other known missing
>> functionality in the Gradle builds? Archetypes? Docker container images?
>>
>>
>> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik  wrote:
>>
>>> I am working on various projects and may not be able to pause my
>>> work for a couple of weeks while the build/test process is migrated.
>>>
>>> What is everyone thinking about Romain's suggestion because If I'm
>>> the only person in such a situation, I would be willing to go along with
>>> the plan.
>>>
>>> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>


 Le 7 mars 2018 20:21, "Lukasz Cwik"  a écrit :

 Note that Alan Myrvold has been making steady progress making the
 release process via Gradle a reality:
 1) Creating a jenkins job which can run the quickstart validation
 against nightly snapshots and also can be used for release candidates (
 https://github.com/apache/beam/pull/4252)
 2) Building a release candidate using Gradle (
 https://github.com/apache/beam/pull/4812)

 Also, Gradle is the tool that has been selected already and there
 was a community discussion about what was needed for the migration to 
 occur
 with a clear set of criteria. Romain, it doesn't seem like we should 
 ignore
 that criteria or are you suggesting we change that criteria (if yes, 
 how)?


 No, no. My goal is just to quit this state.

 Let s draft a plan:

 1. 2.4 is released - i assume it is done with mvn here
 2. We drop all poms and jenkins mvn config
 3. We fix all build issues if so (let say in a week)
 4. Pr can nees updates but no more mvn merge

 April is gradle month :)

 Wdyt?




 On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

>
>
> Le 7 mars 2018 17:34, "Lukasz Cwik"  a écrit :
>
> Thanks for bringing this up Romain but I believe your data points
> on pass rates are only partially correct.
>
>
> Sure sure, it is mainly about my own PR which a very small % of
> the whole project ;).
>
>
>
> In the past week the Java Gradle precommit passed 46.34% of the
> time compared to the Java Maven precommit which passed 46.15% of the 
> time.
> When I looked at these numbers in mid January they were around 37% so 
> there
> has been some improvement. Regardless of the build tool it seems that 
> our
> pass rates aren't stellar for the Java build and are causing the 
> community

Re: Gradle status

[Romain Manni-Bucau]

@Kenneth: this looks like my experience as well but is Idea - not even
speaking of eclipse - that bad with gradle? Thought that it was not since
android picked it up but never managed to make it scale on bigger projects
due to the prebuild phases which are super slow, require a full passing
build (which can be tricky with the native expectations of beam for pyyhon
for instance :() and regular updates when working on new modules in PR.

Anyone knows a way to make it better, and more important, working OOTB
whatever the machine you are using (with gradle/java but not go and with a
bad version of python to take a random example ;))? Does it need a fast
build import or idea meta generation?

Le 7 mars 2018 23:22, "Jason Kuster"  a écrit :

> +1
>
>
> On Wed, Mar 7, 2018 at 2:21 PM Robert Bradshaw 
> wrote:
>
>> +1 to a fixit day. I'd be happy to help out myself.
>>
>>
>> On Wed, Mar 7, 2018 at 1:49 PM Henning Rohde  wrote:
>>
>>> +1 to a Gradle fixit day/week. I agree with Romain that we should make
>>> an effort quit this dual state. The Go, Go PreCommit and various container
>>> image Gradle builds, I think, are in a reasonable state (modulo some
>>> documentation updates).
>>>
>>> On Wed, Mar 7, 2018 at 1:29 PM, Kenneth Knowles  wrote:
>>>
 SGTM. I should also say that every day is Gradle fixit day for me, as I
 have been using only Gradle (with IntelliJ) for a while :-). If anyone is
 hesitant, definitely it is ready to be used for normal dev.

 Seems like changing the messaging in onboarding docs is the main thing
 to fixit.

 Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
 spam level the performance tests are mostly not healthy anyhow. So is there
 any high level blocker to switching them or is it just someone sitting down
 with each one?

 Kenn


 On Wed, Mar 7, 2018 at 1:22 PM Lukasz Cwik  wrote:

> Largest outstanding areas are:
> * Documentation relevant to the contributors guide/release
> process/testing
> * Performance tests
>
> There has been good progress towards:
> * Release artifact validations and generation
> * ValidatesRunner post commits
> * Pre commits
> * Container builds
>
>
> On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax  wrote:
>
>> I think Alan was making progress on the Gradle build.
>>
>> What do people think of a "fixit" day for Gradle work? (or given that
>> people are distributed, maybe a fixit week, where everyone takes one day
>> from the week).
>>
>>
>> On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles 
>> wrote:
>>
>>> I also cannot drop everything to work on Gradle build, but maybe it
>>> isn't that drastic anyhow. Now that we have ValidatesRunner and 
>>> NeedsRunner
>>> tests and some progress on the release, is there any other known missing
>>> functionality in the Gradle builds? Archetypes? Docker container images?
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik  wrote:
>>>
 I am working on various projects and may not be able to pause my
 work for a couple of weeks while the build/test process is migrated.

 What is everyone thinking about Romain's suggestion because If I'm
 the only person in such a situation, I would be willing to go along 
 with
 the plan.

 On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

>
>
> Le 7 mars 2018 20:21, "Lukasz Cwik"  a écrit :
>
> Note that Alan Myrvold has been making steady progress making the
> release process via Gradle a reality:
> 1) Creating a jenkins job which can run the quickstart validation
> against nightly snapshots and also can be used for release candidates 
> (
> https://github.com/apache/beam/pull/4252)
> 2) Building a release candidate using Gradle (
> https://github.com/apache/beam/pull/4812)
>
> Also, Gradle is the tool that has been selected already and there
> was a community discussion about what was needed for the migration to 
> occur
> with a clear set of criteria. Romain, it doesn't seem like we should 
> ignore
> that criteria or are you suggesting we change that criteria (if yes, 
> how)?
>
>
> No, no. My goal is just to quit this state.
>
> Let s draft a plan:
>
> 1. 2.4 is released - i assume it is done with mvn here
> 2. We drop all poms and jenkins mvn config
> 3. We fix all build issues if so (let say in a week)
> 4. Pr can nees updates but no more mvn merge
>
> April is gradle month :)
>
> Wdyt?
>
>
>
>
> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-B

Re: Gradle status

[Łukasz Gajowy]

2018-03-07 22:29 GMT+01:00 Kenneth Knowles :

>
> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
> spam level the performance tests are mostly not healthy anyhow. So is there
> any high level blocker to switching them or is it just someone sitting down
> with each one?
>
>
I thought I could share my point of view here as I am working on the
Performance Test part for a while now. I wouldn't say those are "mostly not
healthy". The situation is as follows:

- totally not working: Python, Spark performance tests (don't know why, I'm
not maintaining those tests. Should we disable them?)
- flaky: the recently re-enabled JDBC Performance Test. It's seems to be
flaky mostly due to: https://issues.apache.org/jira/browse/BEAM-3798
- working well, rarely failing: AvroIO, TextIO, Compressed Text, TFRecordIO

Also, some test failures are caused due to pending PRs (we sometimes run
concrete tests to check if PRs won't break them). This also causes failures
sometimes.

I can help with switching the Performance tests to Gradle as this part
seems to be free to take.

Łukasz

Re: Gradle status

[Chamikara Jayalath]

+1 for the general idea of fixit day/week for Gradle.

Agree with what Łukasz said. Some of these performance tests are new and
are flaky due to other issues that were discovered during the process of
adding the test.

I think the high level blocker is updating performance testing framework to
use Gradle. This will involve adding Gradle-based logic for invoking
PerfKitBenchmaker, for example [1], and updating PerfKitBenchmarker to
issue a Gradle command for running the benchmark [2]. First task will be to
find the work needed here and updating the relevant JIRA [3].

Thanks,
Cham

[1]
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
[2]
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
[3] https://issues.apache.org/jira/browse/BEAM-3251

On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy 
wrote:

> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles :
>
>>
>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
>> spam level the performance tests are mostly not healthy anyhow. So is there
>> any high level blocker to switching them or is it just someone sitting down
>> with each one?
>>
>>
> I thought I could share my point of view here as I am working on the
> Performance Test part for a while now. I wouldn't say those are "mostly not
> healthy". The situation is as follows:
>
> - totally not working: Python, Spark performance tests (don't know why,
> I'm not maintaining those tests. Should we disable them?)
> - flaky: the recently re-enabled JDBC Performance Test. It's seems to be
> flaky mostly due to: https://issues.apache.org/jira/browse/BEAM-3798
> - working well, rarely failing: AvroIO, TextIO, Compressed Text, TFRecordIO
>
> Also, some test failures are caused due to pending PRs (we sometimes run
> concrete tests to check if PRs won't break them). This also causes failures
> sometimes.
>
> I can help with switching the Performance tests to Gradle as this part
> seems to be free to take.
>
>
> Łukasz
>
>

Re: Gradle status

[Lukasz Cwik]

Romain, would you like to host/plan/run the Gradle fixit day?

On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath 
wrote:

> +1 for the general idea of fixit day/week for Gradle.
>
> Agree with what Łukasz said. Some of these performance tests are new and
> are flaky due to other issues that were discovered during the process of
> adding the test.
>
> I think the high level blocker is updating performance testing framework
> to use Gradle. This will involve adding Gradle-based logic for invoking
> PerfKitBenchmaker, for example [1], and updating PerfKitBenchmarker to
> issue a Gradle command for running the benchmark [2]. First task will be to
> find the work needed here and updating the relevant JIRA [3].
>
> Thanks,
> Cham
>
> [1]  https://github.com/apache/beam/blob/master/sdks/
> java/io/google-cloud-platform/pom.xml#L79
> [2] https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/
> master/perfkitbenchmarker/beam_benchmark_helper.py
> [3] https://issues.apache.org/jira/browse/BEAM-3251
>
> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy 
> wrote:
>
>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles :
>>
>>>
>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
>>> spam level the performance tests are mostly not healthy anyhow. So is there
>>> any high level blocker to switching them or is it just someone sitting down
>>> with each one?
>>>
>>>
>> I thought I could share my point of view here as I am working on the
>> Performance Test part for a while now. I wouldn't say those are "mostly not
>> healthy". The situation is as follows:
>>
>> - totally not working: Python, Spark performance tests (don't know why,
>> I'm not maintaining those tests. Should we disable them?)
>> - flaky: the recently re-enabled JDBC Performance Test. It's seems to be
>> flaky mostly due to: https://issues.apache.org/jira/browse/BEAM-3798
>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>> TFRecordIO
>>
>> Also, some test failures are caused due to pending PRs (we sometimes run
>> concrete tests to check if PRs won't break them). This also causes failures
>> sometimes.
>>
>> I can help with switching the Performance tests to Gradle as this part
>> seems to be free to take.
>>
>>
>> Łukasz
>>
>>

Re: Gradle status

[Kenneth Knowles]

@Romain - Idea/IntelliJ is great with Gradle, way better than Maven, and
what I mean is that I have added enough hints that it works OOTB already.
The rest of my instructions are just how you should override IntelliJ's
defaults to have a proper dev env - mostly just about storing files outside
the git clone.

@Łukasz - yea I just mean the ones that I find in my email and browsing
https://builds.apache.org/view/A-D/view/Beam/ which are the ones you
classify as "totally not working". I would love to disable these.

On the subject of running things on a pending PR - we should not run
postcommit jobs on PRs. We should make separate (optional) precommit jobs
that run the same build commands. That will give a more clear history and
allow trivial configuration of which jobs deserve alert emails and which
are not a problem. This is easy but I've been waiting to do it after Gradle
migration.

On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik  wrote:

> Romain, would you like to host/plan/run the Gradle fixit day?
>
> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath 
> wrote:
>
>> +1 for the general idea of fixit day/week for Gradle.
>>
>> Agree with what Łukasz said. Some of these performance tests are new and
>> are flaky due to other issues that were discovered during the process of
>> adding the test.
>>
>> I think the high level blocker is updating performance testing framework
>> to use Gradle. This will involve adding Gradle-based logic for invoking
>> PerfKitBenchmaker, for example [1], and updating PerfKitBenchmarker to
>> issue a Gradle command for running the benchmark [2]. First task will be to
>> find the work needed here and updating the relevant JIRA [3].
>>
>> Thanks,
>> Cham
>>
>> [1]
>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>> [2]
>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>
>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy 
>> wrote:
>>
>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles :
>>>

 Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
 spam level the performance tests are mostly not healthy anyhow. So is there
 any high level blocker to switching them or is it just someone sitting down
 with each one?


>>> I thought I could share my point of view here as I am working on the
>>> Performance Test part for a while now. I wouldn't say those are "mostly not
>>> healthy". The situation is as follows:
>>>
>>> - totally not working: Python, Spark performance tests (don't know why,
>>> I'm not maintaining those tests. Should we disable them?)
>>> - flaky: the recently re-enabled JDBC Performance Test. It's seems to be
>>> flaky mostly due to: https://issues.apache.org/jira/browse/BEAM-3798
>>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>>> TFRecordIO
>>>
>>> Also, some test failures are caused due to pending PRs (we sometimes run
>>> concrete tests to check if PRs won't break them). This also causes failures
>>> sometimes.
>>>
>>> I can help with switching the Performance tests to Gradle as this part
>>> seems to be free to take.
>>>
>>>
>>> Łukasz
>>>
>>>
>

Re: Gradle status

[Romain Manni-Bucau]

@Luskasz: not sure Im the best to host it since I know more gradle
internals that user interface/ecosystem but happy to help. Will also
require a "sudo" merger for this day to merge fixes asap - guess we can
bypass reviews or have a fast cycle plan for this day to avoid it to be a
week?

Le 8 mars 2018 21:08, "Kenneth Knowles"  a écrit :

@Romain - Idea/IntelliJ is great with Gradle, way better than Maven, and
what I mean is that I have added enough hints that it works OOTB already.
The rest of my instructions are just how you should override IntelliJ's
defaults to have a proper dev env - mostly just about storing files outside
the git clone.


Hmm it is always super slow here and not as integrated as maven which is
smooth thanks to the combination of idea build and maven plugin config
read. Import is also faster cause it just reads meta and doesnt run
anything. Hope it is a "a few times" issues at the moment but not yet sure.



@Łukasz - yea I just mean the ones that I find in my email and browsing
https://builds.apache.org/view/A-D/view/Beam/ which are the ones you
classify as "totally not working". I would love to disable these.

On the subject of running things on a pending PR - we should not run
postcommit jobs on PRs. We should make separate (optional) precommit jobs
that run the same build commands. That will give a more clear history and
allow trivial configuration of which jobs deserve alert emails and which
are not a problem. This is easy but I've been waiting to do it after Gradle
migration.

On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik  wrote:

> Romain, would you like to host/plan/run the Gradle fixit day?
>
> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath 
> wrote:
>
>> +1 for the general idea of fixit day/week for Gradle.
>>
>> Agree with what Łukasz said. Some of these performance tests are new and
>> are flaky due to other issues that were discovered during the process of
>> adding the test.
>>
>> I think the high level blocker is updating performance testing framework
>> to use Gradle. This will involve adding Gradle-based logic for invoking
>> PerfKitBenchmaker, for example [1], and updating PerfKitBenchmarker to
>> issue a Gradle command for running the benchmark [2]. First task will be to
>> find the work needed here and updating the relevant JIRA [3].
>>
>> Thanks,
>> Cham
>>
>> [1]  https://github.com/apache/beam/blob/master/sdks/
>> java/io/google-cloud-platform/pom.xml#L79
>> [2] https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/
>> master/perfkitbenchmarker/beam_benchmark_helper.py
>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>
>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy 
>> wrote:
>>
>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles :
>>>

 Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
 spam level the performance tests are mostly not healthy anyhow. So is there
 any high level blocker to switching them or is it just someone sitting down
 with each one?


>>> I thought I could share my point of view here as I am working on the
>>> Performance Test part for a while now. I wouldn't say those are "mostly not
>>> healthy". The situation is as follows:
>>>
>>> - totally not working: Python, Spark performance tests (don't know why,
>>> I'm not maintaining those tests. Should we disable them?)
>>> - flaky: the recently re-enabled JDBC Performance Test. It's seems to be
>>> flaky mostly due to: https://issues.apache.org/jira/browse/BEAM-3798
>>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>>> TFRecordIO
>>>
>>> Also, some test failures are caused due to pending PRs (we sometimes run
>>> concrete tests to check if PRs won't break them). This also causes failures
>>> sometimes.
>>>
>>> I can help with switching the Performance tests to Gradle as this part
>>> seems to be free to take.
>>>
>>>
>>> Łukasz
>>>
>>>
>

Re: Gradle status

[Kenneth Knowles]

I would like to briefly re-focus this discussion and suggest that we merge
https://github.com/apache/beam/pull/4814.

The only material objection I've heard is that it means the precommit no
longer tests exactly what is built for release. It is a valid concern, but
we have mvn postcommit so the coverage is not lost. The benefits of quicker
reviews and less Jenkins congestion seem worth it to me.

Kenn

On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau 
wrote:

> @Luskasz: not sure Im the best to host it since I know more gradle
> internals that user interface/ecosystem but happy to help. Will also
> require a "sudo" merger for this day to merge fixes asap - guess we can
> bypass reviews or have a fast cycle plan for this day to avoid it to be a
> week?
>
> Le 8 mars 2018 21:08, "Kenneth Knowles"  a écrit :
>
> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven, and
> what I mean is that I have added enough hints that it works OOTB already.
> The rest of my instructions are just how you should override IntelliJ's
> defaults to have a proper dev env - mostly just about storing files outside
> the git clone.
>
>
> Hmm it is always super slow here and not as integrated as maven which is
> smooth thanks to the combination of idea build and maven plugin config
> read. Import is also faster cause it just reads meta and doesnt run
> anything. Hope it is a "a few times" issues at the moment but not yet sure.
>
>
>
> @Łukasz - yea I just mean the ones that I find in my email and browsing
> https://builds.apache.org/view/A-D/view/Beam/ which are the ones you
> classify as "totally not working". I would love to disable these.
>
> On the subject of running things on a pending PR - we should not run
> postcommit jobs on PRs. We should make separate (optional) precommit jobs
> that run the same build commands. That will give a more clear history and
> allow trivial configuration of which jobs deserve alert emails and which
> are not a problem. This is easy but I've been waiting to do it after Gradle
> migration.
>
> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik  wrote:
>
>> Romain, would you like to host/plan/run the Gradle fixit day?
>>
>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath > > wrote:
>>
>>> +1 for the general idea of fixit day/week for Gradle.
>>>
>>> Agree with what Łukasz said. Some of these performance tests are new and
>>> are flaky due to other issues that were discovered during the process of
>>> adding the test.
>>>
>>> I think the high level blocker is updating performance testing framework
>>> to use Gradle. This will involve adding Gradle-based logic for invoking
>>> PerfKitBenchmaker, for example [1], and updating PerfKitBenchmarker to
>>> issue a Gradle command for running the benchmark [2]. First task will be to
>>> find the work needed here and updating the relevant JIRA [3].
>>>
>>> Thanks,
>>> Cham
>>>
>>> [1]
>>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>>> [2]
>>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>
>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy 
>>> wrote:
>>>
 2018-03-07 22:29 GMT+01:00 Kenneth Knowles :

>
> Based on https://builds.apache.org/view/A-D/view/Beam/ and our
> failure spam level the performance tests are mostly not healthy anyhow. So
> is there any high level blocker to switching them or is it just someone
> sitting down with each one?
>
>
 I thought I could share my point of view here as I am working on the
 Performance Test part for a while now. I wouldn't say those are "mostly not
 healthy". The situation is as follows:

 - totally not working: Python, Spark performance tests (don't know why,
 I'm not maintaining those tests. Should we disable them?)
 - flaky: the recently re-enabled JDBC Performance Test. It's seems to
 be flaky mostly due to: https://issues.apache.org/jira/browse/BEAM-3798
 - working well, rarely failing: AvroIO, TextIO, Compressed Text,
 TFRecordIO

 Also, some test failures are caused due to pending PRs (we sometimes
 run concrete tests to check if PRs won't break them). This also causes
 failures sometimes.

 I can help with switching the Performance tests to Gradle as this part
 seems to be free to take.


 Łukasz


>>
>

Re: Gradle status

[Romain Manni-Bucau]

Hi Kenneth,

For now maven covers the full needs of beam. If we start to have this kind
of PR we become dependent of the 2 builds which is what this thread is
about avoiding so tempted to say it must be a PR drop completely maven or
nothing as mentionned before.

Le 9 mars 2018 04:48, "Kenneth Knowles"  a écrit :

> I would like to briefly re-focus this discussion and suggest that we merge
> https://github.com/apache/beam/pull/4814.
>
> The only material objection I've heard is that it means the precommit no
> longer tests exactly what is built for release. It is a valid concern, but
> we have mvn postcommit so the coverage is not lost. The benefits of quicker
> reviews and less Jenkins congestion seem worth it to me.
>
> Kenn
>
> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau 
> wrote:
>
>> @Luskasz: not sure Im the best to host it since I know more gradle
>> internals that user interface/ecosystem but happy to help. Will also
>> require a "sudo" merger for this day to merge fixes asap - guess we can
>> bypass reviews or have a fast cycle plan for this day to avoid it to be a
>> week?
>>
>> Le 8 mars 2018 21:08, "Kenneth Knowles"  a écrit :
>>
>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven, and
>> what I mean is that I have added enough hints that it works OOTB already.
>> The rest of my instructions are just how you should override IntelliJ's
>> defaults to have a proper dev env - mostly just about storing files outside
>> the git clone.
>>
>>
>> Hmm it is always super slow here and not as integrated as maven which is
>> smooth thanks to the combination of idea build and maven plugin config
>> read. Import is also faster cause it just reads meta and doesnt run
>> anything. Hope it is a "a few times" issues at the moment but not yet sure.
>>
>>
>>
>> @Łukasz - yea I just mean the ones that I find in my email and browsing
>> https://builds.apache.org/view/A-D/view/Beam/ which are the ones you
>> classify as "totally not working". I would love to disable these.
>>
>> On the subject of running things on a pending PR - we should not run
>> postcommit jobs on PRs. We should make separate (optional) precommit jobs
>> that run the same build commands. That will give a more clear history and
>> allow trivial configuration of which jobs deserve alert emails and which
>> are not a problem. This is easy but I've been waiting to do it after Gradle
>> migration.
>>
>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik  wrote:
>>
>>> Romain, would you like to host/plan/run the Gradle fixit day?
>>>
>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <
>>> chamik...@google.com> wrote:
>>>
 +1 for the general idea of fixit day/week for Gradle.

 Agree with what Łukasz said. Some of these performance tests are new
 and are flaky due to other issues that were discovered during the process
 of adding the test.

 I think the high level blocker is updating performance testing
 framework to use Gradle. This will involve adding Gradle-based logic for
 invoking PerfKitBenchmaker, for example [1], and updating
 PerfKitBenchmarker to issue a Gradle command for running the benchmark [2].
 First task will be to find the work needed here and updating the relevant
 JIRA [3].

 Thanks,
 Cham

 [1]  https://github.com/apache/beam/blob/master/sdks/
 java/io/google-cloud-platform/pom.xml#L79
 [2] https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/
 master/perfkitbenchmarker/beam_benchmark_helper.py
 [3] https://issues.apache.org/jira/browse/BEAM-3251

 On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy 
 wrote:

> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles :
>
>>
>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our
>> failure spam level the performance tests are mostly not healthy anyhow. 
>> So
>> is there any high level blocker to switching them or is it just someone
>> sitting down with each one?
>>
>>
> I thought I could share my point of view here as I am working on the
> Performance Test part for a while now. I wouldn't say those are "mostly 
> not
> healthy". The situation is as follows:
>
> - totally not working: Python, Spark performance tests (don't know
> why, I'm not maintaining those tests. Should we disable them?)
> - flaky: the recently re-enabled JDBC Performance Test. It's seems to
> be flaky mostly due to: https://issues.apache.org/
> jira/browse/BEAM-3798
> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
> TFRecordIO
>
> Also, some test failures are caused due to pending PRs (we sometimes
> run concrete tests to check if PRs won't break them). This also causes
> failures sometimes.
>
> I can help with switching the Performance tests to Gradle as this part
> seems to be free to take.
>
>
> Łukasz
>
>
>>>
>>

Re: Gradle status

[Lukasz Cwik]

Based upon the criteria that was discussed on the mailing list[1], I would
agree with Kenn about merging PR/4814 (drop Java Maven precommit).

1:
https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E

On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau 
wrote:

> Hi Kenneth,
>
> For now maven covers the full needs of beam. If we start to have this kind
> of PR we become dependent of the 2 builds which is what this thread is
> about avoiding so tempted to say it must be a PR drop completely maven or
> nothing as mentionned before.
>
> Le 9 mars 2018 04:48, "Kenneth Knowles"  a écrit :
>
>> I would like to briefly re-focus this discussion and suggest that we
>> merge https://github.com/apache/beam/pull/4814.
>>
>> The only material objection I've heard is that it means the precommit no
>> longer tests exactly what is built for release. It is a valid concern, but
>> we have mvn postcommit so the coverage is not lost. The benefits of quicker
>> reviews and less Jenkins congestion seem worth it to me.
>>
>> Kenn
>>
>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau 
>> wrote:
>>
>>> @Luskasz: not sure Im the best to host it since I know more gradle
>>> internals that user interface/ecosystem but happy to help. Will also
>>> require a "sudo" merger for this day to merge fixes asap - guess we can
>>> bypass reviews or have a fast cycle plan for this day to avoid it to be a
>>> week?
>>>
>>> Le 8 mars 2018 21:08, "Kenneth Knowles"  a écrit :
>>>
>>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven, and
>>> what I mean is that I have added enough hints that it works OOTB already.
>>> The rest of my instructions are just how you should override IntelliJ's
>>> defaults to have a proper dev env - mostly just about storing files outside
>>> the git clone.
>>>
>>>
>>> Hmm it is always super slow here and not as integrated as maven which is
>>> smooth thanks to the combination of idea build and maven plugin config
>>> read. Import is also faster cause it just reads meta and doesnt run
>>> anything. Hope it is a "a few times" issues at the moment but not yet sure.
>>>
>>>
>>>
>>> @Łukasz - yea I just mean the ones that I find in my email and browsing
>>> https://builds.apache.org/view/A-D/view/Beam/ which are the ones you
>>> classify as "totally not working". I would love to disable these.
>>>
>>> On the subject of running things on a pending PR - we should not run
>>> postcommit jobs on PRs. We should make separate (optional) precommit jobs
>>> that run the same build commands. That will give a more clear history and
>>> allow trivial configuration of which jobs deserve alert emails and which
>>> are not a problem. This is easy but I've been waiting to do it after Gradle
>>> migration.
>>>
>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik  wrote:
>>>
 Romain, would you like to host/plan/run the Gradle fixit day?

 On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <
 chamik...@google.com> wrote:

> +1 for the general idea of fixit day/week for Gradle.
>
> Agree with what Łukasz said. Some of these performance tests are new
> and are flaky due to other issues that were discovered during the process
> of adding the test.
>
> I think the high level blocker is updating performance testing
> framework to use Gradle. This will involve adding Gradle-based logic for
> invoking PerfKitBenchmaker, for example [1], and updating
> PerfKitBenchmarker to issue a Gradle command for running the benchmark 
> [2].
> First task will be to find the work needed here and updating the relevant
> JIRA [3].
>
> Thanks,
> Cham
>
> [1]  https://github.com/apache/beam/blob/master/sdks/java/
> io/google-cloud-platform/pom.xml#L79
> [2] https://github.com/GoogleCloudPlatform/PerfKitBenchmarke
> r/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
> [3] https://issues.apache.org/jira/browse/BEAM-3251
>
> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy 
> wrote:
>
>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles :
>>
>>>
>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our
>>> failure spam level the performance tests are mostly not healthy anyhow. 
>>> So
>>> is there any high level blocker to switching them or is it just someone
>>> sitting down with each one?
>>>
>>>
>> I thought I could share my point of view here as I am working on the
>> Performance Test part for a while now. I wouldn't say those are "mostly 
>> not
>> healthy". The situation is as follows:
>>
>> - totally not working: Python, Spark performance tests (don't know
>> why, I'm not maintaining those tests. Should we disable them?)
>> - flaky: the recently re-enabled JDBC Performance Test. It's seems to
>> be flaky mostly due to: https://issues.apache.org/
>> jira/browse/BEAM-3798
>>

Re: Gradle status

[Kenneth Knowles]

Indeed, we've already had the discussion a couple of times and I think the
criteria are clearly met. Incremental progress is a good thing and we
shouldn't block it.

OTOH I see where Romain is coming from and I have a good example that
supports a slightly different action. Consider
https://github.com/apache/beam/pull/4740 which fixes some errors in how we
use dependency mechanisms.

This PR is green except that I need to fix some Maven pom slightly more.
That is throwaway work. I would love to just not have to do it. But
removing the precommit does not actually make the PR OK to merge. It would
cause postcommits to fail.

We can hope such situations are rare. I think I tend to be hit by this more
often than most, as I work with the project build health quite a bit.

Here is a proposal to support these things: instead of deleting the job in
#4814, move it to not run automatically but only via a phrase. In fact, you
could migrate it to be the manually-invoked version of the postcommit job
as we've discussed a couple times. Then if someone is working on the build
in something like #4740 they can invoke it manually.

Kenn

On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik  wrote:

> Based upon the criteria that was discussed on the mailing list[1], I would
> agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>
> 1:
> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>
> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau 
> wrote:
>
>> Hi Kenneth,
>>
>> For now maven covers the full needs of beam. If we start to have this
>> kind of PR we become dependent of the 2 builds which is what this thread is
>> about avoiding so tempted to say it must be a PR drop completely maven or
>> nothing as mentionned before.
>>
>> Le 9 mars 2018 04:48, "Kenneth Knowles"  a écrit :
>>
>>> I would like to briefly re-focus this discussion and suggest that we
>>> merge https://github.com/apache/beam/pull/4814.
>>>
>>> The only material objection I've heard is that it means the precommit no
>>> longer tests exactly what is built for release. It is a valid concern, but
>>> we have mvn postcommit so the coverage is not lost. The benefits of quicker
>>> reviews and less Jenkins congestion seem worth it to me.
>>>
>>> Kenn
>>>
>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 @Luskasz: not sure Im the best to host it since I know more gradle
 internals that user interface/ecosystem but happy to help. Will also
 require a "sudo" merger for this day to merge fixes asap - guess we can
 bypass reviews or have a fast cycle plan for this day to avoid it to be a
 week?

 Le 8 mars 2018 21:08, "Kenneth Knowles"  a écrit :

 @Romain - Idea/IntelliJ is great with Gradle, way better than Maven,
 and what I mean is that I have added enough hints that it works OOTB
 already. The rest of my instructions are just how you should override
 IntelliJ's defaults to have a proper dev env - mostly just about storing
 files outside the git clone.


 Hmm it is always super slow here and not as integrated as maven which
 is smooth thanks to the combination of idea build and maven plugin config
 read. Import is also faster cause it just reads meta and doesnt run
 anything. Hope it is a "a few times" issues at the moment but not yet sure.



 @Łukasz - yea I just mean the ones that I find in my email and
 browsing https://builds.apache.org/view/A-D/view/Beam/ which are the
 ones you classify as "totally not working". I would love to disable these.

 On the subject of running things on a pending PR - we should not run
 postcommit jobs on PRs. We should make separate (optional) precommit jobs
 that run the same build commands. That will give a more clear history and
 allow trivial configuration of which jobs deserve alert emails and which
 are not a problem. This is easy but I've been waiting to do it after Gradle
 migration.

 On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik  wrote:

> Romain, would you like to host/plan/run the Gradle fixit day?
>
> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <
> chamik...@google.com> wrote:
>
>> +1 for the general idea of fixit day/week for Gradle.
>>
>> Agree with what Łukasz said. Some of these performance tests are new
>> and are flaky due to other issues that were discovered during the process
>> of adding the test.
>>
>> I think the high level blocker is updating performance testing
>> framework to use Gradle. This will involve adding Gradle-based logic for
>> invoking PerfKitBenchmaker, for example [1], and updating
>> PerfKitBenchmarker to issue a Gradle command for running the benchmark 
>> [2].
>> First task will be to find the work needed here and updating the relevant
>> JIRA [3].
>>>

Re: Gradle status

[Lukasz Cwik]

Ken, I'm probably not seeing something but how does using the PreCommit as
a proxy improve upon just running the post commit via the phrase it already
supports ('Run Java PostCommit')?

On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles  wrote:

> Indeed, we've already had the discussion a couple of times and I think the
> criteria are clearly met. Incremental progress is a good thing and we
> shouldn't block it.
>
> OTOH I see where Romain is coming from and I have a good example that
> supports a slightly different action. Consider https://github.com/apache/
> beam/pull/4740 which fixes some errors in how we use dependency
> mechanisms.
>
> This PR is green except that I need to fix some Maven pom slightly more.
> That is throwaway work. I would love to just not have to do it. But
> removing the precommit does not actually make the PR OK to merge. It would
> cause postcommits to fail.
>
> We can hope such situations are rare. I think I tend to be hit by this
> more often than most, as I work with the project build health quite a bit.
>
> Here is a proposal to support these things: instead of deleting the job in
> #4814, move it to not run automatically but only via a phrase. In fact, you
> could migrate it to be the manually-invoked version of the postcommit job
> as we've discussed a couple times. Then if someone is working on the build
> in something like #4740 they can invoke it manually.
>
> Kenn
>
> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik  wrote:
>
>> Based upon the criteria that was discussed on the mailing list[1], I
>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>
>> 1: https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f
>> 6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>
>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau > > wrote:
>>
>>> Hi Kenneth,
>>>
>>> For now maven covers the full needs of beam. If we start to have this
>>> kind of PR we become dependent of the 2 builds which is what this thread is
>>> about avoiding so tempted to say it must be a PR drop completely maven or
>>> nothing as mentionned before.
>>>
>>> Le 9 mars 2018 04:48, "Kenneth Knowles"  a écrit :
>>>
 I would like to briefly re-focus this discussion and suggest that we
 merge https://github.com/apache/beam/pull/4814.

 The only material objection I've heard is that it means the precommit
 no longer tests exactly what is built for release. It is a valid concern,
 but we have mvn postcommit so the coverage is not lost. The benefits of
 quicker reviews and less Jenkins congestion seem worth it to me.

 Kenn

 On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> @Luskasz: not sure Im the best to host it since I know more gradle
> internals that user interface/ecosystem but happy to help. Will also
> require a "sudo" merger for this day to merge fixes asap - guess we can
> bypass reviews or have a fast cycle plan for this day to avoid it to be a
> week?
>
> Le 8 mars 2018 21:08, "Kenneth Knowles"  a écrit :
>
> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven,
> and what I mean is that I have added enough hints that it works OOTB
> already. The rest of my instructions are just how you should override
> IntelliJ's defaults to have a proper dev env - mostly just about storing
> files outside the git clone.
>
>
> Hmm it is always super slow here and not as integrated as maven which
> is smooth thanks to the combination of idea build and maven plugin config
> read. Import is also faster cause it just reads meta and doesnt run
> anything. Hope it is a "a few times" issues at the moment but not yet 
> sure.
>
>
>
> @Łukasz - yea I just mean the ones that I find in my email and
> browsing https://builds.apache.org/view/A-D/view/Beam/ which are the
> ones you classify as "totally not working". I would love to disable these.
>
> On the subject of running things on a pending PR - we should not run
> postcommit jobs on PRs. We should make separate (optional) precommit jobs
> that run the same build commands. That will give a more clear history and
> allow trivial configuration of which jobs deserve alert emails and which
> are not a problem. This is easy but I've been waiting to do it after 
> Gradle
> migration.
>
> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik  wrote:
>
>> Romain, would you like to host/plan/run the Gradle fixit day?
>>
>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <
>> chamik...@google.com> wrote:
>>
>>> +1 for the general idea of fixit day/week for Gradle.
>>>
>>> Agree with what Łukasz said. Some of these performance tests are new
>>> and are flaky due to other issues that were discovered during the 
>>> process
>>> of adding the test.
>>>
>>> I thin

Re: Gradle status

[Kenneth Knowles]

Separate history (for easy dashboarding, health stats, etc) and
notification (email to dev@ for postcommits, nothing for precommits) for
pre & post commit targets.

A post commit failure is always a problem to be triaged at high priority,
while a precommit failure is just a natural occurrence.

On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik  wrote:

> Ken, I'm probably not seeing something but how does using the PreCommit as
> a proxy improve upon just running the post commit via the phrase it already
> supports ('Run Java PostCommit')?
>
> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles  wrote:
>
>> Indeed, we've already had the discussion a couple of times and I think
>> the criteria are clearly met. Incremental progress is a good thing and we
>> shouldn't block it.
>>
>> OTOH I see where Romain is coming from and I have a good example that
>> supports a slightly different action. Consider
>> https://github.com/apache/beam/pull/4740 which fixes some errors in how
>> we use dependency mechanisms.
>>
>> This PR is green except that I need to fix some Maven pom slightly more.
>> That is throwaway work. I would love to just not have to do it. But
>> removing the precommit does not actually make the PR OK to merge. It would
>> cause postcommits to fail.
>>
>> We can hope such situations are rare. I think I tend to be hit by this
>> more often than most, as I work with the project build health quite a bit.
>>
>> Here is a proposal to support these things: instead of deleting the job
>> in #4814, move it to not run automatically but only via a phrase. In fact,
>> you could migrate it to be the manually-invoked version of the postcommit
>> job as we've discussed a couple times. Then if someone is working on the
>> build in something like #4740 they can invoke it manually.
>>
>> Kenn
>>
>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik  wrote:
>>
>>> Based upon the criteria that was discussed on the mailing list[1], I
>>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>>
>>> 1:
>>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>>
>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Hi Kenneth,

 For now maven covers the full needs of beam. If we start to have this
 kind of PR we become dependent of the 2 builds which is what this thread is
 about avoiding so tempted to say it must be a PR drop completely maven or
 nothing as mentionned before.

 Le 9 mars 2018 04:48, "Kenneth Knowles"  a écrit :

> I would like to briefly re-focus this discussion and suggest that we
> merge https://github.com/apache/beam/pull/4814.
>
> The only material objection I've heard is that it means the precommit
> no longer tests exactly what is built for release. It is a valid concern,
> but we have mvn postcommit so the coverage is not lost. The benefits of
> quicker reviews and less Jenkins congestion seem worth it to me.
>
> Kenn
>
> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> @Luskasz: not sure Im the best to host it since I know more gradle
>> internals that user interface/ecosystem but happy to help. Will also
>> require a "sudo" merger for this day to merge fixes asap - guess we can
>> bypass reviews or have a fast cycle plan for this day to avoid it to be a
>> week?
>>
>> Le 8 mars 2018 21:08, "Kenneth Knowles"  a écrit :
>>
>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven,
>> and what I mean is that I have added enough hints that it works OOTB
>> already. The rest of my instructions are just how you should override
>> IntelliJ's defaults to have a proper dev env - mostly just about storing
>> files outside the git clone.
>>
>>
>> Hmm it is always super slow here and not as integrated as maven which
>> is smooth thanks to the combination of idea build and maven plugin config
>> read. Import is also faster cause it just reads meta and doesnt run
>> anything. Hope it is a "a few times" issues at the moment but not yet 
>> sure.
>>
>>
>>
>> @Łukasz - yea I just mean the ones that I find in my email and
>> browsing https://builds.apache.org/view/A-D/view/Beam/ which are the
>> ones you classify as "totally not working". I would love to disable 
>> these.
>>
>> On the subject of running things on a pending PR - we should not run
>> postcommit jobs on PRs. We should make separate (optional) precommit jobs
>> that run the same build commands. That will give a more clear history and
>> allow trivial configuration of which jobs deserve alert emails and which
>> are not a problem. This is easy but I've been waiting to do it after 
>> Gradle
>> migration.
>>
>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik

Re: Gradle status

[Romain Manni-Bucau]

Is having a 2 days period to "merge all mvn related pr we can" next week an
option? If so it can be enough and well fix gradle in the fix day (goal
being to reduce pr number a lot).

Le 9 mars 2018 20:40, "Kenneth Knowles"  a écrit :

> Separate history (for easy dashboarding, health stats, etc) and
> notification (email to dev@ for postcommits, nothing for precommits) for
> pre & post commit targets.
>
> A post commit failure is always a problem to be triaged at high priority,
> while a precommit failure is just a natural occurrence.
>
> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik  wrote:
>
>> Ken, I'm probably not seeing something but how does using the PreCommit
>> as a proxy improve upon just running the post commit via the phrase it
>> already supports ('Run Java PostCommit')?
>>
>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles  wrote:
>>
>>> Indeed, we've already had the discussion a couple of times and I think
>>> the criteria are clearly met. Incremental progress is a good thing and we
>>> shouldn't block it.
>>>
>>> OTOH I see where Romain is coming from and I have a good example that
>>> supports a slightly different action. Consider
>>> https://github.com/apache/beam/pull/4740 which fixes some errors in how
>>> we use dependency mechanisms.
>>>
>>> This PR is green except that I need to fix some Maven pom slightly more.
>>> That is throwaway work. I would love to just not have to do it. But
>>> removing the precommit does not actually make the PR OK to merge. It would
>>> cause postcommits to fail.
>>>
>>> We can hope such situations are rare. I think I tend to be hit by this
>>> more often than most, as I work with the project build health quite a bit.
>>>
>>> Here is a proposal to support these things: instead of deleting the job
>>> in #4814, move it to not run automatically but only via a phrase. In fact,
>>> you could migrate it to be the manually-invoked version of the postcommit
>>> job as we've discussed a couple times. Then if someone is working on the
>>> build in something like #4740 they can invoke it manually.
>>>
>>> Kenn
>>>
>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik  wrote:
>>>
 Based upon the criteria that was discussed on the mailing list[1], I
 would agree with Kenn about merging PR/4814 (drop Java Maven precommit).

 1: https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f
 6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E

 On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Hi Kenneth,
>
> For now maven covers the full needs of beam. If we start to have this
> kind of PR we become dependent of the 2 builds which is what this thread 
> is
> about avoiding so tempted to say it must be a PR drop completely maven or
> nothing as mentionned before.
>
> Le 9 mars 2018 04:48, "Kenneth Knowles"  a écrit :
>
>> I would like to briefly re-focus this discussion and suggest that we
>> merge https://github.com/apache/beam/pull/4814.
>>
>> The only material objection I've heard is that it means the precommit
>> no longer tests exactly what is built for release. It is a valid concern,
>> but we have mvn postcommit so the coverage is not lost. The benefits of
>> quicker reviews and less Jenkins congestion seem worth it to me.
>>
>> Kenn
>>
>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> @Luskasz: not sure Im the best to host it since I know more gradle
>>> internals that user interface/ecosystem but happy to help. Will also
>>> require a "sudo" merger for this day to merge fixes asap - guess we can
>>> bypass reviews or have a fast cycle plan for this day to avoid it to be 
>>> a
>>> week?
>>>
>>> Le 8 mars 2018 21:08, "Kenneth Knowles"  a écrit :
>>>
>>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven,
>>> and what I mean is that I have added enough hints that it works OOTB
>>> already. The rest of my instructions are just how you should override
>>> IntelliJ's defaults to have a proper dev env - mostly just about storing
>>> files outside the git clone.
>>>
>>>
>>> Hmm it is always super slow here and not as integrated as maven
>>> which is smooth thanks to the combination of idea build and maven plugin
>>> config read. Import is also faster cause it just reads meta and doesnt 
>>> run
>>> anything. Hope it is a "a few times" issues at the moment but not yet 
>>> sure.
>>>
>>>
>>>
>>> @Łukasz - yea I just mean the ones that I find in my email and
>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which are
>>> the ones you classify as "totally not working". I would love to disable
>>> these.
>>>
>>> On the subject of running things on a pending PR - we should not run
>>> postcommit jobs on PRs. We shoul

Re: Gradle status

[Lukasz Cwik]

Based upon your description it seems as though you would rather have a way
to run existing postcommits without it impacting dashboard/health
stats/notifications/ (We have just run the PostCommits on PRs for
additional validation (like upgrading the Dataflow container image)).

I don't think that keeping the current Java PreCommit as a proxy for the
the Java PostCommit is the right way to go but I also don't have the time
to implement what your actually asking for.
It seems more likely that migrating the PostCommit to Gradle will be less
work then adding the functionality but your argument where the PreCommit is
a proxy for the Java PostCommit also applies to the ValidatesRunner
PostCommits and so forth requiring even more migration to happen before you
don't have to worry about maintaining Maven/breaking post commits.

I'm fine with leaving both the Java/Gradle PreCommits running for now and
hopefully as more of the PostCommits are migrated off we will be able to
remove it.

On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles  wrote:

> Separate history (for easy dashboarding, health stats, etc) and
> notification (email to dev@ for postcommits, nothing for precommits) for
> pre & post commit targets.
>
> A post commit failure is always a problem to be triaged at high priority,
> while a precommit failure is just a natural occurrence.
>
> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik  wrote:
>
>> Ken, I'm probably not seeing something but how does using the PreCommit
>> as a proxy improve upon just running the post commit via the phrase it
>> already supports ('Run Java PostCommit')?
>>
>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles  wrote:
>>
>>> Indeed, we've already had the discussion a couple of times and I think
>>> the criteria are clearly met. Incremental progress is a good thing and we
>>> shouldn't block it.
>>>
>>> OTOH I see where Romain is coming from and I have a good example that
>>> supports a slightly different action. Consider
>>> https://github.com/apache/beam/pull/4740 which fixes some errors in how
>>> we use dependency mechanisms.
>>>
>>> This PR is green except that I need to fix some Maven pom slightly more.
>>> That is throwaway work. I would love to just not have to do it. But
>>> removing the precommit does not actually make the PR OK to merge. It would
>>> cause postcommits to fail.
>>>
>>> We can hope such situations are rare. I think I tend to be hit by this
>>> more often than most, as I work with the project build health quite a bit.
>>>
>>> Here is a proposal to support these things: instead of deleting the job
>>> in #4814, move it to not run automatically but only via a phrase. In fact,
>>> you could migrate it to be the manually-invoked version of the postcommit
>>> job as we've discussed a couple times. Then if someone is working on the
>>> build in something like #4740 they can invoke it manually.
>>>
>>> Kenn
>>>
>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik  wrote:
>>>
 Based upon the criteria that was discussed on the mailing list[1], I
 would agree with Kenn about merging PR/4814 (drop Java Maven precommit).

 1: https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f
 6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E

 On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Hi Kenneth,
>
> For now maven covers the full needs of beam. If we start to have this
> kind of PR we become dependent of the 2 builds which is what this thread 
> is
> about avoiding so tempted to say it must be a PR drop completely maven or
> nothing as mentionned before.
>
> Le 9 mars 2018 04:48, "Kenneth Knowles"  a écrit :
>
>> I would like to briefly re-focus this discussion and suggest that we
>> merge https://github.com/apache/beam/pull/4814.
>>
>> The only material objection I've heard is that it means the precommit
>> no longer tests exactly what is built for release. It is a valid concern,
>> but we have mvn postcommit so the coverage is not lost. The benefits of
>> quicker reviews and less Jenkins congestion seem worth it to me.
>>
>> Kenn
>>
>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> @Luskasz: not sure Im the best to host it since I know more gradle
>>> internals that user interface/ecosystem but happy to help. Will also
>>> require a "sudo" merger for this day to merge fixes asap - guess we can
>>> bypass reviews or have a fast cycle plan for this day to avoid it to be 
>>> a
>>> week?
>>>
>>> Le 8 mars 2018 21:08, "Kenneth Knowles"  a écrit :
>>>
>>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven,
>>> and what I mean is that I have added enough hints that it works OOTB
>>> already. The rest of my instructions are just how you should override
>>> IntelliJ's defaults to have a prope

Re: Gradle status

[Kenneth Knowles]

On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik  wrote:

> Based upon your description it seems as though you would rather have a way
> to run existing postcommits without it impacting dashboard/health
> stats/notifications/ (We have just run the PostCommits on PRs for
> additional validation (like upgrading the Dataflow container image)).
>

Yes, that is exactly what I have described.

I don't think that keeping the current Java PreCommit as a proxy for the
> the Java PostCommit is the right way to go but I also don't have the time
> to implement what your actually asking for.
>

Mostly I thought this might be very easy based on the fact that they are
nearly identical. If not, oh well.

Kenn


It seems more likely that migrating the PostCommit to Gradle will be less
> work then adding the functionality but your argument where the PreCommit is
> a proxy for the Java PostCommit also applies to the ValidatesRunner
> PostCommits and so forth requiring even more migration to happen before you
> don't have to worry about maintaining Maven/breaking post commits.
>
> I'm fine with leaving both the Java/Gradle PreCommits running for now and
> hopefully as more of the PostCommits are migrated off we will be able to
> remove it.
>
> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles  wrote:
>
>> Separate history (for easy dashboarding, health stats, etc) and
>> notification (email to dev@ for postcommits, nothing for precommits) for
>> pre & post commit targets.
>>
>> A post commit failure is always a problem to be triaged at high priority,
>> while a precommit failure is just a natural occurrence.
>>
>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik  wrote:
>>
>>> Ken, I'm probably not seeing something but how does using the PreCommit
>>> as a proxy improve upon just running the post commit via the phrase it
>>> already supports ('Run Java PostCommit')?
>>>
>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles  wrote:
>>>
 Indeed, we've already had the discussion a couple of times and I think
 the criteria are clearly met. Incremental progress is a good thing and we
 shouldn't block it.

 OTOH I see where Romain is coming from and I have a good example that
 supports a slightly different action. Consider
 https://github.com/apache/beam/pull/4740 which fixes some errors in
 how we use dependency mechanisms.

 This PR is green except that I need to fix some Maven pom slightly
 more. That is throwaway work. I would love to just not have to do it. But
 removing the precommit does not actually make the PR OK to merge. It would
 cause postcommits to fail.

 We can hope such situations are rare. I think I tend to be hit by this
 more often than most, as I work with the project build health quite a bit.

 Here is a proposal to support these things: instead of deleting the job
 in #4814, move it to not run automatically but only via a phrase. In fact,
 you could migrate it to be the manually-invoked version of the postcommit
 job as we've discussed a couple times. Then if someone is working on the
 build in something like #4740 they can invoke it manually.

 Kenn

 On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik  wrote:

> Based upon the criteria that was discussed on the mailing list[1], I
> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>
> 1:
> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>
> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Hi Kenneth,
>>
>> For now maven covers the full needs of beam. If we start to have this
>> kind of PR we become dependent of the 2 builds which is what this thread 
>> is
>> about avoiding so tempted to say it must be a PR drop completely maven or
>> nothing as mentionned before.
>>
>> Le 9 mars 2018 04:48, "Kenneth Knowles"  a écrit :
>>
>>> I would like to briefly re-focus this discussion and suggest that we
>>> merge https://github.com/apache/beam/pull/4814.
>>>
>>> The only material objection I've heard is that it means the
>>> precommit no longer tests exactly what is built for release. It is a 
>>> valid
>>> concern, but we have mvn postcommit so the coverage is not lost. The
>>> benefits of quicker reviews and less Jenkins congestion seem worth it 
>>> to me.
>>>
>>> Kenn
>>>
>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 @Luskasz: not sure Im the best to host it since I know more gradle
 internals that user interface/ecosystem but happy to help. Will also
 require a "sudo" merger for this day to merge fixes asap - guess we can
 bypass reviews or have a fast cycle plan for this day to avoid it to 
 be a
 week?
>>>

Re: Gradle status

[Romain Manni-Bucau]

hey guys,

2.4 is out, do we plan to drop all maven descriptors tomorrow or on monday?


Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book


2018-03-09 21:42 GMT+01:00 Kenneth Knowles :

> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik  wrote:
>
>> Based upon your description it seems as though you would rather have a
>> way to run existing postcommits without it impacting dashboard/health
>> stats/notifications/ (We have just run the PostCommits on PRs for
>> additional validation (like upgrading the Dataflow container image)).
>>
>
> Yes, that is exactly what I have described.
>
> I don't think that keeping the current Java PreCommit as a proxy for the
>> the Java PostCommit is the right way to go but I also don't have the time
>> to implement what your actually asking for.
>>
>
> Mostly I thought this might be very easy based on the fact that they are
> nearly identical. If not, oh well.
>
> Kenn
>
>
> It seems more likely that migrating the PostCommit to Gradle will be less
>> work then adding the functionality but your argument where the PreCommit is
>> a proxy for the Java PostCommit also applies to the ValidatesRunner
>> PostCommits and so forth requiring even more migration to happen before you
>> don't have to worry about maintaining Maven/breaking post commits.
>>
>> I'm fine with leaving both the Java/Gradle PreCommits running for now and
>> hopefully as more of the PostCommits are migrated off we will be able to
>> remove it.
>>
>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles  wrote:
>>
>>> Separate history (for easy dashboarding, health stats, etc) and
>>> notification (email to dev@ for postcommits, nothing for precommits)
>>> for pre & post commit targets.
>>>
>>> A post commit failure is always a problem to be triaged at high
>>> priority, while a precommit failure is just a natural occurrence.
>>>
>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik  wrote:
>>>
 Ken, I'm probably not seeing something but how does using the PreCommit
 as a proxy improve upon just running the post commit via the phrase it
 already supports ('Run Java PostCommit')?

 On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles 
 wrote:

> Indeed, we've already had the discussion a couple of times and I think
> the criteria are clearly met. Incremental progress is a good thing and we
> shouldn't block it.
>
> OTOH I see where Romain is coming from and I have a good example that
> supports a slightly different action. Consider
> https://github.com/apache/beam/pull/4740 which fixes some errors in
> how we use dependency mechanisms.
>
> This PR is green except that I need to fix some Maven pom slightly
> more. That is throwaway work. I would love to just not have to do it. But
> removing the precommit does not actually make the PR OK to merge. It would
> cause postcommits to fail.
>
> We can hope such situations are rare. I think I tend to be hit by this
> more often than most, as I work with the project build health quite a bit.
>
> Here is a proposal to support these things: instead of deleting the
> job in #4814, move it to not run automatically but only via a phrase. In
> fact, you could migrate it to be the manually-invoked version of the
> postcommit job as we've discussed a couple times. Then if someone is
> working on the build in something like #4740 they can invoke it manually.
>
> Kenn
>
> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik  wrote:
>
>> Based upon the criteria that was discussed on the mailing list[1], I
>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>
>> 1: https://lists.apache.org/thread.html/
>> 7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%
>> 3Cdev.beam.apache.org%3E
>>
>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Hi Kenneth,
>>>
>>> For now maven covers the full needs of beam. If we start to have
>>> this kind of PR we become dependent of the 2 builds which is what this
>>> thread is about avoiding so tempted to say it must be a PR drop 
>>> completely
>>> maven or nothing as mentionned before.
>>>
>>> Le 9 mars 2018 04:48, "Kenneth Knowles"  a écrit :
>>>
 I would like to briefly re-focus this discussion and suggest that
 we merge https://github.com/apache/beam/pull/4814.

 The only material objection I've heard is that it means the
 precommit no longer tests exactly what is built for release. It is a 
 valid
 concern, but we have mvn postcommit so the coverage

Re: Gradle status

[Ismaël Mejía]

I don't think that removing all maven descriptors was the expected
path, no ? Or even a good idea at this moment.

I understood that what we were going to do was to replace
incrementally the CI until we cover the whole maven functionality and
then remove it, from looking at the JIRA ticket
https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
covering the complete maven functionality in particular for the
release part that could be the biggest pain point.


On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
 wrote:
> hey guys,
>
> 2.4 is out, do we plan to drop all maven descriptors tomorrow or on monday?
>
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
> 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
>>
>> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik  wrote:
>>>
>>> Based upon your description it seems as though you would rather have a
>>> way to run existing postcommits without it impacting dashboard/health
>>> stats/notifications/ (We have just run the PostCommits on PRs for
>>> additional validation (like upgrading the Dataflow container image)).
>>
>>
>> Yes, that is exactly what I have described.
>>
>>> I don't think that keeping the current Java PreCommit as a proxy for the
>>> the Java PostCommit is the right way to go but I also don't have the time to
>>> implement what your actually asking for.
>>
>>
>> Mostly I thought this might be very easy based on the fact that they are
>> nearly identical. If not, oh well.
>>
>> Kenn
>>
>>
>>> It seems more likely that migrating the PostCommit to Gradle will be less
>>> work then adding the functionality but your argument where the PreCommit is
>>> a proxy for the Java PostCommit also applies to the ValidatesRunner
>>> PostCommits and so forth requiring even more migration to happen before you
>>> don't have to worry about maintaining Maven/breaking post commits.
>>>
>>> I'm fine with leaving both the Java/Gradle PreCommits running for now and
>>> hopefully as more of the PostCommits are migrated off we will be able to
>>> remove it.
>>>
>>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles  wrote:

 Separate history (for easy dashboarding, health stats, etc) and
 notification (email to dev@ for postcommits, nothing for precommits) for 
 pre
 & post commit targets.

 A post commit failure is always a problem to be triaged at high
 priority, while a precommit failure is just a natural occurrence.

 On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik  wrote:
>
> Ken, I'm probably not seeing something but how does using the PreCommit
> as a proxy improve upon just running the post commit via the phrase it
> already supports ('Run Java PostCommit')?
>
> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles 
> wrote:
>>
>> Indeed, we've already had the discussion a couple of times and I think
>> the criteria are clearly met. Incremental progress is a good thing and we
>> shouldn't block it.
>>
>> OTOH I see where Romain is coming from and I have a good example that
>> supports a slightly different action. Consider
>> https://github.com/apache/beam/pull/4740 which fixes some errors in how 
>> we
>> use dependency mechanisms.
>>
>> This PR is green except that I need to fix some Maven pom slightly
>> more. That is throwaway work. I would love to just not have to do it. But
>> removing the precommit does not actually make the PR OK to merge. It 
>> would
>> cause postcommits to fail.
>>
>> We can hope such situations are rare. I think I tend to be hit by this
>> more often than most, as I work with the project build health quite a 
>> bit.
>>
>> Here is a proposal to support these things: instead of deleting the
>> job in #4814, move it to not run automatically but only via a phrase. In
>> fact, you could migrate it to be the manually-invoked version of the
>> postcommit job as we've discussed a couple times. Then if someone is 
>> working
>> on the build in something like #4740 they can invoke it manually.
>>
>> Kenn
>>
>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik  wrote:
>>>
>>> Based upon the criteria that was discussed on the mailing list[1], I
>>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>>
>>> 1:
>>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>>
>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau
>>>  wrote:

 Hi Kenneth,

 For now maven covers the full needs of beam. If we start to have
 this kind of PR we become dependent of the 2 builds which is what this
 thread is about avoiding so tempted to say it must be a PR drop 
 completely
 maven or nothing as mentionned before.

 Le 9 mars 2018 04:48, "Ke

Re: Gradle status

[Romain Manni-Bucau]

Not really Ismaël, this thread was about to do it at once and have 1 day to
fix it all.

As mentionned at the very beginning nobody maintains the 2 system so it
must stop after months so either we drop maven or gradle *at once*
or we keep a state where each dev does what he wants and the build system
just doesn't work.

2018-03-22 15:42 GMT+01:00 Ismaël Mejía :

> I don't think that removing all maven descriptors was the expected
> path, no ? Or even a good idea at this moment.
>
> I understood that what we were going to do was to replace
> incrementally the CI until we cover the whole maven functionality and
> then remove it, from looking at the JIRA ticket
> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
> covering the complete maven functionality in particular for the
> release part that could be the biggest pain point.
>
>
> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>  wrote:
> > hey guys,
> >
> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
> monday?
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
> >>
> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik  wrote:
> >>>
> >>> Based upon your description it seems as though you would rather have a
> >>> way to run existing postcommits without it impacting dashboard/health
> >>> stats/notifications/ (We have just run the PostCommits on PRs for
> >>> additional validation (like upgrading the Dataflow container image)).
> >>
> >>
> >> Yes, that is exactly what I have described.
> >>
> >>> I don't think that keeping the current Java PreCommit as a proxy for
> the
> >>> the Java PostCommit is the right way to go but I also don't have the
> time to
> >>> implement what your actually asking for.
> >>
> >>
> >> Mostly I thought this might be very easy based on the fact that they are
> >> nearly identical. If not, oh well.
> >>
> >> Kenn
> >>
> >>
> >>> It seems more likely that migrating the PostCommit to Gradle will be
> less
> >>> work then adding the functionality but your argument where the
> PreCommit is
> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
> >>> PostCommits and so forth requiring even more migration to happen
> before you
> >>> don't have to worry about maintaining Maven/breaking post commits.
> >>>
> >>> I'm fine with leaving both the Java/Gradle PreCommits running for now
> and
> >>> hopefully as more of the PostCommits are migrated off we will be able
> to
> >>> remove it.
> >>>
> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles 
> wrote:
> 
>  Separate history (for easy dashboarding, health stats, etc) and
>  notification (email to dev@ for postcommits, nothing for precommits)
> for pre
>  & post commit targets.
> 
>  A post commit failure is always a problem to be triaged at high
>  priority, while a precommit failure is just a natural occurrence.
> 
>  On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik  wrote:
> >
> > Ken, I'm probably not seeing something but how does using the
> PreCommit
> > as a proxy improve upon just running the post commit via the phrase
> it
> > already supports ('Run Java PostCommit')?
> >
> > On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles 
> > wrote:
> >>
> >> Indeed, we've already had the discussion a couple of times and I
> think
> >> the criteria are clearly met. Incremental progress is a good thing
> and we
> >> shouldn't block it.
> >>
> >> OTOH I see where Romain is coming from and I have a good example
> that
> >> supports a slightly different action. Consider
> >> https://github.com/apache/beam/pull/4740 which fixes some errors
> in how we
> >> use dependency mechanisms.
> >>
> >> This PR is green except that I need to fix some Maven pom slightly
> >> more. That is throwaway work. I would love to just not have to do
> it. But
> >> removing the precommit does not actually make the PR OK to merge.
> It would
> >> cause postcommits to fail.
> >>
> >> We can hope such situations are rare. I think I tend to be hit by
> this
> >> more often than most, as I work with the project build health quite
> a bit.
> >>
> >> Here is a proposal to support these things: instead of deleting the
> >> job in #4814, move it to not run automatically but only via a
> phrase. In
> >> fact, you could migrate it to be the manually-invoked version of the
> >> postcommit job as we've discussed a couple times. Then if someone
> is working
> >> on the build in something like #4740 they can invoke it manually.
> >>
> >> Kenn
> >>
> >> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik 
> wrote:
> >>>
> >>> Based upon the criteria that was discussed on the mailing list[1],
> I
> >>> would agree with Kenn about merging PR/4814 (drop Java Maven
> precommit).
> >>>
> >>> 1:
> >>>

Re: Gradle status

[Henning Rohde]

My understanding was the same as Ismaël's. I don't think breaking the build
with a large known gaps (but not fully known cost) is practical. Also, most
items in the jira are not even assigned yet.


On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau 
wrote:

> Not really Ismaël, this thread was about to do it at once and have 1 day
> to fix it all.
>
> As mentionned at the very beginning nobody maintains the 2 system so it
> must stop after months so either we drop maven or gradle *at once*
> or we keep a state where each dev does what he wants and the build system
> just doesn't work.
>
> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :
>
>> I don't think that removing all maven descriptors was the expected
>> path, no ? Or even a good idea at this moment.
>>
>> I understood that what we were going to do was to replace
>> incrementally the CI until we cover the whole maven functionality and
>> then remove it, from looking at the JIRA ticket
>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
>> covering the complete maven functionality in particular for the
>> release part that could be the biggest pain point.
>>
>>
>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>  wrote:
>> > hey guys,
>> >
>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
>> monday?
>> >
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >
>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
>> >>
>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik  wrote:
>> >>>
>> >>> Based upon your description it seems as though you would rather have a
>> >>> way to run existing postcommits without it impacting dashboard/health
>> >>> stats/notifications/ (We have just run the PostCommits on PRs for
>> >>> additional validation (like upgrading the Dataflow container image)).
>> >>
>> >>
>> >> Yes, that is exactly what I have described.
>> >>
>> >>> I don't think that keeping the current Java PreCommit as a proxy for
>> the
>> >>> the Java PostCommit is the right way to go but I also don't have the
>> time to
>> >>> implement what your actually asking for.
>> >>
>> >>
>> >> Mostly I thought this might be very easy based on the fact that they
>> are
>> >> nearly identical. If not, oh well.
>> >>
>> >> Kenn
>> >>
>> >>
>> >>> It seems more likely that migrating the PostCommit to Gradle will be
>> less
>> >>> work then adding the functionality but your argument where the
>> PreCommit is
>> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
>> >>> PostCommits and so forth requiring even more migration to happen
>> before you
>> >>> don't have to worry about maintaining Maven/breaking post commits.
>> >>>
>> >>> I'm fine with leaving both the Java/Gradle PreCommits running for now
>> and
>> >>> hopefully as more of the PostCommits are migrated off we will be able
>> to
>> >>> remove it.
>> >>>
>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles 
>> wrote:
>> 
>>  Separate history (for easy dashboarding, health stats, etc) and
>>  notification (email to dev@ for postcommits, nothing for
>> precommits) for pre
>>  & post commit targets.
>> 
>>  A post commit failure is always a problem to be triaged at high
>>  priority, while a precommit failure is just a natural occurrence.
>> 
>>  On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik 
>> wrote:
>> >
>> > Ken, I'm probably not seeing something but how does using the
>> PreCommit
>> > as a proxy improve upon just running the post commit via the phrase
>> it
>> > already supports ('Run Java PostCommit')?
>> >
>> > On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles 
>> > wrote:
>> >>
>> >> Indeed, we've already had the discussion a couple of times and I
>> think
>> >> the criteria are clearly met. Incremental progress is a good thing
>> and we
>> >> shouldn't block it.
>> >>
>> >> OTOH I see where Romain is coming from and I have a good example
>> that
>> >> supports a slightly different action. Consider
>> >> https://github.com/apache/beam/pull/4740 which fixes some errors
>> in how we
>> >> use dependency mechanisms.
>> >>
>> >> This PR is green except that I need to fix some Maven pom slightly
>> >> more. That is throwaway work. I would love to just not have to do
>> it. But
>> >> removing the precommit does not actually make the PR OK to merge.
>> It would
>> >> cause postcommits to fail.
>> >>
>> >> We can hope such situations are rare. I think I tend to be hit by
>> this
>> >> more often than most, as I work with the project build health
>> quite a bit.
>> >>
>> >> Here is a proposal to support these things: instead of deleting the
>> >> job in #4814, move it to not run automatically but only via a
>> phrase. In
>> >> fact, you could migrate it to be the manually-invoked version of
>> the
>> >> postcommit job as we've discussed a couple times. Then if 

Re: Gradle status

[Lukasz Cwik]

Romain, from the previous discussions several people agreed that running a
fixit that migrated Maven to Gradle over a 1 or 2 day period was worthwhile
but there was nobody in the community with the time commitment to organize
and run it so the status quo plan remained where the Gradle migration will
happen incrementally.


On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde  wrote:

> My understanding was the same as Ismaël's. I don't think breaking the
> build with a large known gaps (but not fully known cost) is practical.
> Also, most items in the jira are not even assigned yet.
>
>
> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau 
> wrote:
>
>> Not really Ismaël, this thread was about to do it at once and have 1 day
>> to fix it all.
>>
>> As mentionned at the very beginning nobody maintains the 2 system so it
>> must stop after months so either we drop maven or gradle *at once*
>> or we keep a state where each dev does what he wants and the build system
>> just doesn't work.
>>
>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :
>>
>>> I don't think that removing all maven descriptors was the expected
>>> path, no ? Or even a good idea at this moment.
>>>
>>> I understood that what we were going to do was to replace
>>> incrementally the CI until we cover the whole maven functionality and
>>> then remove it, from looking at the JIRA ticket
>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
>>> covering the complete maven functionality in particular for the
>>> release part that could be the biggest pain point.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>>  wrote:
>>> > hey guys,
>>> >
>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
>>> monday?
>>> >
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >
>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
>>> >>
>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik  wrote:
>>> >>>
>>> >>> Based upon your description it seems as though you would rather have
>>> a
>>> >>> way to run existing postcommits without it impacting dashboard/health
>>> >>> stats/notifications/ (We have just run the PostCommits on PRs for
>>> >>> additional validation (like upgrading the Dataflow container image)).
>>> >>
>>> >>
>>> >> Yes, that is exactly what I have described.
>>> >>
>>> >>> I don't think that keeping the current Java PreCommit as a proxy for
>>> the
>>> >>> the Java PostCommit is the right way to go but I also don't have the
>>> time to
>>> >>> implement what your actually asking for.
>>> >>
>>> >>
>>> >> Mostly I thought this might be very easy based on the fact that they
>>> are
>>> >> nearly identical. If not, oh well.
>>> >>
>>> >> Kenn
>>> >>
>>> >>
>>> >>> It seems more likely that migrating the PostCommit to Gradle will be
>>> less
>>> >>> work then adding the functionality but your argument where the
>>> PreCommit is
>>> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
>>> >>> PostCommits and so forth requiring even more migration to happen
>>> before you
>>> >>> don't have to worry about maintaining Maven/breaking post commits.
>>> >>>
>>> >>> I'm fine with leaving both the Java/Gradle PreCommits running for
>>> now and
>>> >>> hopefully as more of the PostCommits are migrated off we will be
>>> able to
>>> >>> remove it.
>>> >>>
>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles 
>>> wrote:
>>> 
>>>  Separate history (for easy dashboarding, health stats, etc) and
>>>  notification (email to dev@ for postcommits, nothing for
>>> precommits) for pre
>>>  & post commit targets.
>>> 
>>>  A post commit failure is always a problem to be triaged at high
>>>  priority, while a precommit failure is just a natural occurrence.
>>> 
>>>  On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik 
>>> wrote:
>>> >
>>> > Ken, I'm probably not seeing something but how does using the
>>> PreCommit
>>> > as a proxy improve upon just running the post commit via the
>>> phrase it
>>> > already supports ('Run Java PostCommit')?
>>> >
>>> > On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles 
>>> > wrote:
>>> >>
>>> >> Indeed, we've already had the discussion a couple of times and I
>>> think
>>> >> the criteria are clearly met. Incremental progress is a good
>>> thing and we
>>> >> shouldn't block it.
>>> >>
>>> >> OTOH I see where Romain is coming from and I have a good example
>>> that
>>> >> supports a slightly different action. Consider
>>> >> https://github.com/apache/beam/pull/4740 which fixes some errors
>>> in how we
>>> >> use dependency mechanisms.
>>> >>
>>> >> This PR is green except that I need to fix some Maven pom slightly
>>> >> more. That is throwaway work. I would love to just not have to do
>>> it. But
>>> >> removing the precommit does not actually make the PR OK to merge.
>>> It would
>>> >> cause postcommits to 

Re: Gradle status

[Romain Manni-Bucau]

@Valentyn: concretely any user can PR and be part of that process so anyone
can do it wrong (me first)
@Luskasz, Hennking: fine but what do we do? last months prooved beam cant
maintain 2 systems, easier with that state is then to drop gradle since it
is a 0 investment compared to the opposite


Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book


2018-03-22 17:24 GMT+01:00 Lukasz Cwik :

> Romain, from the previous discussions several people agreed that running a
> fixit that migrated Maven to Gradle over a 1 or 2 day period was worthwhile
> but there was nobody in the community with the time commitment to organize
> and run it so the status quo plan remained where the Gradle migration will
> happen incrementally.
>
>
> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde  wrote:
>
>> My understanding was the same as Ismaël's. I don't think breaking the
>> build with a large known gaps (but not fully known cost) is practical.
>> Also, most items in the jira are not even assigned yet.
>>
>>
>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau 
>> wrote:
>>
>>> Not really Ismaël, this thread was about to do it at once and have 1 day
>>> to fix it all.
>>>
>>> As mentionned at the very beginning nobody maintains the 2 system so it
>>> must stop after months so either we drop maven or gradle *at once*
>>> or we keep a state where each dev does what he wants and the build
>>> system just doesn't work.
>>>
>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :
>>>
 I don't think that removing all maven descriptors was the expected
 path, no ? Or even a good idea at this moment.

 I understood that what we were going to do was to replace
 incrementally the CI until we cover the whole maven functionality and
 then remove it, from looking at the JIRA ticket
 https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
 covering the complete maven functionality in particular for the
 release part that could be the biggest pain point.


 On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
  wrote:
 > hey guys,
 >
 > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
 monday?
 >
 >
 > Romain Manni-Bucau
 > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
 >
 > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
 >>
 >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik 
 wrote:
 >>>
 >>> Based upon your description it seems as though you would rather
 have a
 >>> way to run existing postcommits without it impacting
 dashboard/health
 >>> stats/notifications/ (We have just run the PostCommits on PRs
 for
 >>> additional validation (like upgrading the Dataflow container
 image)).
 >>
 >>
 >> Yes, that is exactly what I have described.
 >>
 >>> I don't think that keeping the current Java PreCommit as a proxy
 for the
 >>> the Java PostCommit is the right way to go but I also don't have
 the time to
 >>> implement what your actually asking for.
 >>
 >>
 >> Mostly I thought this might be very easy based on the fact that they
 are
 >> nearly identical. If not, oh well.
 >>
 >> Kenn
 >>
 >>
 >>> It seems more likely that migrating the PostCommit to Gradle will
 be less
 >>> work then adding the functionality but your argument where the
 PreCommit is
 >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
 >>> PostCommits and so forth requiring even more migration to happen
 before you
 >>> don't have to worry about maintaining Maven/breaking post commits.
 >>>
 >>> I'm fine with leaving both the Java/Gradle PreCommits running for
 now and
 >>> hopefully as more of the PostCommits are migrated off we will be
 able to
 >>> remove it.
 >>>
 >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles 
 wrote:
 
  Separate history (for easy dashboarding, health stats, etc) and
  notification (email to dev@ for postcommits, nothing for
 precommits) for pre
  & post commit targets.
 
  A post commit failure is always a problem to be triaged at high
  priority, while a precommit failure is just a natural occurrence.
 
  On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik 
 wrote:
 >
 > Ken, I'm probably not seeing something but how does using the
 PreCommit
 > as a proxy improve upon just running the post commit via the
 phrase it
 > already supports ('Run Java PostCommit')?
 >
 > On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles 
>>

Re: Gradle status

[Lukasz Cwik]

what do we do? "Gradle migration will happen incrementally."

"last months prooved beam cant maintain 2 systems, easier with that state
is then to drop gradle since it is a 0 investment compared to the opposite"
Its unfortunate that you feel this way but many people do not share your
opinion.


On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau 
wrote:

> @Valentyn: concretely any user can PR and be part of that process so
> anyone can do it wrong (me first)
> @Luskasz, Hennking: fine but what do we do? last months prooved beam cant
> maintain 2 systems, easier with that state is then to drop gradle since it
> is a 0 investment compared to the opposite
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>
>> Romain, from the previous discussions several people agreed that running
>> a fixit that migrated Maven to Gradle over a 1 or 2 day period was
>> worthwhile but there was nobody in the community with the time commitment
>> to organize and run it so the status quo plan remained where the Gradle
>> migration will happen incrementally.
>>
>>
>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde  wrote:
>>
>>> My understanding was the same as Ismaël's. I don't think breaking the
>>> build with a large known gaps (but not fully known cost) is practical.
>>> Also, most items in the jira are not even assigned yet.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Not really Ismaël, this thread was about to do it at once and have 1
 day to fix it all.

 As mentionned at the very beginning nobody maintains the 2 system so it
 must stop after months so either we drop maven or gradle *at once*
 or we keep a state where each dev does what he wants and the build
 system just doesn't work.

 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :

> I don't think that removing all maven descriptors was the expected
> path, no ? Or even a good idea at this moment.
>
> I understood that what we were going to do was to replace
> incrementally the CI until we cover the whole maven functionality and
> then remove it, from looking at the JIRA ticket
> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
> covering the complete maven functionality in particular for the
> release part that could be the biggest pain point.
>
>
> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>  wrote:
> > hey guys,
> >
> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
> monday?
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
> >>
> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik 
> wrote:
> >>>
> >>> Based upon your description it seems as though you would rather
> have a
> >>> way to run existing postcommits without it impacting
> dashboard/health
> >>> stats/notifications/ (We have just run the PostCommits on PRs
> for
> >>> additional validation (like upgrading the Dataflow container
> image)).
> >>
> >>
> >> Yes, that is exactly what I have described.
> >>
> >>> I don't think that keeping the current Java PreCommit as a proxy
> for the
> >>> the Java PostCommit is the right way to go but I also don't have
> the time to
> >>> implement what your actually asking for.
> >>
> >>
> >> Mostly I thought this might be very easy based on the fact that
> they are
> >> nearly identical. If not, oh well.
> >>
> >> Kenn
> >>
> >>
> >>> It seems more likely that migrating the PostCommit to Gradle will
> be less
> >>> work then adding the functionality but your argument where the
> PreCommit is
> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
> >>> PostCommits and so forth requiring even more migration to happen
> before you
> >>> don't have to worry about maintaining Maven/breaking post commits.
> >>>
> >>> I'm fine with leaving both the Java/Gradle PreCommits running for
> now and
> >>> hopefully as more of the PostCommits are migrated off we will be
> able to
> >>> remove it.
> >>>
> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles 
> wrote:
> 
>  Separate history (for easy dashboarding, health stats, etc) and
>  notification (email to dev@ for postcommits, nothing for
> precommits) for pre
>  & post commit targets.
> 
>  A post commit fa

Re: Gradle status

[Romain Manni-Bucau]

2018-03-22 17:45 GMT+01:00 Lukasz Cwik :

> what do we do? "Gradle migration will happen incrementally."
>
> "last months prooved beam cant maintain 2 systems, easier with that state
> is then to drop gradle since it is a 0 investment compared to the opposite"
> Its unfortunate that you feel this way but many people do not share your
> opinion.
>

And a lot do so when a project is 50-50 it is time to act.

Incrementally kind of means never (makes 4 months and nothing really
changed in PRs and habits, gradle maintener(s) are still alone)


>
>
> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau 
> wrote:
>
>> @Valentyn: concretely any user can PR and be part of that process so
>> anyone can do it wrong (me first)
>> @Luskasz, Hennking: fine but what do we do? last months prooved beam cant
>> maintain 2 systems, easier with that state is then to drop gradle since it
>> is a 0 investment compared to the opposite
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>>
>>> Romain, from the previous discussions several people agreed that running
>>> a fixit that migrated Maven to Gradle over a 1 or 2 day period was
>>> worthwhile but there was nobody in the community with the time commitment
>>> to organize and run it so the status quo plan remained where the Gradle
>>> migration will happen incrementally.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde 
>>> wrote:
>>>
 My understanding was the same as Ismaël's. I don't think breaking the
 build with a large known gaps (but not fully known cost) is practical.
 Also, most items in the jira are not even assigned yet.


 On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Not really Ismaël, this thread was about to do it at once and have 1
> day to fix it all.
>
> As mentionned at the very beginning nobody maintains the 2 system so
> it must stop after months so either we drop maven or gradle *at once*
> or we keep a state where each dev does what he wants and the build
> system just doesn't work.
>
> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :
>
>> I don't think that removing all maven descriptors was the expected
>> path, no ? Or even a good idea at this moment.
>>
>> I understood that what we were going to do was to replace
>> incrementally the CI until we cover the whole maven functionality and
>> then remove it, from looking at the JIRA ticket
>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
>> covering the complete maven functionality in particular for the
>> release part that could be the biggest pain point.
>>
>>
>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>  wrote:
>> > hey guys,
>> >
>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
>> monday?
>> >
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >
>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
>> >>
>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik 
>> wrote:
>> >>>
>> >>> Based upon your description it seems as though you would rather
>> have a
>> >>> way to run existing postcommits without it impacting
>> dashboard/health
>> >>> stats/notifications/ (We have just run the PostCommits on PRs
>> for
>> >>> additional validation (like upgrading the Dataflow container
>> image)).
>> >>
>> >>
>> >> Yes, that is exactly what I have described.
>> >>
>> >>> I don't think that keeping the current Java PreCommit as a proxy
>> for the
>> >>> the Java PostCommit is the right way to go but I also don't have
>> the time to
>> >>> implement what your actually asking for.
>> >>
>> >>
>> >> Mostly I thought this might be very easy based on the fact that
>> they are
>> >> nearly identical. If not, oh well.
>> >>
>> >> Kenn
>> >>
>> >>
>> >>> It seems more likely that migrating the PostCommit to Gradle will
>> be less
>> >>> work then adding the functionality but your argument where the
>> PreCommit is
>> >>> a proxy for the Java PostCommit also applies to the
>> ValidatesRunner
>> >>> PostCommits and so forth requiring even more migration to happen
>> before you
>> >>> don't have to worry about maintaining Maven/breaking post commits.
>> >>>
>> >>> I'm fine with leaving both the Java/Gradle PreCommits running for
>> now and
>> >>> hopefully as more of the PostCommits are migrated off we 

Re: Gradle status

[Alan Myrvold]

I think the investment in gradle is worthwhile, and incrementally we will
continue to make progress. From what I've seem, gradle is a good fit for
this project and a path to a faster, more reliable build system.

pull/4812  creates the release
artifacts, although it is not hooked up yet with authentication.

I expect to help make incremental progress over the next month converting
some of the integration tests, but welcome incremental improvements from
others.



On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau 
wrote:

>
>
> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>
>> what do we do? "Gradle migration will happen incrementally."
>>
>> "last months prooved beam cant maintain 2 systems, easier with that
>> state is then to drop gradle since it is a 0 investment compared to the
>> opposite"
>> Its unfortunate that you feel this way but many people do not share your
>> opinion.
>>
>
> And a lot do so when a project is 50-50 it is time to act.
>
> Incrementally kind of means never (makes 4 months and nothing really
> changed in PRs and habits, gradle maintener(s) are still alone)
>
>
>>
>>
>> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau 
>> wrote:
>>
>>> @Valentyn: concretely any user can PR and be part of that process so
>>> anyone can do it wrong (me first)
>>> @Luskasz, Hennking: fine but what do we do? last months prooved beam
>>> cant maintain 2 systems, easier with that state is then to drop gradle
>>> since it is a 0 investment compared to the opposite
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau  |  Blog
>>>  | Old Blog
>>>  | Github
>>>  | LinkedIn
>>>  | Book
>>> 
>>>
>>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>>>
 Romain, from the previous discussions several people agreed that
 running a fixit that migrated Maven to Gradle over a 1 or 2 day period was
 worthwhile but there was nobody in the community with the time commitment
 to organize and run it so the status quo plan remained where the Gradle
 migration will happen incrementally.


 On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde 
 wrote:

> My understanding was the same as Ismaël's. I don't think breaking the
> build with a large known gaps (but not fully known cost) is practical.
> Also, most items in the jira are not even assigned yet.
>
>
> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Not really Ismaël, this thread was about to do it at once and have 1
>> day to fix it all.
>>
>> As mentionned at the very beginning nobody maintains the 2 system so
>> it must stop after months so either we drop maven or gradle *at once*
>> or we keep a state where each dev does what he wants and the build
>> system just doesn't work.
>>
>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :
>>
>>> I don't think that removing all maven descriptors was the expected
>>> path, no ? Or even a good idea at this moment.
>>>
>>> I understood that what we were going to do was to replace
>>> incrementally the CI until we cover the whole maven functionality and
>>> then remove it, from looking at the JIRA ticket
>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far
>>> from
>>> covering the complete maven functionality in particular for the
>>> release part that could be the biggest pain point.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>>  wrote:
>>> > hey guys,
>>> >
>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or
>>> on monday?
>>> >
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >
>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
>>> >>
>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik 
>>> wrote:
>>> >>>
>>> >>> Based upon your description it seems as though you would rather
>>> have a
>>> >>> way to run existing postcommits without it impacting
>>> dashboard/health
>>> >>> stats/notifications/ (We have just run the PostCommits on
>>> PRs for
>>> >>> additional validation (like upgrading the Dataflow container
>>> image)).
>>> >>
>>> >>
>>> >> Yes, that is exactly what I have described.
>>> >>
>>> >>> I don't think that keeping the current Java PreCommit as a proxy
>>> for the
>>> >>> the Java PostCommit is the right way to go but I also don't have
>>> the time to
>>> >>> implement what your actually asking for.
>>> >>
>>> >>
>>> >> Mostly I thought this might be very easy based on the fact that
>>> they are
>>

Re: Gradle status

[Romain Manni-Bucau]

Ok so to be clear for any contributor (which is the goal of this thread):
maven is still the main build system and no need to maintain gradle in PR
then until beam switches.

Im more than fine with that.

Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :

> I think the investment in gradle is worthwhile, and incrementally we will
> continue to make progress. From what I've seem, gradle is a good fit for
> this project and a path to a faster, more reliable build system.
>
> pull/4812  creates the release
> artifacts, although it is not hooked up yet with authentication.
>
> I expect to help make incremental progress over the next month converting
> some of the integration tests, but welcome incremental improvements from
> others.
>
>
>
> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau 
> wrote:
>
>>
>>
>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>>
>>> what do we do? "Gradle migration will happen incrementally."
>>>
>>> "last months prooved beam cant maintain 2 systems, easier with that
>>> state is then to drop gradle since it is a 0 investment compared to the
>>> opposite"
>>> Its unfortunate that you feel this way but many people do not share your
>>> opinion.
>>>
>>
>> And a lot do so when a project is 50-50 it is time to act.
>>
>> Incrementally kind of means never (makes 4 months and nothing really
>> changed in PRs and habits, gradle maintener(s) are still alone)
>>
>>
>>>
>>>
>>> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 @Valentyn: concretely any user can PR and be part of that process so
 anyone can do it wrong (me first)
 @Luskasz, Hennking: fine but what do we do? last months prooved beam
 cant maintain 2 systems, easier with that state is then to drop gradle
 since it is a 0 investment compared to the opposite


 Romain Manni-Bucau
 @rmannibucau  |  Blog
  | Old Blog
  | Github
  | LinkedIn
  | Book
 

 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :

> Romain, from the previous discussions several people agreed that
> running a fixit that migrated Maven to Gradle over a 1 or 2 day period was
> worthwhile but there was nobody in the community with the time commitment
> to organize and run it so the status quo plan remained where the Gradle
> migration will happen incrementally.
>
>
> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde 
> wrote:
>
>> My understanding was the same as Ismaël's. I don't think breaking the
>> build with a large known gaps (but not fully known cost) is practical.
>> Also, most items in the jira are not even assigned yet.
>>
>>
>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Not really Ismaël, this thread was about to do it at once and have 1
>>> day to fix it all.
>>>
>>> As mentionned at the very beginning nobody maintains the 2 system so
>>> it must stop after months so either we drop maven or gradle *at once*
>>> or we keep a state where each dev does what he wants and the build
>>> system just doesn't work.
>>>
>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :
>>>
 I don't think that removing all maven descriptors was the expected
 path, no ? Or even a good idea at this moment.

 I understood that what we were going to do was to replace
 incrementally the CI until we cover the whole maven functionality
 and
 then remove it, from looking at the JIRA ticket
 https://issues.apache.org/jira/browse/BEAM-3249 we are still far
 from
 covering the complete maven functionality in particular for the
 release part that could be the biggest pain point.


 On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
  wrote:
 > hey guys,
 >
 > 2.4 is out, do we plan to drop all maven descriptors tomorrow or
 on monday?
 >
 >
 > Romain Manni-Bucau
 > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
 >
 > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
 >>
 >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik 
 wrote:
 >>>
 >>> Based upon your description it seems as though you would rather
 have a
 >>> way to run existing postcommits without it impacting
 dashboard/health
 >>> stats/notifications/ (We have just run the PostCommits on
 PRs for
 >>> additional validation (like upgrading the Dataflow container
 image)).
 >>
 >>

Re: Gradle status

[Lukasz Cwik]

Romain, that is incorrect. Contributors are to maintain both systems until
Gradle replaces that functionality in Maven and then it only needs to be
maintained in Gradle.


On Thu, Mar 22, 2018 at 10:30 AM Romain Manni-Bucau 
wrote:

> Ok so to be clear for any contributor (which is the goal of this thread):
> maven is still the main build system and no need to maintain gradle in PR
> then until beam switches.
>
> Im more than fine with that.
>
> Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :
>
>> I think the investment in gradle is worthwhile, and incrementally we will
>> continue to make progress. From what I've seem, gradle is a good fit for
>> this project and a path to a faster, more reliable build system.
>>
>> pull/4812  creates the release
>> artifacts, although it is not hooked up yet with authentication.
>>
>> I expect to help make incremental progress over the next month converting
>> some of the integration tests, but welcome incremental improvements from
>> others.
>>
>>
>>
>> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau 
>> wrote:
>>
>>>
>>>
>>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>>>
 what do we do? "Gradle migration will happen incrementally."

 "last months prooved beam cant maintain 2 systems, easier with that
 state is then to drop gradle since it is a 0 investment compared to the
 opposite"
 Its unfortunate that you feel this way but many people do not share
 your opinion.

>>>
>>> And a lot do so when a project is 50-50 it is time to act.
>>>
>>> Incrementally kind of means never (makes 4 months and nothing really
>>> changed in PRs and habits, gradle maintener(s) are still alone)
>>>
>>>


 On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> @Valentyn: concretely any user can PR and be part of that process so
> anyone can do it wrong (me first)
> @Luskasz, Hennking: fine but what do we do? last months prooved beam
> cant maintain 2 systems, easier with that state is then to drop gradle
> since it is a 0 investment compared to the opposite
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>
>> Romain, from the previous discussions several people agreed that
>> running a fixit that migrated Maven to Gradle over a 1 or 2 day period 
>> was
>> worthwhile but there was nobody in the community with the time commitment
>> to organize and run it so the status quo plan remained where the Gradle
>> migration will happen incrementally.
>>
>>
>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde 
>> wrote:
>>
>>> My understanding was the same as Ismaël's. I don't think breaking
>>> the build with a large known gaps (but not fully known cost) is 
>>> practical.
>>> Also, most items in the jira are not even assigned yet.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Not really Ismaël, this thread was about to do it at once and have
 1 day to fix it all.

 As mentionned at the very beginning nobody maintains the 2 system
 so it must stop after months so either we drop maven or gradle *at 
 once*
 or we keep a state where each dev does what he wants and the build
 system just doesn't work.

 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :

> I don't think that removing all maven descriptors was the expected
> path, no ? Or even a good idea at this moment.
>
> I understood that what we were going to do was to replace
> incrementally the CI until we cover the whole maven functionality
> and
> then remove it, from looking at the JIRA ticket
> https://issues.apache.org/jira/browse/BEAM-3249 we are still far
> from
> covering the complete maven functionality in particular for the
> release part that could be the biggest pain point.
>
>
> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>  wrote:
> > hey guys,
> >
> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or
> on monday?
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
> >>
> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik 
> wrote:
> >>>
> 

Re: Gradle status

[Dan Halperin]

It seems that a few groups are talking past each other.

* A sizable contingent is interested in a move to Gradle -- it shows
promise, but the work is incomplete.
* Another contingent noticing the large burden of maintaining multiple
build systems. FWICT, both test suites have been broken quite a lot
recently, mainly the gradle ones, which is a cost to the community. This is
creating a barrier to entry for new contributors – especially those who
don't work in the same room or do their primary development in a different
repository.

I don't see this situation being resolved to anyone's satisfaction until
there's only one build system left. The onus is clearly on the Gradle
promoters to finish the work.

Luke made a suggestion 2.5 months ago that we should have a process vote if
this situation is untenable. It seems like we're there.

Thanks,
Dan

On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau 
wrote:

> Ok so to be clear for any contributor (which is the goal of this thread):
> maven is still the main build system and no need to maintain gradle in PR
> then until beam switches.
>
> Im more than fine with that.
>
> Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :
>
>> I think the investment in gradle is worthwhile, and incrementally we will
>> continue to make progress. From what I've seem, gradle is a good fit for
>> this project and a path to a faster, more reliable build system.
>>
>> pull/4812  creates the release
>> artifacts, although it is not hooked up yet with authentication.
>>
>> I expect to help make incremental progress over the next month converting
>> some of the integration tests, but welcome incremental improvements from
>> others.
>>
>>
>>
>> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau 
>> wrote:
>>
>>>
>>>
>>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>>>
 what do we do? "Gradle migration will happen incrementally."

 "last months prooved beam cant maintain 2 systems, easier with that
 state is then to drop gradle since it is a 0 investment compared to the
 opposite"
 Its unfortunate that you feel this way but many people do not share
 your opinion.

>>>
>>> And a lot do so when a project is 50-50 it is time to act.
>>>
>>> Incrementally kind of means never (makes 4 months and nothing really
>>> changed in PRs and habits, gradle maintener(s) are still alone)
>>>
>>>


 On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> @Valentyn: concretely any user can PR and be part of that process so
> anyone can do it wrong (me first)
> @Luskasz, Hennking: fine but what do we do? last months prooved beam
> cant maintain 2 systems, easier with that state is then to drop gradle
> since it is a 0 investment compared to the opposite
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>
>> Romain, from the previous discussions several people agreed that
>> running a fixit that migrated Maven to Gradle over a 1 or 2 day period 
>> was
>> worthwhile but there was nobody in the community with the time commitment
>> to organize and run it so the status quo plan remained where the Gradle
>> migration will happen incrementally.
>>
>>
>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde 
>> wrote:
>>
>>> My understanding was the same as Ismaël's. I don't think breaking
>>> the build with a large known gaps (but not fully known cost) is 
>>> practical.
>>> Also, most items in the jira are not even assigned yet.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Not really Ismaël, this thread was about to do it at once and have
 1 day to fix it all.

 As mentionned at the very beginning nobody maintains the 2 system
 so it must stop after months so either we drop maven or gradle *at 
 once*
 or we keep a state where each dev does what he wants and the build
 system just doesn't work.

 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :

> I don't think that removing all maven descriptors was the expected
> path, no ? Or even a good idea at this moment.
>
> I understood that what we were going to do was to replace
> incrementally the CI until we cover the whole maven functionality
> and
> then remove it, from looking at the JIRA ticket
> https://issues.apache.org/jira/browse/BEAM-3249 we are still far
>>

Re: Gradle status

[Romain Manni-Bucau]

Le 22 mars 2018 18:49, "Dan Halperin"  a écrit :

It seems that a few groups are talking past each other.

* A sizable contingent is interested in a move to Gradle -- it shows
promise, but the work is incomplete.
* Another contingent noticing the large burden of maintaining multiple
build systems. FWICT, both test suites have been broken quite a lot
recently, mainly the gradle ones, which is a cost to the community. This is
creating a barrier to entry for new contributors – especially those who
don't work in the same room or do their primary development in a different
repository.

I don't see this situation being resolved to anyone's satisfaction until
there's only one build system left. The onus is clearly on the Gradle
promoters to finish the work.

Luke made a suggestion 2.5 months ago that we should have a process vote if
this situation is untenable. It seems like we're there.


Yes but beam voted to move to gradle so we should but we shouldnt maintain
2 build systems for more than 2 months (weway overpassed that) and
therefore the vote should be cancelled or validated by an action.

I understand you want gradle but you dont want to pay the cost to move to
gradle, it doesnt work for the project do please another option
(rollbacking gradle or removing maven, all other options are negative for
the project and a pain for committers and contributors whatever you think).



Thanks,
Dan

On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau 
wrote:

> Ok so to be clear for any contributor (which is the goal of this thread):
> maven is still the main build system and no need to maintain gradle in PR
> then until beam switches.
>
> Im more than fine with that.
>
> Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :
>
>> I think the investment in gradle is worthwhile, and incrementally we will
>> continue to make progress. From what I've seem, gradle is a good fit for
>> this project and a path to a faster, more reliable build system.
>>
>> pull/4812  creates the release
>> artifacts, although it is not hooked up yet with authentication.
>>
>> I expect to help make incremental progress over the next month converting
>> some of the integration tests, but welcome incremental improvements from
>> others.
>>
>>
>>
>> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau 
>> wrote:
>>
>>>
>>>
>>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>>>
 what do we do? "Gradle migration will happen incrementally."

 "last months prooved beam cant maintain 2 systems, easier with that
 state is then to drop gradle since it is a 0 investment compared to the
 opposite"
 Its unfortunate that you feel this way but many people do not share
 your opinion.

>>>
>>> And a lot do so when a project is 50-50 it is time to act.
>>>
>>> Incrementally kind of means never (makes 4 months and nothing really
>>> changed in PRs and habits, gradle maintener(s) are still alone)
>>>
>>>


 On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> @Valentyn: concretely any user can PR and be part of that process so
> anyone can do it wrong (me first)
> @Luskasz, Hennking: fine but what do we do? last months prooved beam
> cant maintain 2 systems, easier with that state is then to drop gradle
> since it is a 0 investment compared to the opposite
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>
>> Romain, from the previous discussions several people agreed that
>> running a fixit that migrated Maven to Gradle over a 1 or 2 day period 
>> was
>> worthwhile but there was nobody in the community with the time commitment
>> to organize and run it so the status quo plan remained where the Gradle
>> migration will happen incrementally.
>>
>>
>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde 
>> wrote:
>>
>>> My understanding was the same as Ismaël's. I don't think breaking
>>> the build with a large known gaps (but not fully known cost) is 
>>> practical.
>>> Also, most items in the jira are not even assigned yet.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Not really Ismaël, this thread was about to do it at once and have
 1 day to fix it all.

 As mentionned at the very beginning nobody maintains the 2 system
 so it must stop after months so either we drop maven or gradle *at 
 once*
 or we keep a state where each dev does what he wants and 

Re: Gradle status

[Chamikara Jayalath]

I don't think incremental progress is a bad thing as long as we are making
progress towards the goal. Do we need better metrics (a weekly email ?)
about the progress towards moving everything to Gradle ? I agree with
others who pointed out that there are many unresolved JIRAs and simply
deleting Maven artifacts could break many things (for example, performance
tests).

Thanks,
Cham

On Thu, Mar 22, 2018 at 10:56 AM Romain Manni-Bucau 
wrote:

>
>
> Le 22 mars 2018 18:49, "Dan Halperin"  a écrit :
>
> It seems that a few groups are talking past each other.
>
> * A sizable contingent is interested in a move to Gradle -- it shows
> promise, but the work is incomplete.
> * Another contingent noticing the large burden of maintaining multiple
> build systems. FWICT, both test suites have been broken quite a lot
> recently, mainly the gradle ones, which is a cost to the community. This is
> creating a barrier to entry for new contributors – especially those who
> don't work in the same room or do their primary development in a different
> repository.
>
> I don't see this situation being resolved to anyone's satisfaction until
> there's only one build system left. The onus is clearly on the Gradle
> promoters to finish the work.
>
> Luke made a suggestion 2.5 months ago that we should have a process vote
> if this situation is untenable. It seems like we're there.
>
>
> Yes but beam voted to move to gradle so we should but we shouldnt maintain
> 2 build systems for more than 2 months (weway overpassed that) and
> therefore the vote should be cancelled or validated by an action.
>
> I understand you want gradle but you dont want to pay the cost to move to
> gradle, it doesnt work for the project do please another option
> (rollbacking gradle or removing maven, all other options are negative for
> the project and a pain for committers and contributors whatever you think).
>
>
>
> Thanks,
> Dan
>
> On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Ok so to be clear for any contributor (which is the goal of this thread):
>> maven is still the main build system and no need to maintain gradle in PR
>> then until beam switches.
>>
>> Im more than fine with that.
>>
>> Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :
>>
>>> I think the investment in gradle is worthwhile, and incrementally we
>>> will continue to make progress. From what I've seem, gradle is a good fit
>>> for this project and a path to a faster, more reliable build system.
>>>
>>> pull/4812  creates the
>>> release artifacts, although it is not hooked up yet with authentication.
>>>
>>> I expect to help make incremental progress over the next month
>>> converting some of the integration tests, but welcome incremental
>>> improvements from others.
>>>
>>>
>>>
>>> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>


 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :

> what do we do? "Gradle migration will happen incrementally."
>
> "last months prooved beam cant maintain 2 systems, easier with that
> state is then to drop gradle since it is a 0 investment compared to the
> opposite"
> Its unfortunate that you feel this way but many people do not share
> your opinion.
>

 And a lot do so when a project is 50-50 it is time to act.

 Incrementally kind of means never (makes 4 months and nothing really
 changed in PRs and habits, gradle maintener(s) are still alone)


>
>
> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> @Valentyn: concretely any user can PR and be part of that process so
>> anyone can do it wrong (me first)
>> @Luskasz, Hennking: fine but what do we do? last months prooved beam
>> cant maintain 2 systems, easier with that state is then to drop gradle
>> since it is a 0 investment compared to the opposite
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>>
>>> Romain, from the previous discussions several people agreed that
>>> running a fixit that migrated Maven to Gradle over a 1 or 2 day period 
>>> was
>>> worthwhile but there was nobody in the community with the time 
>>> commitment
>>> to organize and run it so the status quo plan remained where the Gradle
>>> migration will happen incrementally.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde 
>>> wrote:
>>>
 My understanding was the same as Ismaël's. I

Re: Gradle status

[Dan Halperin]

On Thu, Mar 22, 2018 at 11:19 AM, Chamikara Jayalath 
wrote:

> I don't think incremental progress is a bad thing as long as we are making
> progress towards the goal. Do we need better metrics (a weekly email ?)
> about the progress towards moving everything to Gradle ? I agree with
> others who pointed out that there are many unresolved JIRAs and simply
> deleting Maven artifacts could break many things (for example, performance
> tests).
>

The problem does not seem to be incremental progress on its face, or a lack
of metrics.

The problem is that there are two build systems with separate features and
issues, doubling (or worse) Jenkins cycles, mental effort, maintenance
burden, etc. It hurts the community and casual contributors.

As Luke suggested,
> A process vote can be happen if the in-between state is too painful to
maintain.

Given that the in-between state has lasted so long, and there is it may be
time.

Dan



>
> Thanks,
> Cham
>
>
> On Thu, Mar 22, 2018 at 10:56 AM Romain Manni-Bucau 
> wrote:
>
>>
>>
>> Le 22 mars 2018 18:49, "Dan Halperin"  a écrit :
>>
>> It seems that a few groups are talking past each other.
>>
>> * A sizable contingent is interested in a move to Gradle -- it shows
>> promise, but the work is incomplete.
>> * Another contingent noticing the large burden of maintaining multiple
>> build systems. FWICT, both test suites have been broken quite a lot
>> recently, mainly the gradle ones, which is a cost to the community. This is
>> creating a barrier to entry for new contributors – especially those who
>> don't work in the same room or do their primary development in a different
>> repository.
>>
>> I don't see this situation being resolved to anyone's satisfaction until
>> there's only one build system left. The onus is clearly on the Gradle
>> promoters to finish the work.
>>
>> Luke made a suggestion 2.5 months ago that we should have a process vote
>> if this situation is untenable. It seems like we're there.
>>
>>
>> Yes but beam voted to move to gradle so we should but we shouldnt
>> maintain 2 build systems for more than 2 months (weway overpassed that) and
>> therefore the vote should be cancelled or validated by an action.
>>
>> I understand you want gradle but you dont want to pay the cost to move to
>> gradle, it doesnt work for the project do please another option
>> (rollbacking gradle or removing maven, all other options are negative for
>> the project and a pain for committers and contributors whatever you think).
>>
>>
>>
>> Thanks,
>> Dan
>>
>> On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Ok so to be clear for any contributor (which is the goal of this
>>> thread): maven is still the main build system and no need to maintain
>>> gradle in PR then until beam switches.
>>>
>>> Im more than fine with that.
>>>
>>> Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :
>>>
 I think the investment in gradle is worthwhile, and incrementally we
 will continue to make progress. From what I've seem, gradle is a good fit
 for this project and a path to a faster, more reliable build system.

 pull/4812  creates the
 release artifacts, although it is not hooked up yet with authentication.

 I expect to help make incremental progress over the next month
 converting some of the integration tests, but welcome incremental
 improvements from others.



 On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

>
>
> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>
>> what do we do? "Gradle migration will happen incrementally."
>>
>> "last months prooved beam cant maintain 2 systems, easier with that
>> state is then to drop gradle since it is a 0 investment compared to the
>> opposite"
>> Its unfortunate that you feel this way but many people do not share
>> your opinion.
>>
>
> And a lot do so when a project is 50-50 it is time to act.
>
> Incrementally kind of means never (makes 4 months and nothing really
> changed in PRs and habits, gradle maintener(s) are still alone)
>
>
>>
>>
>> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> @Valentyn: concretely any user can PR and be part of that process so
>>> anyone can do it wrong (me first)
>>> @Luskasz, Hennking: fine but what do we do? last months prooved beam
>>> cant maintain 2 systems, easier with that state is then to drop gradle
>>> since it is a 0 investment compared to the opposite
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau  |  Blog
>>>  | Old Blog
>>>  | Github
>>>  | LinkedIn
>>>  | 

Re: Gradle status

[Reuven Lax]

Let's back up for a second.

Earlier in the thread we agreed to organize a community "fixit" day to try
and migrate remaining Maven items to Gradle. I had thought that Romain had
volunteered to run this, but reading back in the thread it appears that I
misunderstood this. I would suggest that we organize this first, and make
the concerted push to migrate the remaining items. After this is done, we
can evaluate the state we're left in and hold a process vote if necessary.

I can volunteer to help coordinate this fixit.

Reuven

On Thu, Mar 22, 2018 at 1:35 PM Dan Halperin  wrote:

> On Thu, Mar 22, 2018 at 11:19 AM, Chamikara Jayalath  > wrote:
>
>> I don't think incremental progress is a bad thing as long as we are
>> making progress towards the goal. Do we need better metrics (a weekly email
>> ?) about the progress towards moving everything to Gradle ? I agree with
>> others who pointed out that there are many unresolved JIRAs and simply
>> deleting Maven artifacts could break many things (for example, performance
>> tests).
>>
>
> The problem does not seem to be incremental progress on its face, or a
> lack of metrics.
>
> The problem is that there are two build systems with separate features and
> issues, doubling (or worse) Jenkins cycles, mental effort, maintenance
> burden, etc. It hurts the community and casual contributors.
>
> As Luke suggested,
> > A process vote can be happen if the in-between state is too painful to
> maintain.
>
> Given that the in-between state has lasted so long, and there is it may be
> time.
>
> Dan
>
>
>
>>
>> Thanks,
>> Cham
>>
>>
>> On Thu, Mar 22, 2018 at 10:56 AM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>>
>>>
>>> Le 22 mars 2018 18:49, "Dan Halperin"  a écrit :
>>>
>>> It seems that a few groups are talking past each other.
>>>
>>> * A sizable contingent is interested in a move to Gradle -- it shows
>>> promise, but the work is incomplete.
>>> * Another contingent noticing the large burden of maintaining multiple
>>> build systems. FWICT, both test suites have been broken quite a lot
>>> recently, mainly the gradle ones, which is a cost to the community. This is
>>> creating a barrier to entry for new contributors – especially those who
>>> don't work in the same room or do their primary development in a different
>>> repository.
>>>
>>> I don't see this situation being resolved to anyone's satisfaction until
>>> there's only one build system left. The onus is clearly on the Gradle
>>> promoters to finish the work.
>>>
>>> Luke made a suggestion 2.5 months ago that we should have a process vote
>>> if this situation is untenable. It seems like we're there.
>>>
>>>
>>> Yes but beam voted to move to gradle so we should but we shouldnt
>>> maintain 2 build systems for more than 2 months (weway overpassed that) and
>>> therefore the vote should be cancelled or validated by an action.
>>>
>>> I understand you want gradle but you dont want to pay the cost to move
>>> to gradle, it doesnt work for the project do please another option
>>> (rollbacking gradle or removing maven, all other options are negative for
>>> the project and a pain for committers and contributors whatever you think).
>>>
>>>
>>>
>>> Thanks,
>>> Dan
>>>
>>> On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Ok so to be clear for any contributor (which is the goal of this
 thread): maven is still the main build system and no need to maintain
 gradle in PR then until beam switches.

 Im more than fine with that.

 Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :

> I think the investment in gradle is worthwhile, and incrementally we
> will continue to make progress. From what I've seem, gradle is a good fit
> for this project and a path to a faster, more reliable build system.
>
> pull/4812  creates the
> release artifacts, although it is not hooked up yet with authentication.
>
> I expect to help make incremental progress over the next month
> converting some of the integration tests, but welcome incremental
> improvements from others.
>
>
>
> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>>
>>
>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>>
>>> what do we do? "Gradle migration will happen incrementally."
>>>
>>> "last months prooved beam cant maintain 2 systems, easier with that
>>> state is then to drop gradle since it is a 0 investment compared to the
>>> opposite"
>>> Its unfortunate that you feel this way but many people do not share
>>> your opinion.
>>>
>>
>> And a lot do so when a project is 50-50 it is time to act.
>>
>> Incrementally kind of means never (makes 4 months and nothing really
>> changed in PRs and habits, gradle maintener(s) are still alone)
>>
>>
>>>
>>>
>>> On

Re: Gradle status

[Romain Manni-Bucau]

Can work Reuven, where is the "todo" list? Thought we were done (= the
replacement was not blocking the dev) multiple times due to other threads
but reading this week mails it sounds like we are not yet here.


Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book


2018-03-22 22:51 GMT+01:00 Reuven Lax :

> Let's back up for a second.
>
> Earlier in the thread we agreed to organize a community "fixit" day to try
> and migrate remaining Maven items to Gradle. I had thought that Romain had
> volunteered to run this, but reading back in the thread it appears that I
> misunderstood this. I would suggest that we organize this first, and make
> the concerted push to migrate the remaining items. After this is done, we
> can evaluate the state we're left in and hold a process vote if necessary.
>
> I can volunteer to help coordinate this fixit.
>
> Reuven
>
> On Thu, Mar 22, 2018 at 1:35 PM Dan Halperin  wrote:
>
>> On Thu, Mar 22, 2018 at 11:19 AM, Chamikara Jayalath <
>> chamik...@google.com> wrote:
>>
>>> I don't think incremental progress is a bad thing as long as we are
>>> making progress towards the goal. Do we need better metrics (a weekly email
>>> ?) about the progress towards moving everything to Gradle ? I agree with
>>> others who pointed out that there are many unresolved JIRAs and simply
>>> deleting Maven artifacts could break many things (for example, performance
>>> tests).
>>>
>>
>> The problem does not seem to be incremental progress on its face, or a
>> lack of metrics.
>>
>> The problem is that there are two build systems with separate features
>> and issues, doubling (or worse) Jenkins cycles, mental effort, maintenance
>> burden, etc. It hurts the community and casual contributors.
>>
>> As Luke suggested,
>> > A process vote can be happen if the in-between state is too painful to
>> maintain.
>>
>> Given that the in-between state has lasted so long, and there is it may
>> be time.
>>
>> Dan
>>
>>
>>
>>>
>>> Thanks,
>>> Cham
>>>
>>>
>>> On Thu, Mar 22, 2018 at 10:56 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>


 Le 22 mars 2018 18:49, "Dan Halperin"  a écrit :

 It seems that a few groups are talking past each other.

 * A sizable contingent is interested in a move to Gradle -- it shows
 promise, but the work is incomplete.
 * Another contingent noticing the large burden of maintaining multiple
 build systems. FWICT, both test suites have been broken quite a lot
 recently, mainly the gradle ones, which is a cost to the community. This is
 creating a barrier to entry for new contributors – especially those who
 don't work in the same room or do their primary development in a different
 repository.

 I don't see this situation being resolved to anyone's satisfaction
 until there's only one build system left. The onus is clearly on the Gradle
 promoters to finish the work.

 Luke made a suggestion 2.5 months ago that we should have a process
 vote if this situation is untenable. It seems like we're there.


 Yes but beam voted to move to gradle so we should but we shouldnt
 maintain 2 build systems for more than 2 months (weway overpassed that) and
 therefore the vote should be cancelled or validated by an action.

 I understand you want gradle but you dont want to pay the cost to move
 to gradle, it doesnt work for the project do please another option
 (rollbacking gradle or removing maven, all other options are negative for
 the project and a pain for committers and contributors whatever you think).



 Thanks,
 Dan

 On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Ok so to be clear for any contributor (which is the goal of this
> thread): maven is still the main build system and no need to maintain
> gradle in PR then until beam switches.
>
> Im more than fine with that.
>
> Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :
>
>> I think the investment in gradle is worthwhile, and incrementally we
>> will continue to make progress. From what I've seem, gradle is a good fit
>> for this project and a path to a faster, more reliable build system.
>>
>> pull/4812  creates the
>> release artifacts, although it is not hooked up yet with authentication.
>>
>> I expect to help make incremental progress over the next month
>> converting some of the integration tests, but welcome incremental
>> improvements from others.
>>
>>
>>
>> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau <

Re: Gradle status

[Reuven Lax]

I'll send an email tomorrow with a few proposed dates and set up a burndown
list of tasks.

Reuven

On Thu, Mar 22, 2018 at 11:24 PM Romain Manni-Bucau 
wrote:

> Can work Reuven, where is the "todo" list? Thought we were done (= the
> replacement was not blocking the dev) multiple times due to other threads
> but reading this week mails it sounds like we are not yet here.
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-03-22 22:51 GMT+01:00 Reuven Lax :
>
>> Let's back up for a second.
>>
>> Earlier in the thread we agreed to organize a community "fixit" day to
>> try and migrate remaining Maven items to Gradle. I had thought that Romain
>> had volunteered to run this, but reading back in the thread it appears that
>> I misunderstood this. I would suggest that we organize this first, and make
>> the concerted push to migrate the remaining items. After this is done, we
>> can evaluate the state we're left in and hold a process vote if necessary.
>>
>> I can volunteer to help coordinate this fixit.
>>
>> Reuven
>>
>> On Thu, Mar 22, 2018 at 1:35 PM Dan Halperin  wrote:
>>
>>> On Thu, Mar 22, 2018 at 11:19 AM, Chamikara Jayalath <
>>> chamik...@google.com> wrote:
>>>
 I don't think incremental progress is a bad thing as long as we are
 making progress towards the goal. Do we need better metrics (a weekly email
 ?) about the progress towards moving everything to Gradle ? I agree with
 others who pointed out that there are many unresolved JIRAs and simply
 deleting Maven artifacts could break many things (for example, performance
 tests).

>>>
>>> The problem does not seem to be incremental progress on its face, or a
>>> lack of metrics.
>>>
>>> The problem is that there are two build systems with separate features
>>> and issues, doubling (or worse) Jenkins cycles, mental effort, maintenance
>>> burden, etc. It hurts the community and casual contributors.
>>>
>>> As Luke suggested,
>>> > A process vote can be happen if the in-between state is too painful to
>>> maintain.
>>>
>>> Given that the in-between state has lasted so long, and there is it may
>>> be time.
>>>
>>> Dan
>>>
>>>
>>>

 Thanks,
 Cham


 On Thu, Mar 22, 2018 at 10:56 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

>
>
> Le 22 mars 2018 18:49, "Dan Halperin"  a écrit :
>
> It seems that a few groups are talking past each other.
>
> * A sizable contingent is interested in a move to Gradle -- it shows
> promise, but the work is incomplete.
> * Another contingent noticing the large burden of maintaining multiple
> build systems. FWICT, both test suites have been broken quite a lot
> recently, mainly the gradle ones, which is a cost to the community. This 
> is
> creating a barrier to entry for new contributors – especially those who
> don't work in the same room or do their primary development in a different
> repository.
>
> I don't see this situation being resolved to anyone's satisfaction
> until there's only one build system left. The onus is clearly on the 
> Gradle
> promoters to finish the work.
>
> Luke made a suggestion 2.5 months ago that we should have a process
> vote if this situation is untenable. It seems like we're there.
>
>
> Yes but beam voted to move to gradle so we should but we shouldnt
> maintain 2 build systems for more than 2 months (weway overpassed that) 
> and
> therefore the vote should be cancelled or validated by an action.
>
> I understand you want gradle but you dont want to pay the cost to move
> to gradle, it doesnt work for the project do please another option
> (rollbacking gradle or removing maven, all other options are negative for
> the project and a pain for committers and contributors whatever you 
> think).
>
>
>
> Thanks,
> Dan
>
> On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Ok so to be clear for any contributor (which is the goal of this
>> thread): maven is still the main build system and no need to maintain
>> gradle in PR then until beam switches.
>>
>> Im more than fine with that.
>>
>> Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :
>>
>>> I think the investment in gradle is worthwhile, and incrementally we
>>> will continue to make progress. From what I've seem, gradle is a good 
>>> fit
>>> for this project and a path to a faster, more reliable build system.
>>>
>>> pull/4812  creates the
>>> releas

Re: Gradle status

[Scott Wegner]

Thanks for organizing, Reuven. I too would like to see us move back to a
single build system to reduce complexity. Count me in for the fixit.

On Thu, Mar 22, 2018 at 11:27 PM Reuven Lax  wrote:

> I'll send an email tomorrow with a few proposed dates and set up a
> burndown list of tasks.
>
> Reuven
>
> On Thu, Mar 22, 2018 at 11:24 PM Romain Manni-Bucau 
> wrote:
>
>> Can work Reuven, where is the "todo" list? Thought we were done (= the
>> replacement was not blocking the dev) multiple times due to other threads
>> but reading this week mails it sounds like we are not yet here.
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>> 2018-03-22 22:51 GMT+01:00 Reuven Lax :
>>
>>> Let's back up for a second.
>>>
>>> Earlier in the thread we agreed to organize a community "fixit" day to
>>> try and migrate remaining Maven items to Gradle. I had thought that Romain
>>> had volunteered to run this, but reading back in the thread it appears that
>>> I misunderstood this. I would suggest that we organize this first, and make
>>> the concerted push to migrate the remaining items. After this is done, we
>>> can evaluate the state we're left in and hold a process vote if necessary.
>>>
>>> I can volunteer to help coordinate this fixit.
>>>
>>> Reuven
>>>
>>> On Thu, Mar 22, 2018 at 1:35 PM Dan Halperin 
>>> wrote:
>>>
 On Thu, Mar 22, 2018 at 11:19 AM, Chamikara Jayalath <
 chamik...@google.com> wrote:

> I don't think incremental progress is a bad thing as long as we are
> making progress towards the goal. Do we need better metrics (a weekly 
> email
> ?) about the progress towards moving everything to Gradle ? I agree with
> others who pointed out that there are many unresolved JIRAs and simply
> deleting Maven artifacts could break many things (for example, performance
> tests).
>

 The problem does not seem to be incremental progress on its face, or a
 lack of metrics.

 The problem is that there are two build systems with separate features
 and issues, doubling (or worse) Jenkins cycles, mental effort, maintenance
 burden, etc. It hurts the community and casual contributors.

 As Luke suggested,
 > A process vote can be happen if the in-between state is too painful
 to maintain.

 Given that the in-between state has lasted so long, and there is it may
 be time.

 Dan



>
> Thanks,
> Cham
>
>
> On Thu, Mar 22, 2018 at 10:56 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>>
>>
>> Le 22 mars 2018 18:49, "Dan Halperin"  a écrit :
>>
>> It seems that a few groups are talking past each other.
>>
>> * A sizable contingent is interested in a move to Gradle -- it shows
>> promise, but the work is incomplete.
>> * Another contingent noticing the large burden of maintaining
>> multiple build systems. FWICT, both test suites have been broken quite a
>> lot recently, mainly the gradle ones, which is a cost to the community.
>> This is creating a barrier to entry for new contributors – especially 
>> those
>> who don't work in the same room or do their primary development in a
>> different repository.
>>
>> I don't see this situation being resolved to anyone's satisfaction
>> until there's only one build system left. The onus is clearly on the 
>> Gradle
>> promoters to finish the work.
>>
>> Luke made a suggestion 2.5 months ago that we should have a process
>> vote if this situation is untenable. It seems like we're there.
>>
>>
>> Yes but beam voted to move to gradle so we should but we shouldnt
>> maintain 2 build systems for more than 2 months (weway overpassed that) 
>> and
>> therefore the vote should be cancelled or validated by an action.
>>
>> I understand you want gradle but you dont want to pay the cost to
>> move to gradle, it doesnt work for the project do please another option
>> (rollbacking gradle or removing maven, all other options are negative for
>> the project and a pain for committers and contributors whatever you 
>> think).
>>
>>
>>
>> Thanks,
>> Dan
>>
>> On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Ok so to be clear for any contributor (which is the goal of this
>>> thread): maven is still the main build system and no need to maintain
>>> gradle in PR then until beam switches.
>>>
>>> Im more than fine with that.
>>>
>>> Le 22 mars 2018 18:22, "Alan Myrvold"  a
>>> écrit :
>

Gradle Status: Migrated!

[Scott Wegner]

Many many of you have been hacking diligently on the Gradle build, and I'm
happy to announce that we now have a fully-functioning Gradle build!
There's been a ton of progress since our last update [1]:

* Improved nightly snapshot release [2]
* Improve runner quickstarts [5] [11]
* Python post-commit ported to Gradle [3]
* Update performance testing framework for Gradle [4] [12]
* Generate javadocs from Gradle [6]
* Update to latest Gradle version [7] [21]
* Updated documentation [8] [22]
* Tune CI build resource usage for Jenkins [9] [19]
* Improve shading of test jars [10] [13] [14]
* Add 'errorprone' and 'spotless' static analysis [15] [24]
* Improve IntelliJ project generation [16] [17]
* Reduce number of ValidatesRunner tests [18]
* Update release documentation for Gradle [20]
* Update docker build scripts for Gradle [23]

The build process and Jenkins environment have stabilized and we've
resolved migration blockers. The final step is to use Gradle to produce an
official release. The release documentation has been updated for Gradle and
I recommend we use these docs for the 2.5.0 release. Assuming the release
goes well, we can declare the migration fully validated and stop supporting
dual build systems.

During the migration we identified a number of opportunities to improve the
build even further. Feel free to grab one of the items off of the JIRA:
BEAM-4045 [24]

Thanks again to all those that contributed. This has truly been a community
effort!

[1]
https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
[2] https://github.com/apache/beam/pull/5142
[3] https://github.com/apache/beam/pull/5146
[4] https://github.com/apache/beam/pull/5003
[5] https://github.com/apache/beam/pull/5151
[6] https://github.com/apache/beam/pull/5121
[7] https://github.com/apache/beam/pull/5104
[8] https://github.com/apache/beam/pull/5183
[9] https://github.com/apache/beam/pull/5171
[10] https://github.com/apache/beam/pull/5117
[11] https://github.com/apache/beam/pull/5200
[12] https://github.com/apache/beam/pull/5051
[13] https://github.com/apache/beam/pull/4740
[14] https://github.com/apache/beam/pull/4702
[15] https://github.com/apache/beam/pull/4701
[16] https://github.com/apache/beam/pull/4626
[17] https://github.com/apache/beam/pull/4625
[18] https://github.com/apache/beam/pull/5193
[19] https://github.com/apache/beam/pull/5222
[20] https://github.com/apache/beam/pull/5187
[21] https://github.com/apache/beam/pull/5217
[22] https://github.com/apache/beam/pull/5115
[23] https://github.com/apache/beam/pull/5252
[24] https://github.com/apache/beam/pull/5161
[25] https://issues.apache.org/jira/browse/BEAM-4045
-- 


Got feedback? http://go/swegner-feedback

Gradle Status [April 6]

[Scott Wegner]

I wanted to start a thread to summarize the current state of Gradle
migration. We've made lots of good progress so far this week. Here's the
status from what I can tell-- please add or correct anything I missed:

* Release artifacts can be built and published for Snapshot and officlal
releases [1]
* Gradle-generated releases have been validated with the the Apache Beam
archetype generation quickstart; still needs additional validation.
* Generated release pom files have correct project metadata [2]
* The python pre-commits are now working in Gradle [3]
* Ismaël has started a collaborative doc of Gradle tips [4] as we all learn
the new system-- please add your own. This will eventually feed into
official documentation on the website.
* Łukasz Gajowy is working on migrating performance testing framework [5]
* Daniel is working on updating documentation to refer to Gradle instead of
maven

If I missed anything, please add it to this thread.

The general roadmap we're working towards is:
(a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
(b) Postcommits migrated to Gradle
(c) Migrate documentation from maven to Gradle
(d) Migrate perfkit suites to use Gradle

For those of you that are hacking: thanks for your help so far! Progress is
being roughly tracked on the Kanban [6]; please make sure the issues
assigned to you are up-to-date. Many of the changes are staged on
lukecwik's local branch [7]; we'll work on merging them back soon.


[1] https://github.com/lukecwik/incubator-beam/pull/7
[2] https://github.com/lukecwik/incubator-beam/pull/3
[3] https://github.com/apache/beam/pull/5032
[4]
https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
[5] https://github.com/apache/beam/pull/5003
[6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
[7] https://github.com/lukecwik/incubator-beam/tree/gradle
-- 


Got feedback? http://go/swegner-feedback

Gradle Status [April 11]

[Scott Wegner]

Thanks everyone for the continued effort towards the Gradle migration. As a
high-level summary of our progress since Friday: we have a viable build,
with a number of minor issues that we're still working out. Please take a
look at the new documentation in our contribution guide and log any bugs
that you find.

Here's a more detailed view of improvements from just the past few days..

Release artifacts:
*  Pom.xml generation logic now in master [1]
* Nightly snapshots are now produced using Gradle [2]
* Excluded modules propagated to dependencies when generating * pom.xml
* Artifact JARs are properly shaded [3]
* Working on fixing dependency scopes in generated pom [4]
PreCommits / Postcommits:
* All PreCommits and PostCommits migrated [5]; working on deflaking [6] [7]
[8] [9]
* Jenkins results now include JUnit test results [10] and build scan for
easier debugging [11]
* Spark ValidatesRunner PostCommit passes [12] [13]
* Flink ValidatesRunner PostCommit more reliable [14]
Documentation / IDE Setup:
* Contribution Guide [15] is now updated with Gradle commands [16] [17]
Performance Benchmarks:
* Working on getting Nexmark benchmarks migrated [18]

If I missed anything, please add it to this thread.

We are continuing to use this general roadmap:
(a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
(b) Postcommits migrated to Gradle
(c) Migrate documentation from maven to Gradle
(d) Migrate perfkit suites to use Gradle

Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has created
a separate issue to track post-migration cleanup items: BEAM-4045 [20].
Feel free to grab any unassigned items off of either list.


[1] https://github.com/apache/beam/pull/5054
[2] https://github.com/apache/beam/pull/5057
[3] https://github.com/apache/beam/pull/5087
[4] https://github.com/apache/beam/pull/5098
[5] https://github.com/apache/beam/pull/5047
[6] https://github.com/apache/beam/pull/5088
[7] https://github.com/apache/beam/pull/5086
[8] https://github.com/apache/beam/pull/5066
[9] https://github.com/apache/beam/pull/5059
[10] https://github.com/apache/beam/pull/5045
[11] https://github.com/apache/beam/pull/5091
[12] https://github.com/apache/beam/pull/5093
[13] https://github.com/apache/beam/pull/5069
[14] https://github.com/apache/beam/pull/5068
[15] https://beam.apache.org/contribute/contribution-guide/
[16] https://github.com/apache/beam-site/pull/412
[17] https://github.com/apache/beam-site/pull/414
[18] https://github.com/apache/beam/pull/5051
[19] https://issues.apache.org/jira/browse/BEAM-3249
[20] https://issues.apache.org/jira/browse/BEAM-4045

On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:

> I wanted to start a thread to summarize the current state of Gradle
> migration. We've made lots of good progress so far this week. Here's the
> status from what I can tell-- please add or correct anything I missed:
>
> * Release artifacts can be built and published for Snapshot and officlal
> releases [1]
> * Gradle-generated releases have been validated with the the Apache Beam
> archetype generation quickstart; still needs additional validation.
> * Generated release pom files have correct project metadata [2]
> * The python pre-commits are now working in Gradle [3]
> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
> learn the new system-- please add your own. This will eventually feed into
> official documentation on the website.
> * Łukasz Gajowy is working on migrating performance testing framework [5]
> * Daniel is working on updating documentation to refer to Gradle instead
> of maven
>
> If I missed anything, please add it to this thread.
>
> The general roadmap we're working towards is:
> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to Gradle
> (d) Migrate perfkit suites to use Gradle
>
> For those of you that are hacking: thanks for your help so far! Progress
> is being roughly tracked on the Kanban [6]; please make sure the issues
> assigned to you are up-to-date. Many of the changes are staged on
> lukecwik's local branch [7]; we'll work on merging them back soon.
>
>
> [1] https://github.com/lukecwik/incubator-beam/pull/7
> [2] https://github.com/lukecwik/incubator-beam/pull/3
> [3] https://github.com/apache/beam/pull/5032
> [4]
> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
> [5] https://github.com/apache/beam/pull/5003
> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
> --
>
>
> Got feedback? http://go/swegner-feedback
>
-- 


Got feedback? http://go/swegner-feedback

Gradle Status [April 13]

[Pablo Estrada]

Hello everyone,
here's a high-level summary of Gradle migration progress since the last
update. The progress this Friday was a bit slower, though we expect to
continue at a good pace next week. Thanks again to everyone that is putting
time into the migration.

Ongoing work / PRs:
* Migration of JavaDocs and other documentation continues [1][2]
* Cleanup for use of shaded tests is ongoing [3], though with some fixes
pending
* Progress ongoing to migrate IO perf tests[4]
* Changes to the contributor's guide are in review[5]

Some issues that remain:
* Dataflow ValidatesRunner tests are timing out, and being investigated[6]
* Issues have been observed trying to build in Mac OS. Need investigation
[7][7-1]

As noted earlier by Scott, the main issue to track migration is
BEAM-3249[8], and an issue to track cleanup items has been created [9].
Both good resources to track progress.

Please add to this thread any other progress updates that I might have
missed.

Best
-P.

[1] https://github.com/apache/beam/pull/5121
[2] https://github.com/apache/beam-site/pull/419
[3] https://github.com/apache/beam/pull/5117
[4] https://github.com/apache/beam/pull/5003
[5] https://github.com/apache/beam-site/pull/414
[6] https://issues.apache.org/jira/browse/BEAM-4059
[7] https://the-asf.slack.com/archives/C9H0YNP3P/p152363491398
[7-1] https://issues.apache.org/jira/browse/BEAM-4075
[8] https://issues.apache.org/jira/browse/BEAM-3249
[9] https://issues.apache.org/jira/browse/BEAM-4045

-- 
Got feedback? go/pabloem-feedback

Gradle Status [April 17th]

[Scott Wegner]

Gradle migration continues. Our last update was on April 13th, and since
then there has been significant progress:

Release artifacts:
* Upgrade to latest Gradle PR5104
* Remove evaluationDependsOn and use shaded test jars PR5117
* Nightly java snapshot release fixed by PR5136 and PR 5142 and now passing

PreCommits / Postcommits:
* Add better Gradle documentation for quickstarts PR5115
* Python Precommit failures due to environment inconsistencies being
temporarily repaired (PR 5149)
* Porting of Python postcommit to a Gradle task instead of a shell script
(PR 5155, PR 5146)
* Increase parallelism to deflake Dataflow postcommits PR5143
* Fix Spark quickstarts PR5151
* Restrict Jenkins host machines due to environmental issues PR5149

Documentation / IDE Setup:
* Update Eclipse documentation for Gradle PR419

Performance Benchmarks:
* Add Gradle run task to Nexmark PR5051
* Update performance testing framework to use Gradle PR5003
* Rebuild before running Performance tests PR5153

We continue to track migration blockers as sub-tasks in BEAM-3249, which is
now available as a Kanban board: https://s.apache.org/beam-gradle-migration.
To summarize the work left for migration:

* Validate metadata in release pom file and jars.
* Update release guide documentation
* Fix Jenkins flakiness caused by Gradle
* Convert benchmark jobs to use Gradle
-- 


Got feedback? http://go/swegner-feedback

Re: Gradle Status: Migrated!

[Romain Manni-Bucau]

Hi Scott

While https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
is open, gradle is a concurrent of maven but maven must stay the default
build tool cause gradle breaks users.


Le 1 mai 2018 01:59, "Scott Wegner"  a écrit :

> Many many of you have been hacking diligently on the Gradle build, and I'm
> happy to announce that we now have a fully-functioning Gradle build!
> There's been a ton of progress since our last update [1]:
>
> * Improved nightly snapshot release [2]
> * Improve runner quickstarts [5] [11]
> * Python post-commit ported to Gradle [3]
> * Update performance testing framework for Gradle [4] [12]
> * Generate javadocs from Gradle [6]
> * Update to latest Gradle version [7] [21]
> * Updated documentation [8] [22]
> * Tune CI build resource usage for Jenkins [9] [19]
> * Improve shading of test jars [10] [13] [14]
> * Add 'errorprone' and 'spotless' static analysis [15] [24]
> * Improve IntelliJ project generation [16] [17]
> * Reduce number of ValidatesRunner tests [18]
> * Update release documentation for Gradle [20]
> * Update docker build scripts for Gradle [23]
>
> The build process and Jenkins environment have stabilized and we've
> resolved migration blockers. The final step is to use Gradle to produce an
> official release. The release documentation has been updated for Gradle and
> I recommend we use these docs for the 2.5.0 release. Assuming the release
> goes well, we can declare the migration fully validated and stop supporting
> dual build systems.
>
> During the migration we identified a number of opportunities to improve
> the build even further. Feel free to grab one of the items off of the JIRA:
> BEAM-4045 [24]
>
> Thanks again to all those that contributed. This has truly been a
> community effort!
>
> [1] https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec31061
> 3e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
> [2] https://github.com/apache/beam/pull/5142
> [3] https://github.com/apache/beam/pull/5146
> [4] https://github.com/apache/beam/pull/5003
> [5] https://github.com/apache/beam/pull/5151
> [6] https://github.com/apache/beam/pull/5121
> [7] https://github.com/apache/beam/pull/5104
> [8] https://github.com/apache/beam/pull/5183
> [9] https://github.com/apache/beam/pull/5171
> [10] https://github.com/apache/beam/pull/5117
> [11] https://github.com/apache/beam/pull/5200
> [12] https://github.com/apache/beam/pull/5051
> [13] https://github.com/apache/beam/pull/4740
> [14] https://github.com/apache/beam/pull/4702
> [15] https://github.com/apache/beam/pull/4701
> [16] https://github.com/apache/beam/pull/4626
> [17] https://github.com/apache/beam/pull/4625
> [18] https://github.com/apache/beam/pull/5193
> [19] https://github.com/apache/beam/pull/5222
> [20] https://github.com/apache/beam/pull/5187
> [21] https://github.com/apache/beam/pull/5217
> [22] https://github.com/apache/beam/pull/5115
> [23] https://github.com/apache/beam/pull/5252
> [24] https://github.com/apache/beam/pull/5161
> [25] https://issues.apache.org/jira/browse/BEAM-4045
> --
>
>
> Got feedback? http://go/swegner-feedback
>

Re: Gradle Status: Migrated!

[Łukasz Gajowy]

Hi Scott,

thanks for the update! Just a clarification about IO performance tests:
those were fully migrated in Beam and all task necessary for running them
are there but Jenkins jobs still run mvn commands. This is due the fact
that PerfkitBenchmarker code (which is invoked by Jenkins and constructs
the commands by itself) was not updated yet. This should be finished before
fully dropping mvn.

More on that topic here, in comments:
https://issues.apache.org/jira/browse/BEAM-3942
PR changing the commands to gradle is waiting for PerfKit devs review here:
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648

Best regards,

2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau :

> Hi Scott
>
> While https://issues.apache.org/jira/plugins/servlet/
> mobile#issue/BEAM-4057 is open, gradle is a concurrent of maven but maven
> must stay the default build tool cause gradle breaks users.
>
>
> Le 1 mai 2018 01:59, "Scott Wegner"  a écrit :
>
>> Many many of you have been hacking diligently on the Gradle build, and
>> I'm happy to announce that we now have a fully-functioning Gradle build!
>> There's been a ton of progress since our last update [1]:
>>
>> * Improved nightly snapshot release [2]
>> * Improve runner quickstarts [5] [11]
>> * Python post-commit ported to Gradle [3]
>> * Update performance testing framework for Gradle [4] [12]
>> * Generate javadocs from Gradle [6]
>> * Update to latest Gradle version [7] [21]
>> * Updated documentation [8] [22]
>> * Tune CI build resource usage for Jenkins [9] [19]
>> * Improve shading of test jars [10] [13] [14]
>> * Add 'errorprone' and 'spotless' static analysis [15] [24]
>> * Improve IntelliJ project generation [16] [17]
>> * Reduce number of ValidatesRunner tests [18]
>> * Update release documentation for Gradle [20]
>> * Update docker build scripts for Gradle [23]
>>
>> The build process and Jenkins environment have stabilized and we've
>> resolved migration blockers. The final step is to use Gradle to produce an
>> official release. The release documentation has been updated for Gradle and
>> I recommend we use these docs for the 2.5.0 release. Assuming the release
>> goes well, we can declare the migration fully validated and stop supporting
>> dual build systems.
>>
>> During the migration we identified a number of opportunities to improve
>> the build even further. Feel free to grab one of the items off of the JIRA:
>> BEAM-4045 [24]
>>
>> Thanks again to all those that contributed. This has truly been a
>> community effort!
>>
>> [1] https://lists.apache.org/thread.html/5f6bae323acc1b05096
>> 2e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
>> [2] https://github.com/apache/beam/pull/5142
>> [3] https://github.com/apache/beam/pull/5146
>> [4] https://github.com/apache/beam/pull/5003
>> [5] https://github.com/apache/beam/pull/5151
>> [6] https://github.com/apache/beam/pull/5121
>> [7] https://github.com/apache/beam/pull/5104
>> [8] https://github.com/apache/beam/pull/5183
>> [9] https://github.com/apache/beam/pull/5171
>> [10] https://github.com/apache/beam/pull/5117
>> [11] https://github.com/apache/beam/pull/5200
>> [12] https://github.com/apache/beam/pull/5051
>> [13] https://github.com/apache/beam/pull/4740
>> [14] https://github.com/apache/beam/pull/4702
>> [15] https://github.com/apache/beam/pull/4701
>> [16] https://github.com/apache/beam/pull/4626
>> [17] https://github.com/apache/beam/pull/4625
>> [18] https://github.com/apache/beam/pull/5193
>> [19] https://github.com/apache/beam/pull/5222
>> [20] https://github.com/apache/beam/pull/5187
>> [21] https://github.com/apache/beam/pull/5217
>> [22] https://github.com/apache/beam/pull/5115
>> [23] https://github.com/apache/beam/pull/5252
>> [24] https://github.com/apache/beam/pull/5161
>> [25] https://issues.apache.org/jira/browse/BEAM-4045
>> --
>>
>>
>> Got feedback? http://go/swegner-feedback
>>
>

Re: Gradle Status: Migrated!

[Jean-Baptiste Onofré]

By the way, I'm curious: did someone evaluate the build time gap between Maven
and Gradle ? One of the main reason to migrate to Gradle was the inc build and
build time. The builds I have launched are quite the same in duration. I will do
deeper tests to evaluate the gap.

Regards
JB

On 05/01/2018 12:48 PM, Łukasz Gajowy wrote:
> Hi Scott, 
> 
> thanks for the update! Just a clarification about IO performance tests: those
> were fully migrated in Beam and all task necessary for running them are there
> but Jenkins jobs still run mvn commands. This is due the fact that
> PerfkitBenchmarker code (which is invoked by Jenkins and constructs the 
> commands
> by itself) was not updated yet. This should be finished before fully dropping 
> mvn. 
> 
> More on that topic here, in
> comments: https://issues.apache.org/jira/browse/BEAM-3942
> PR changing the commands to gradle is waiting for PerfKit devs review
> here: https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648
> 
> Best regards,
> 
> 2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau  >:
> 
> Hi Scott
> 
> While 
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
>  is
> open, gradle is a concurrent of maven but maven must stay the default 
> build
> tool cause gradle breaks users.
> 
> 
> Le 1 mai 2018 01:59, "Scott Wegner"  > a écrit :
> 
> Many many of you have been hacking diligently on the Gradle build, and
> I'm happy to announce that we now have a fully-functioning Gradle 
> build!
> There's been a ton of progress since our last update [1]:
> 
> * Improved nightly snapshot release [2]
> * Improve runner quickstarts [5] [11]
> * Python post-commit ported to Gradle [3]
> * Update performance testing framework for Gradle [4] [12]
> * Generate javadocs from Gradle [6]
> * Update to latest Gradle version [7] [21]
> * Updated documentation [8] [22]
> * Tune CI build resource usage for Jenkins [9] [19]
> * Improve shading of test jars [10] [13] [14]
> * Add 'errorprone' and 'spotless' static analysis [15] [24]
> * Improve IntelliJ project generation [16] [17]
> * Reduce number of ValidatesRunner tests [18]
> * Update release documentation for Gradle [20]
> * Update docker build scripts for Gradle [23]
> 
> The build process and Jenkins environment have stabilized and we've
> resolved migration blockers. The final step is to use Gradle to 
> produce
> an official release. The release documentation has been updated for
> Gradle and I recommend we use these docs for the 2.5.0 release. 
> Assuming
> the release goes well, we can declare the migration fully validated 
> and
> stop supporting dual build systems.
> 
> During the migration we identified a number of opportunities to 
> improve
> the build even further. Feel free to grab one of the items off of the
> JIRA: BEAM-4045 [24]
> 
> Thanks again to all those that contributed. This has truly been a
> community effort!
> 
> [1] 
> https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
> 
> 
> [2] https://github.com/apache/beam/pull/5142
>  
> [3] https://github.com/apache/beam/pull/5146
>  
> [4] https://github.com/apache/beam/pull/5003
>  
> [5] https://github.com/apache/beam/pull/5151
>  
> [6] https://github.com/apache/beam/pull/5121
>  
> [7] https://github.com/apache/beam/pull/5104
>  
> [8] https://github.com/apache/beam/pull/5183
>  
> [9] https://github.com/apache/beam/pull/5171
>  
> [10] https://github.com/apache/beam/pull/5117
>  
> [11] https://github.com/apache/beam/pull/5200
>  
> [12] https://github.com/apache/beam/pull/5051
>  
> [13] https://github.com/apache/beam/pull/4740
>  
> [14] https://github.com/apache/beam/pull/4702
>  
> [15] https:

Re: Gradle Status: Migrated!

[Reuven Lax]

Luke did gather data which showed that on our Jenkins executors the Gradle
build was much faster than the Maven build. Also right now we have
incremental builds turned off, but once we're confident enough to enable
them (at least for local development) that will often drop build times a
lot.

On Tue, May 1, 2018 at 4:01 AM Jean-Baptiste Onofré  wrote:

> By the way, I'm curious: did someone evaluate the build time gap between
> Maven
> and Gradle ? One of the main reason to migrate to Gradle was the inc build
> and
> build time. The builds I have launched are quite the same in duration. I
> will do
> deeper tests to evaluate the gap.
>
> Regards
> JB
>
> On 05/01/2018 12:48 PM, Łukasz Gajowy wrote:
> > Hi Scott,
> >
> > thanks for the update! Just a clarification about IO performance tests:
> those
> > were fully migrated in Beam and all task necessary for running them are
> there
> > but Jenkins jobs still run mvn commands. This is due the fact that
> > PerfkitBenchmarker code (which is invoked by Jenkins and constructs the
> commands
> > by itself) was not updated yet. This should be finished before fully
> dropping mvn.
> >
> > More on that topic here, in
> > comments: https://issues.apache.org/jira/browse/BEAM-3942
> > PR changing the commands to gradle is waiting for PerfKit devs review
> > here:
> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648
> >
> > Best regards,
> >
> > 2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau  > >:
> >
> > Hi Scott
> >
> > While
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
> > <
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057> is
> > open, gradle is a concurrent of maven but maven must stay the
> default build
> > tool cause gradle breaks users.
> >
> >
> > Le 1 mai 2018 01:59, "Scott Wegner"  > > a écrit :
> >
> > Many many of you have been hacking diligently on the Gradle
> build, and
> > I'm happy to announce that we now have a fully-functioning
> Gradle build!
> > There's been a ton of progress since our last update [1]:
> >
> > * Improved nightly snapshot release [2]
> > * Improve runner quickstarts [5] [11]
> > * Python post-commit ported to Gradle [3]
> > * Update performance testing framework for Gradle [4] [12]
> > * Generate javadocs from Gradle [6]
> > * Update to latest Gradle version [7] [21]
> > * Updated documentation [8] [22]
> > * Tune CI build resource usage for Jenkins [9] [19]
> > * Improve shading of test jars [10] [13] [14]
> > * Add 'errorprone' and 'spotless' static analysis [15] [24]
> > * Improve IntelliJ project generation [16] [17]
> > * Reduce number of ValidatesRunner tests [18]
> > * Update release documentation for Gradle [20]
> > * Update docker build scripts for Gradle [23]
> >
> > The build process and Jenkins environment have stabilized and
> we've
> > resolved migration blockers. The final step is to use Gradle to
> produce
> > an official release. The release documentation has been updated
> for
> > Gradle and I recommend we use these docs for the 2.5.0 release.
> Assuming
> > the release goes well, we can declare the migration fully
> validated and
> > stop supporting dual build systems.
> >
> > During the migration we identified a number of opportunities to
> improve
> > the build even further. Feel free to grab one of the items off
> of the
> > JIRA: BEAM-4045 [24]
> >
> > Thanks again to all those that contributed. This has truly been a
> > community effort!
> >
> > [1]
> https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
> > <
> https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
> >
> > [2] https://github.com/apache/beam/pull/5142
> > 
> > [3] https://github.com/apache/beam/pull/5146
> > 
> > [4] https://github.com/apache/beam/pull/5003
> > 
> > [5] https://github.com/apache/beam/pull/5151
> > 
> > [6] https://github.com/apache/beam/pull/5121
> > 
> > [7] https://github.com/apache/beam/pull/5104
> > 
> > [8] https://github.com/apache/beam/pull/5183
> > 
> > [9] https://github.com/apache/beam/pull/5171
> > 
> > [10] https://github.com/apache/beam/

Re: Gradle Status: Migrated!

[Jean-Baptiste Onofré]

Thanks, for me, Maven 3.5.2 takes quite the same time than Gradle (using 
the wrapper). It's maybe related to my environment.


Anyway, I'm doing a complete build review both in term of building time, 
and equivalence (artifacts publishing, test, plugin execution).


I will provide an update soon.

Regards
JB

On 01/05/2018 16:57, Reuven Lax wrote:
Luke did gather data which showed that on our Jenkins executors the 
Gradle build was much faster than the Maven build. Also right now we 
have incremental builds turned off, but once we're confident enough to 
enable them (at least for local development) that will often drop build 
times a lot.


On Tue, May 1, 2018 at 4:01 AM Jean-Baptiste Onofré > wrote:


By the way, I'm curious: did someone evaluate the build time gap
between Maven
and Gradle ? One of the main reason to migrate to Gradle was the inc
build and
build time. The builds I have launched are quite the same in
duration. I will do
deeper tests to evaluate the gap.

Regards
JB

On 05/01/2018 12:48 PM, Łukasz Gajowy wrote:
 > Hi Scott,
 >
 > thanks for the update! Just a clarification about IO performance
tests: those
 > were fully migrated in Beam and all task necessary for running
them are there
 > but Jenkins jobs still run mvn commands. This is due the fact that
 > PerfkitBenchmarker code (which is invoked by Jenkins and
constructs the commands
 > by itself) was not updated yet. This should be finished before
fully dropping mvn.
 >
 > More on that topic here, in
 > comments: https://issues.apache.org/jira/browse/BEAM-3942
 > PR changing the commands to gradle is waiting for PerfKit devs review
 > here:
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648
 >
 > Best regards,
 >
 > 2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau
mailto:rmannibu...@gmail.com>
 > >>:
 >
 >     Hi Scott
 >
 >     While
https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
 >   
   is

 >     open, gradle is a concurrent of maven but maven must stay the
default build
 >     tool cause gradle breaks users.
 >
 >
 >     Le 1 mai 2018 01:59, "Scott Wegner" mailto:sweg...@google.com>
 >     >> a
écrit :
 >
 >         Many many of you have been hacking diligently on the
Gradle build, and
 >         I'm happy to announce that we now have a
fully-functioning Gradle build!
 >         There's been a ton of progress since our last update [1]:
 >
 >         * Improved nightly snapshot release [2]
 >         * Improve runner quickstarts [5] [11]
 >         * Python post-commit ported to Gradle [3]
 >         * Update performance testing framework for Gradle [4] [12]
 >         * Generate javadocs from Gradle [6]
 >         * Update to latest Gradle version [7] [21]
 >         * Updated documentation [8] [22]
 >         * Tune CI build resource usage for Jenkins [9] [19]
 >         * Improve shading of test jars [10] [13] [14]
 >         * Add 'errorprone' and 'spotless' static analysis [15] [24]
 >         * Improve IntelliJ project generation [16] [17]
 >         * Reduce number of ValidatesRunner tests [18]
 >         * Update release documentation for Gradle [20]
 >         * Update docker build scripts for Gradle [23]
 >
 >         The build process and Jenkins environment have stabilized
and we've
 >         resolved migration blockers. The final step is to use
Gradle to produce
 >         an official release. The release documentation has been
updated for
 >         Gradle and I recommend we use these docs for the 2.5.0
release. Assuming
 >         the release goes well, we can declare the migration fully
validated and
 >         stop supporting dual build systems.
 >
 >         During the migration we identified a number of
opportunities to improve
 >         the build even further. Feel free to grab one of the
items off of the
 >         JIRA: BEAM-4045 [24]
 >
 >         Thanks again to all those that contributed. This has
truly been a
 >         community effort!
 >
 >         [1]

https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
 >   
  

 >         [2] https://github.com/apache/beam/pull/5142
 >         
 >         [3] https://github.com/apache/beam/pull/5146

Re: Gradle Status: Migrated!

[Kenneth Knowles]

Raw execution time for tasks from clean is not the only thing to test. I
would say it is not even important. Try these from clean:

 - Gradle: ./gradlew :beam-sdks-java-io-mongodb:test && ./gradlew :
beam-sdks-java-io-mongodb:test
 - Maven: mvn -pl sdks/java/io/mongodb test -am && mvn -pl
sdks/java/io/mongodb test -am

Quick run on my laptop:

 - Gradle: 66s (65s then 1s)
 - Maven: 317s (173s then 144s)

Of course, the mvn command runs a bunch of useless executions AND it is
incorrect because it isn't using built jars. That's part of the point -
there is no way to do what you want with mvn. Let's try to make a command
that avoids useless work and builds the jars:

 - Maven:  (mvn -pl sdks/java/io/mongodb install -DskipTests -am && mvn -pl
sdks/java/io/mongodb test) && (each time)

That takes 102s the first time and 64s the second time. And that is about
the realistic workflow for someone trying to get something done. Even if we
touch a file Gradle finishes in 20s. So the best case for mvn is this
head-to-head:

 - Gradle: 65s + 20s + 20s + 20s + 20s + ...
 - Maven: 102s + 64s + 64s + 64s + 64s + ...

Kenn


On Tue, May 1, 2018 at 8:09 AM Jean-Baptiste Onofré  wrote:

> Thanks, for me, Maven 3.5.2 takes quite the same time than Gradle (using
> the wrapper). It's maybe related to my environment.
>
> Anyway, I'm doing a complete build review both in term of building time,
> and equivalence (artifacts publishing, test, plugin execution).
>
> I will provide an update soon.
>
> Regards
> JB
>
> On 01/05/2018 16:57, Reuven Lax wrote:
> > Luke did gather data which showed that on our Jenkins executors the
> > Gradle build was much faster than the Maven build. Also right now we
> > have incremental builds turned off, but once we're confident enough to
> > enable them (at least for local development) that will often drop build
> > times a lot.
> >
> > On Tue, May 1, 2018 at 4:01 AM Jean-Baptiste Onofré  > > wrote:
> >
> > By the way, I'm curious: did someone evaluate the build time gap
> > between Maven
> > and Gradle ? One of the main reason to migrate to Gradle was the inc
> > build and
> > build time. The builds I have launched are quite the same in
> > duration. I will do
> > deeper tests to evaluate the gap.
> >
> > Regards
> > JB
> >
> > On 05/01/2018 12:48 PM, Łukasz Gajowy wrote:
> >  > Hi Scott,
> >  >
> >  > thanks for the update! Just a clarification about IO performance
> > tests: those
> >  > were fully migrated in Beam and all task necessary for running
> > them are there
> >  > but Jenkins jobs still run mvn commands. This is due the fact that
> >  > PerfkitBenchmarker code (which is invoked by Jenkins and
> > constructs the commands
> >  > by itself) was not updated yet. This should be finished before
> > fully dropping mvn.
> >  >
> >  > More on that topic here, in
> >  > comments: https://issues.apache.org/jira/browse/BEAM-3942
> >  > PR changing the commands to gradle is waiting for PerfKit devs
> review
> >  > here:
> > https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648
> >  >
> >  > Best regards,
> >  >
> >  > 2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau
> > mailto:rmannibu...@gmail.com>
> >  > >>:
> >  >
> >  > Hi Scott
> >  >
> >  > While
> >
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
> >  >
> >   <
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057> is
> >  > open, gradle is a concurrent of maven but maven must stay the
> > default build
> >  > tool cause gradle breaks users.
> >  >
> >  >
> >  > Le 1 mai 2018 01:59, "Scott Wegner"  > 
> >  > >> a
> > écrit :
> >  >
> >  > Many many of you have been hacking diligently on the
> > Gradle build, and
> >  > I'm happy to announce that we now have a
> > fully-functioning Gradle build!
> >  > There's been a ton of progress since our last update [1]:
> >  >
> >  > * Improved nightly snapshot release [2]
> >  > * Improve runner quickstarts [5] [11]
> >  > * Python post-commit ported to Gradle [3]
> >  > * Update performance testing framework for Gradle [4] [12]
> >  > * Generate javadocs from Gradle [6]
> >  > * Update to latest Gradle version [7] [21]
> >  > * Updated documentation [8] [22]
> >  > * Tune CI build resource usage for Jenkins [9] [19]
> >  > * Improve shading of test jars [10] [13] [14]
> >  > * Add 'errorprone' and 'spotless' static analysis [15]
> [24]
> >  > * Improve IntelliJ project generation [16] [17]
> >   

Re: Gradle Status: Migrated!

[Jean-Baptiste Onofré]

Thanks for the update Kenn, that makes sense.

I'm checking the artifacts generated by Gradle right now.

Regards
JB

On 01/05/2018 17:42, Kenneth Knowles wrote:
Raw execution time for tasks from clean is not the only thing to test. I 
would say it is not even important. Try these from clean:


  - Gradle: ./gradlew :beam-sdks-java-io-mongodb:test && ./gradlew 
:beam-sdks-java-io-mongodb:test
  - Maven: mvn -pl sdks/java/io/mongodb test -am && mvn -pl 
sdks/java/io/mongodb test -am


Quick run on my laptop:

  - Gradle: 66s (65s then 1s)
  - Maven: 317s (173s then 144s)

Of course, the mvn command runs a bunch of useless executions AND it is 
incorrect because it isn't using built jars. That's part of the point - 
there is no way to do what you want with mvn. Let's try to make a 
command that avoids useless work and builds the jars:


  - Maven:  (mvn -pl sdks/java/io/mongodb install -DskipTests -am && mvn 
-pl sdks/java/io/mongodb test) && (each time)


That takes 102s the first time and 64s the second time. And that is 
about the realistic workflow for someone trying to get something done. 
Even if we touch a file Gradle finishes in 20s. So the best case for mvn 
is this head-to-head:


  - Gradle: 65s + 20s + 20s + 20s + 20s + ...
  - Maven: 102s + 64s + 64s + 64s + 64s + ...

Kenn


On Tue, May 1, 2018 at 8:09 AM Jean-Baptiste Onofré > wrote:


Thanks, for me, Maven 3.5.2 takes quite the same time than Gradle
(using
the wrapper). It's maybe related to my environment.

Anyway, I'm doing a complete build review both in term of building
time,
and equivalence (artifacts publishing, test, plugin execution).

I will provide an update soon.

Regards
JB

On 01/05/2018 16:57, Reuven Lax wrote:
 > Luke did gather data which showed that on our Jenkins executors the
 > Gradle build was much faster than the Maven build. Also right now we
 > have incremental builds turned off, but once we're confident
enough to
 > enable them (at least for local development) that will often drop
build
 > times a lot.
 >
 > On Tue, May 1, 2018 at 4:01 AM Jean-Baptiste Onofré
mailto:j...@nanthrax.net>
 > >> wrote:
 >
 >     By the way, I'm curious: did someone evaluate the build time gap
 >     between Maven
 >     and Gradle ? One of the main reason to migrate to Gradle was
the inc
 >     build and
 >     build time. The builds I have launched are quite the same in
 >     duration. I will do
 >     deeper tests to evaluate the gap.
 >
 >     Regards
 >     JB
 >
 >     On 05/01/2018 12:48 PM, Łukasz Gajowy wrote:
 >      > Hi Scott,
 >      >
 >      > thanks for the update! Just a clarification about IO
performance
 >     tests: those
 >      > were fully migrated in Beam and all task necessary for running
 >     them are there
 >      > but Jenkins jobs still run mvn commands. This is due the
fact that
 >      > PerfkitBenchmarker code (which is invoked by Jenkins and
 >     constructs the commands
 >      > by itself) was not updated yet. This should be finished before
 >     fully dropping mvn.
 >      >
 >      > More on that topic here, in
 >      > comments: https://issues.apache.org/jira/browse/BEAM-3942
 >      > PR changing the commands to gradle is waiting for PerfKit
devs review
 >      > here:
 > https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648
 >      >
 >      > Best regards,
 >      >
 >      > 2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau
 >     mailto:rmannibu...@gmail.com>
>
 >      >        >
 >      >     Hi Scott
 >      >
 >      >     While
 > https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
 >      >
 > 
   is

 >      >     open, gradle is a concurrent of maven but maven must
stay the
 >     default build
 >      >     tool cause gradle breaks users.
 >      >
 >      >
 >      >     Le 1 mai 2018 01:59, "Scott Wegner"
mailto:sweg...@google.com>
 >     >
 >      >     
     écrit :
 >      >
 >      >         Many many of you have been hacking diligently on the
 >     Gradle build, and
 >      >         I'm happy to announce that we now have a
 >     fully-functioning Gradle build!
 >      >         There's been a t

Re: Gradle Status: Migrated!

[Henning Rohde]

JB - for your comparison, please also omit cross-compiling all the Go
examples because they are only built using Gradle.




On Tue, May 1, 2018 at 8:59 AM Jean-Baptiste Onofré  wrote:

> Thanks for the update Kenn, that makes sense.
>
> I'm checking the artifacts generated by Gradle right now.
>
> Regards
> JB
>
> On 01/05/2018 17:42, Kenneth Knowles wrote:
> > Raw execution time for tasks from clean is not the only thing to test. I
> > would say it is not even important. Try these from clean:
> >
> >   - Gradle: ./gradlew :beam-sdks-java-io-mongodb:test && ./gradlew
> > :beam-sdks-java-io-mongodb:test
> >   - Maven: mvn -pl sdks/java/io/mongodb test -am && mvn -pl
> > sdks/java/io/mongodb test -am
> >
> > Quick run on my laptop:
> >
> >   - Gradle: 66s (65s then 1s)
> >   - Maven: 317s (173s then 144s)
> >
> > Of course, the mvn command runs a bunch of useless executions AND it is
> > incorrect because it isn't using built jars. That's part of the point -
> > there is no way to do what you want with mvn. Let's try to make a
> > command that avoids useless work and builds the jars:
> >
> >   - Maven:  (mvn -pl sdks/java/io/mongodb install -DskipTests -am && mvn
> > -pl sdks/java/io/mongodb test) && (each time)
> >
> > That takes 102s the first time and 64s the second time. And that is
> > about the realistic workflow for someone trying to get something done.
> > Even if we touch a file Gradle finishes in 20s. So the best case for mvn
> > is this head-to-head:
> >
> >   - Gradle: 65s + 20s + 20s + 20s + 20s + ...
> >   - Maven: 102s + 64s + 64s + 64s + 64s + ...
> >
> > Kenn
> >
> >
> > On Tue, May 1, 2018 at 8:09 AM Jean-Baptiste Onofré  > > wrote:
> >
> > Thanks, for me, Maven 3.5.2 takes quite the same time than Gradle
> > (using
> > the wrapper). It's maybe related to my environment.
> >
> > Anyway, I'm doing a complete build review both in term of building
> > time,
> > and equivalence (artifacts publishing, test, plugin execution).
> >
> > I will provide an update soon.
> >
> > Regards
> > JB
> >
> > On 01/05/2018 16:57, Reuven Lax wrote:
> >  > Luke did gather data which showed that on our Jenkins executors
> the
> >  > Gradle build was much faster than the Maven build. Also right now
> we
> >  > have incremental builds turned off, but once we're confident
> > enough to
> >  > enable them (at least for local development) that will often drop
> > build
> >  > times a lot.
> >  >
> >  > On Tue, May 1, 2018 at 4:01 AM Jean-Baptiste Onofré
> > mailto:j...@nanthrax.net>
> >  > >> wrote:
> >  >
> >  > By the way, I'm curious: did someone evaluate the build time
> gap
> >  > between Maven
> >  > and Gradle ? One of the main reason to migrate to Gradle was
> > the inc
> >  > build and
> >  > build time. The builds I have launched are quite the same in
> >  > duration. I will do
> >  > deeper tests to evaluate the gap.
> >  >
> >  > Regards
> >  > JB
> >  >
> >  > On 05/01/2018 12:48 PM, Łukasz Gajowy wrote:
> >  >  > Hi Scott,
> >  >  >
> >  >  > thanks for the update! Just a clarification about IO
> > performance
> >  > tests: those
> >  >  > were fully migrated in Beam and all task necessary for
> running
> >  > them are there
> >  >  > but Jenkins jobs still run mvn commands. This is due the
> > fact that
> >  >  > PerfkitBenchmarker code (which is invoked by Jenkins and
> >  > constructs the commands
> >  >  > by itself) was not updated yet. This should be finished
> before
> >  > fully dropping mvn.
> >  >  >
> >  >  > More on that topic here, in
> >  >  > comments: https://issues.apache.org/jira/browse/BEAM-3942
> >  >  > PR changing the commands to gradle is waiting for PerfKit
> > devs review
> >  >  > here:
> >  >
> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648
> >  >  >
> >  >  > Best regards,
> >  >  >
> >  >  > 2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau
> >  > mailto:rmannibu...@gmail.com>
> > >
> >  >  >  >   >  >  >  >
> >  >  > Hi Scott
> >  >  >
> >  >  > While
> >  >
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
> >  >  >
> >  >
> >   <
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057> is
> >  >  > open, gradle is a concurrent of maven but maven must
> > stay the
> >  > default build
> >  > 

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

Hi Scott,

is it right that 2 doesn't handle the hierachy anymore and that it doesn't
handle profiles for runners as it is currently with maven?


Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book


2018-04-06 18:32 GMT+02:00 Scott Wegner :

> I wanted to start a thread to summarize the current state of Gradle
> migration. We've made lots of good progress so far this week. Here's the
> status from what I can tell-- please add or correct anything I missed:
>
> * Release artifacts can be built and published for Snapshot and officlal
> releases [1]
> * Gradle-generated releases have been validated with the the Apache Beam
> archetype generation quickstart; still needs additional validation.
> * Generated release pom files have correct project metadata [2]
> * The python pre-commits are now working in Gradle [3]
> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
> learn the new system-- please add your own. This will eventually feed into
> official documentation on the website.
> * Łukasz Gajowy is working on migrating performance testing framework [5]
> * Daniel is working on updating documentation to refer to Gradle instead
> of maven
>
> If I missed anything, please add it to this thread.
>
> The general roadmap we're working towards is:
> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to Gradle
> (d) Migrate perfkit suites to use Gradle
>
> For those of you that are hacking: thanks for your help so far! Progress
> is being roughly tracked on the Kanban [6]; please make sure the issues
> assigned to you are up-to-date. Many of the changes are staged on
> lukecwik's local branch [7]; we'll work on merging them back soon.
>
>
> [1] https://github.com/lukecwik/incubator-beam/pull/7
> [2] https://github.com/lukecwik/incubator-beam/pull/3
> [3] https://github.com/apache/beam/pull/5032
> [4] https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDf
> RDVkxzeDlbdVSQ/edit
> [5] https://github.com/apache/beam/pull/5003
> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
> --
>
>
> Got feedback? http://go/swegner-feedback
>

Re: Gradle Status [April 6]

[Lukasz Cwik]

Romain, are you talking about the profiles that exist as part of the
archetype examples?

If so, then those still exist and haven't been changed. If not, can you
provide a link to the profile in a pom file to be clearer?

On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau 
wrote:

> Hi Scott,
>
> is it right that 2 doesn't handle the hierachy anymore and that it doesn't
> handle profiles for runners as it is currently with maven?
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-04-06 18:32 GMT+02:00 Scott Wegner :
>
>> I wanted to start a thread to summarize the current state of Gradle
>> migration. We've made lots of good progress so far this week. Here's the
>> status from what I can tell-- please add or correct anything I missed:
>>
>> * Release artifacts can be built and published for Snapshot and officlal
>> releases [1]
>> * Gradle-generated releases have been validated with the the Apache Beam
>> archetype generation quickstart; still needs additional validation.
>> * Generated release pom files have correct project metadata [2]
>> * The python pre-commits are now working in Gradle [3]
>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>> learn the new system-- please add your own. This will eventually feed into
>> official documentation on the website.
>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>> * Daniel is working on updating documentation to refer to Gradle instead
>> of maven
>>
>> If I missed anything, please add it to this thread.
>>
>> The general roadmap we're working towards is:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> For those of you that are hacking: thanks for your help so far! Progress
>> is being roughly tracked on the Kanban [6]; please make sure the issues
>> assigned to you are up-to-date. Many of the changes are staged on
>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>
>>
>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>> [3] https://github.com/apache/beam/pull/5032
>> [4]
>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
>> [5] https://github.com/apache/beam/pull/5003
>> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
>> --
>>
>>
>> Got feedback? http://go/swegner-feedback
>>
>
>

Re: Gradle Status [April 6]

[Kenneth Knowles]

I'm working on finding a solution for launching the Nexmark suite with each
runner. This doesn't have to be done via Gradle, but we anyhow need built
artifacts that don't require user classpath intervention.

It looks to me like the examples are also missing this - they have separate
configuration e.g. sparkRunnerPreCommit but that is overspecified compared
to a free-form launching of a main() program with a runner profile.

On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  wrote:

> Romain, are you talking about the profiles that exist as part of the
> archetype examples?
>
> If so, then those still exist and haven't been changed. If not, can you
> provide a link to the profile in a pom file to be clearer?
>
> On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau 
> wrote:
>
>> Hi Scott,
>>
>> is it right that 2 doesn't handle the hierachy anymore and that it
>> doesn't handle profiles for runners as it is currently with maven?
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>> 2018-04-06 18:32 GMT+02:00 Scott Wegner :
>>
>>> I wanted to start a thread to summarize the current state of Gradle
>>> migration. We've made lots of good progress so far this week. Here's the
>>> status from what I can tell-- please add or correct anything I missed:
>>>
>>> * Release artifacts can be built and published for Snapshot and officlal
>>> releases [1]
>>> * Gradle-generated releases have been validated with the the Apache Beam
>>> archetype generation quickstart; still needs additional validation.
>>> * Generated release pom files have correct project metadata [2]
>>> * The python pre-commits are now working in Gradle [3]
>>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>>> learn the new system-- please add your own. This will eventually feed into
>>> official documentation on the website.
>>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>>> * Daniel is working on updating documentation to refer to Gradle instead
>>> of maven
>>>
>>> If I missed anything, please add it to this thread.
>>>
>>> The general roadmap we're working towards is:
>>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>>> (b) Postcommits migrated to Gradle
>>> (c) Migrate documentation from maven to Gradle
>>> (d) Migrate perfkit suites to use Gradle
>>>
>>> For those of you that are hacking: thanks for your help so far! Progress
>>> is being roughly tracked on the Kanban [6]; please make sure the issues
>>> assigned to you are up-to-date. Many of the changes are staged on
>>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>>
>>>
>>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>>> [3] https://github.com/apache/beam/pull/5032
>>> [4]
>>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
>>> [5] https://github.com/apache/beam/pull/5003
>>> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>>> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
>>> --
>>>
>>>
>>> Got feedback? http://go/swegner-feedback
>>>
>>
>>

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

Le 6 avr. 2018 20:09, "Lukasz Cwik"  a écrit :

Romain, are you talking about the profiles that exist as part of the
archetype examples?


Was more thinking to this kind of profiles
https://github.com/apache/beam/blob/master/sdks/java/nexmark/pom.xml (which
should hit all IO at some point to ensure their portability)

Idea is to be able to extract the deps for a runner from a particular pom
since it sometimes requires some dependencies work for conflicts.





If so, then those still exist and haven't been changed. If not, can you
provide a link to the profile in a pom file to be clearer?

On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau 
wrote:

> Hi Scott,
>
> is it right that 2 doesn't handle the hierachy anymore and that it doesn't
> handle profiles for runners as it is currently with maven?
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-04-06 18:32 GMT+02:00 Scott Wegner :
>
>> I wanted to start a thread to summarize the current state of Gradle
>> migration. We've made lots of good progress so far this week. Here's the
>> status from what I can tell-- please add or correct anything I missed:
>>
>> * Release artifacts can be built and published for Snapshot and officlal
>> releases [1]
>> * Gradle-generated releases have been validated with the the Apache Beam
>> archetype generation quickstart; still needs additional validation.
>> * Generated release pom files have correct project metadata [2]
>> * The python pre-commits are now working in Gradle [3]
>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>> learn the new system-- please add your own. This will eventually feed into
>> official documentation on the website.
>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>> * Daniel is working on updating documentation to refer to Gradle instead
>> of maven
>>
>> If I missed anything, please add it to this thread.
>>
>> The general roadmap we're working towards is:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> For those of you that are hacking: thanks for your help so far! Progress
>> is being roughly tracked on the Kanban [6]; please make sure the issues
>> assigned to you are up-to-date. Many of the changes are staged on
>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>
>>
>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>> [3] https://github.com/apache/beam/pull/5032
>> [4] https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDf
>> RDVkxzeDlbdVSQ/edit
>> [5] https://github.com/apache/beam/pull/5003
>> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
>> --
>>
>>
>> Got feedback? http://go/swegner-feedback
>>
>
>

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

Why building a zip per runner which its stack and just pointing out on that
zip and let beam lazy load the runner:

--runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or the
fromSystemProperties() if it gets merged a day ;))

Le 6 avr. 2018 20:21, "Kenneth Knowles"  a écrit :

> I'm working on finding a solution for launching the Nexmark suite with
> each runner. This doesn't have to be done via Gradle, but we anyhow need
> built artifacts that don't require user classpath intervention.
>
> It looks to me like the examples are also missing this - they have
> separate configuration e.g. sparkRunnerPreCommit but that is overspecified
> compared to a free-form launching of a main() program with a runner profile.
>
> On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  wrote:
>
>> Romain, are you talking about the profiles that exist as part of the
>> archetype examples?
>>
>> If so, then those still exist and haven't been changed. If not, can you
>> provide a link to the profile in a pom file to be clearer?
>>
>> On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau 
>> wrote:
>>
>>> Hi Scott,
>>>
>>> is it right that 2 doesn't handle the hierachy anymore and that it
>>> doesn't handle profiles for runners as it is currently with maven?
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau  |  Blog
>>>  | Old Blog
>>>  | Github
>>>  | LinkedIn
>>>  | Book
>>> 
>>>
>>> 2018-04-06 18:32 GMT+02:00 Scott Wegner :
>>>
 I wanted to start a thread to summarize the current state of Gradle
 migration. We've made lots of good progress so far this week. Here's the
 status from what I can tell-- please add or correct anything I missed:

 * Release artifacts can be built and published for Snapshot and
 officlal releases [1]
 * Gradle-generated releases have been validated with the the Apache
 Beam archetype generation quickstart; still needs additional validation.
 * Generated release pom files have correct project metadata [2]
 * The python pre-commits are now working in Gradle [3]
 * Ismaël has started a collaborative doc of Gradle tips [4] as we all
 learn the new system-- please add your own. This will eventually feed into
 official documentation on the website.
 * Łukasz Gajowy is working on migrating performance testing framework
 [5]
 * Daniel is working on updating documentation to refer to Gradle
 instead of maven

 If I missed anything, please add it to this thread.

 The general roadmap we're working towards is:
 (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
 (b) Postcommits migrated to Gradle
 (c) Migrate documentation from maven to Gradle
 (d) Migrate perfkit suites to use Gradle

 For those of you that are hacking: thanks for your help so far!
 Progress is being roughly tracked on the Kanban [6]; please make sure the
 issues assigned to you are up-to-date. Many of the changes are staged on
 lukecwik's local branch [7]; we'll work on merging them back soon.


 [1] https://github.com/lukecwik/incubator-beam/pull/7
 [2] https://github.com/lukecwik/incubator-beam/pull/3
 [3] https://github.com/apache/beam/pull/5032
 [4] https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDf
 RDVkxzeDlbdVSQ/edit
 [5] https://github.com/apache/beam/pull/5003
 [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242

 [7] https://github.com/lukecwik/incubator-beam/tree/gradle
 --


 Got feedback? http://go/swegner-feedback

>>>
>>>

Re: Gradle Status [April 6]

[Scott Wegner]

Here's an end-of-day update on migration work:

* Snapshot unsigned dailies and signed release builds are working (!!).
PR/5048 [1] merges changes from Luke's branch
  * python precommit failing... will investigate python precommit Monday
* All Precommits are gradle only
* All Postcommits except performance tests and Java_JDK_Versions_Test  use
gradle (after PR/5047 [2] merged)
* Nightly snapshot release using gradle is ready; needs PR/5048 to be
merged before switching
* ValidatesRunner_Spark failing consistently; investigating

Thanks for another productive day of hacking. I'll pick up again on Monday.

[1] https://github.com/apache/beam/pull/5048
[2] https://github.com/apache/beam/pull/5047


On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau 
wrote:

> Why building a zip per runner which its stack and just pointing out on
> that zip and let beam lazy load the runner:
>
> --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or the
> fromSystemProperties() if it gets merged a day ;))
>
> Le 6 avr. 2018 20:21, "Kenneth Knowles"  a écrit :
>
>> I'm working on finding a solution for launching the Nexmark suite with
>> each runner. This doesn't have to be done via Gradle, but we anyhow need
>> built artifacts that don't require user classpath intervention.
>>
>> It looks to me like the examples are also missing this - they have
>> separate configuration e.g. sparkRunnerPreCommit but that is overspecified
>> compared to a free-form launching of a main() program with a runner profile.
>>
>> On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  wrote:
>>
>>> Romain, are you talking about the profiles that exist as part of the
>>> archetype examples?
>>>
>>> If so, then those still exist and haven't been changed. If not, can you
>>> provide a link to the profile in a pom file to be clearer?
>>>
>>> On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Hi Scott,

 is it right that 2 doesn't handle the hierachy anymore and that it
 doesn't handle profiles for runners as it is currently with maven?


 Romain Manni-Bucau
 @rmannibucau  |  Blog
  | Old Blog
  | Github
  | LinkedIn
  | Book
 

 2018-04-06 18:32 GMT+02:00 Scott Wegner :

> I wanted to start a thread to summarize the current state of Gradle
> migration. We've made lots of good progress so far this week. Here's the
> status from what I can tell-- please add or correct anything I missed:
>
> * Release artifacts can be built and published for Snapshot and
> officlal releases [1]
> * Gradle-generated releases have been validated with the the Apache
> Beam archetype generation quickstart; still needs additional validation.
> * Generated release pom files have correct project metadata [2]
> * The python pre-commits are now working in Gradle [3]
> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
> learn the new system-- please add your own. This will eventually feed into
> official documentation on the website.
> * Łukasz Gajowy is working on migrating performance testing framework
> [5]
> * Daniel is working on updating documentation to refer to Gradle
> instead of maven
>
> If I missed anything, please add it to this thread.
>
> The general roadmap we're working towards is:
> (a) Publish release artifacts with Gradle (SNAPSHOT and signed
> releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to Gradle
> (d) Migrate perfkit suites to use Gradle
>
> For those of you that are hacking: thanks for your help so far!
> Progress is being roughly tracked on the Kanban [6]; please make sure the
> issues assigned to you are up-to-date. Many of the changes are staged on
> lukecwik's local branch [7]; we'll work on merging them back soon.
>
>
> [1] https://github.com/lukecwik/incubator-beam/pull/7
> [2] https://github.com/lukecwik/incubator-beam/pull/3
> [3] https://github.com/apache/beam/pull/5032
> [4]
> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
> [5] https://github.com/apache/beam/pull/5003
> [6]
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
> --
>
>
> Got feedback? http://go/swegner-feedback
>

 --


Got feedback? http://go/swegner-feedback

Re: Gradle Status [April 6]

[Reuven Lax]

So if I understand correctly, we've migrated all precommit, most
postcommits, and we have a working release process using Gradle. There are
a few bugs left, but at this pace it sounds like we're close to fully
migrated.

I know that multiple people put it long hours last getting this done last
week (just look at the Slack messages!). This is awesome progress, and a
hearty thank you to everyone who put in their time.

Reuven

On Fri, Apr 6, 2018 at 7:52 PM Scott Wegner  wrote:

> Here's an end-of-day update on migration work:
>
> * Snapshot unsigned dailies and signed release builds are working (!!).
> PR/5048 [1] merges changes from Luke's branch
>   * python precommit failing... will investigate python precommit Monday
> * All Precommits are gradle only
> * All Postcommits except performance tests and Java_JDK_Versions_Test  use
> gradle (after PR/5047 [2] merged)
> * Nightly snapshot release using gradle is ready; needs PR/5048 to be
> merged before switching
> * ValidatesRunner_Spark failing consistently; investigating
>
> Thanks for another productive day of hacking. I'll pick up again on Monday.
>
> [1] https://github.com/apache/beam/pull/5048
> [2] https://github.com/apache/beam/pull/5047
>
>
> On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau 
> wrote:
>
>> Why building a zip per runner which its stack and just pointing out on
>> that zip and let beam lazy load the runner:
>>
>> --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or the
>> fromSystemProperties() if it gets merged a day ;))
>>
>> Le 6 avr. 2018 20:21, "Kenneth Knowles"  a écrit :
>>
>>> I'm working on finding a solution for launching the Nexmark suite with
>>> each runner. This doesn't have to be done via Gradle, but we anyhow need
>>> built artifacts that don't require user classpath intervention.
>>>
>>> It looks to me like the examples are also missing this - they have
>>> separate configuration e.g. sparkRunnerPreCommit but that is overspecified
>>> compared to a free-form launching of a main() program with a runner profile.
>>>
>>> On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  wrote:
>>>
 Romain, are you talking about the profiles that exist as part of the
 archetype examples?

 If so, then those still exist and haven't been changed. If not, can you
 provide a link to the profile in a pom file to be clearer?

 On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Hi Scott,
>
> is it right that 2 doesn't handle the hierachy anymore and that it
> doesn't handle profiles for runners as it is currently with maven?
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-04-06 18:32 GMT+02:00 Scott Wegner :
>
>> I wanted to start a thread to summarize the current state of Gradle
>> migration. We've made lots of good progress so far this week. Here's the
>> status from what I can tell-- please add or correct anything I missed:
>>
>> * Release artifacts can be built and published for Snapshot and
>> officlal releases [1]
>> * Gradle-generated releases have been validated with the the Apache
>> Beam archetype generation quickstart; still needs additional validation.
>> * Generated release pom files have correct project metadata [2]
>> * The python pre-commits are now working in Gradle [3]
>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>> learn the new system-- please add your own. This will eventually feed 
>> into
>> official documentation on the website.
>> * Łukasz Gajowy is working on migrating performance testing framework
>> [5]
>> * Daniel is working on updating documentation to refer to Gradle
>> instead of maven
>>
>> If I missed anything, please add it to this thread.
>>
>> The general roadmap we're working towards is:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed
>> releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> For those of you that are hacking: thanks for your help so far!
>> Progress is being roughly tracked on the Kanban [6]; please make sure the
>> issues assigned to you are up-to-date. Many of the changes are staged on
>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>
>>
>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>> [3] https://github.com/apache/beam/pull/5032
>> [4]
>> https://docs.goog

Re: Gradle Status [April 6]

[Dan Halperin]

On Sat, Apr 7, 2018 at 12:43 Reuven Lax  wrote:

> So if I understand correctly, we've migrated all precommit, most
> postcommits, and we have a working release process using Gradle. There are
> a few bugs left, but at this pace it sounds like we're close to fully
> migrated.
>
> I know that multiple people put it long hours last getting this done last
> week (just look at the Slack messages!). This is awesome progress, and a
> hearty thank you to everyone who put in their time.
>
> Reuven
>
> On Fri, Apr 6, 2018 at 7:52 PM Scott Wegner  wrote:
>
>> Here's an end-of-day update on migration work:
>>
>> * Snapshot unsigned dailies and signed release builds are working (!!).
>> PR/5048 [1] merges changes from Luke's branch
>>   * python precommit failing... will investigate python precommit Monday
>> * All Precommits are gradle only
>> * All Postcommits except performance tests and Java_JDK_Versions_Test
>> use gradle (after PR/5047 [2] merged)
>> * Nightly snapshot release using gradle is ready; needs PR/5048 to be
>> merged before switching
>> * ValidatesRunner_Spark failing consistently; investigating
>>
>> Thanks for another productive day of hacking. I'll pick up again on
>> Monday.
>>
>> [1] https://github.com/apache/beam/pull/5048
>> [2] https://github.com/apache/beam/pull/5047
>>
>>
This is really phenomenal work, and a huge boost to the community. Thanks,
everyone who participated!
Dan


>
>> On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau 
>> wrote:
>>
>>> Why building a zip per runner which its stack and just pointing out on
>>> that zip and let beam lazy load the runner:
>>>
>>> --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or the
>>> fromSystemProperties() if it gets merged a day ;))
>>>
>>> Le 6 avr. 2018 20:21, "Kenneth Knowles"  a écrit :
>>>
 I'm working on finding a solution for launching the Nexmark suite with
 each runner. This doesn't have to be done via Gradle, but we anyhow need
 built artifacts that don't require user classpath intervention.

 It looks to me like the examples are also missing this - they have
 separate configuration e.g. sparkRunnerPreCommit but that is overspecified
 compared to a free-form launching of a main() program with a runner 
 profile.

 On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  wrote:

> Romain, are you talking about the profiles that exist as part of the
> archetype examples?
>
> If so, then those still exist and haven't been changed. If not, can
> you provide a link to the profile in a pom file to be clearer?
>
> On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Hi Scott,
>>
>> is it right that 2 doesn't handle the hierachy anymore and that it
>> doesn't handle profiles for runners as it is currently with maven?
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>> 2018-04-06 18:32 GMT+02:00 Scott Wegner :
>>
>>> I wanted to start a thread to summarize the current state of Gradle
>>> migration. We've made lots of good progress so far this week. Here's the
>>> status from what I can tell-- please add or correct anything I missed:
>>>
>>> * Release artifacts can be built and published for Snapshot and
>>> officlal releases [1]
>>> * Gradle-generated releases have been validated with the the Apache
>>> Beam archetype generation quickstart; still needs additional validation.
>>> * Generated release pom files have correct project metadata [2]
>>> * The python pre-commits are now working in Gradle [3]
>>> * Ismaël has started a collaborative doc of Gradle tips [4] as we
>>> all learn the new system-- please add your own. This will eventually 
>>> feed
>>> into official documentation on the website.
>>> * Łukasz Gajowy is working on migrating performance testing
>>> framework [5]
>>> * Daniel is working on updating documentation to refer to Gradle
>>> instead of maven
>>>
>>> If I missed anything, please add it to this thread.
>>>
>>> The general roadmap we're working towards is:
>>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed
>>> releases)
>>> (b) Postcommits migrated to Gradle
>>> (c) Migrate documentation from maven to Gradle
>>> (d) Migrate perfkit suites to use Gradle
>>>
>>> For those of you that are hacking: thanks for your help so far!
>>> Progress is being roughly tracked on the Kanban [6]; please make sure 
>>> the
>>> issues assigned to you are up-to-date. Many of the changes are staged on

Re: Gradle Status [April 6]

[Jean-Baptiste Onofré]

Hi all,

I did multiple gradle build since last week and I would like to share 
one of my concern: it's about the communities.


If I think our users won't see any change for them due to Gradle build 
(I think that most of our users will still use Maven with artifacts 
provided by Gradle), I'm more concerned by the dev community and the 
contribution.


Maven is well known and straight forward for a large part of potential 
contributors. I think we have to keep in mind that we still have to grow 
up our contributors community.


Today, maybe I'm wrong, but I have the feeling that gradle build is not 
straight forward (build.gradle includes build_rules.gradle, gathering 
all taks all together).


I would like to add a task in the gradle "migration" process: simplify 
the gradle structure and files, and document this.


I know we already have a Jira about the documentation part, but I would 
like to "polish" and use a clean structure for the Gradle resources. As 
already quickly discussed, I think that having one gradle file per tasks 
in the .gradle directory would be helpful.


The goal is really to simplify the contribution.

Do you agree if I add a Jira about "Gradle polish" ?
Thoughts ?

Regards
JB

On 07/04/2018 04:52, Scott Wegner wrote:

Here's an end-of-day update on migration work:

* Snapshot unsigned dailies and signed release builds are working (!!). 
PR/5048 [1] merges changes from Luke's branch

   * python precommit failing... will investigate python precommit Monday
* All Precommits are gradle only
* All Postcommits except performance tests and Java_JDK_Versions_Test  
use gradle (after PR/5047 [2] merged)
* Nightly snapshot release using gradle is ready; needs PR/5048 to be 
merged before switching

* ValidatesRunner_Spark failing consistently; investigating

Thanks for another productive day of hacking. I'll pick up again on Monday.

[1] https://github.com/apache/beam/pull/5048
[2] https://github.com/apache/beam/pull/5047


On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau 
mailto:rmannibu...@gmail.com>> wrote:


Why building a zip per runner which its stack and just pointing out
on that zip and let beam lazy load the runner:

--runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or
the fromSystemProperties() if it gets merged a day ;))

Le 6 avr. 2018 20:21, "Kenneth Knowles" mailto:k...@google.com>> a écrit :

I'm working on finding a solution for launching the Nexmark
suite with each runner. This doesn't have to be done via Gradle,
but we anyhow need built artifacts that don't require user
classpath intervention.

It looks to me like the examples are also missing this - they
have separate configuration e.g. sparkRunnerPreCommit but that
is overspecified compared to a free-form launching of a main()
program with a runner profile.

On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik mailto:lc...@google.com>> wrote:

Romain, are you talking about the profiles that exist as
part of the archetype examples?

If so, then those still exist and haven't been changed. If
not, can you provide a link to the profile in a pom file to
be clearer?

On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau
mailto:rmannibu...@gmail.com>> wrote:

Hi Scott,

is it right that 2 doesn't handle the hierachy anymore
and that it doesn't handle profiles for runners as it is
currently with maven?


Romain Manni-Bucau
@rmannibucau  | Blog
 | Old Blog
 | Github
 | LinkedIn
 | Book



2018-04-06 18:32 GMT+02:00 Scott Wegner
mailto:sweg...@google.com>>:

I wanted to start a thread to summarize the current
state of Gradle migration. We've made lots of good
progress so far this week. Here's the status from
what I can tell-- please add or correct anything I
missed:

* Release artifacts can be built and published for
Snapshot and officlal releases [1]
* Gradle-generated releases have been validated with
the the Apache Beam archetype generation quickstart;
still needs additional validation.
* Generated release pom files have correct project
metadata [2]
* The python pre-commits are now working in Gradle [3]
* Ismaël has starte

Re: Gradle Status [April 6]

[Kenneth Knowles]

Huge +1 to the idea of investing in simplification and polish of the Gradle
files before considering the migration complete.

1. build.gradle files should be as close to straight-line configuration as
possible:
 - You should be able to understand a module's build easily, locally,
without knowing Groovy
 - As little Groovy programming as possible, focused on providing
well-defined utilities

2. Explicit > implicit, especially for config files
 - Ditto about being able to understand a module's build locally
 - Minimize globally inherited config, in favor of explicitly calling
utilities
 - Passing current project to a utility is better than trying to make
something the "closure"
 - If you are going to refer to an identifier, it should have an explicit
point of origin (not strings smashed together in a loop)

When in doubt, a little redundancy to improve readability is worth it in
this context.

Kenn

On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré  wrote:

> Hi all,
>
> I did multiple gradle build since last week and I would like to share
> one of my concern: it's about the communities.
>
> If I think our users won't see any change for them due to Gradle build
> (I think that most of our users will still use Maven with artifacts
> provided by Gradle), I'm more concerned by the dev community and the
> contribution.
>
> Maven is well known and straight forward for a large part of potential
> contributors. I think we have to keep in mind that we still have to grow
> up our contributors community.
>
> Today, maybe I'm wrong, but I have the feeling that gradle build is not
> straight forward (build.gradle includes build_rules.gradle, gathering
> all taks all together).
>
> I would like to add a task in the gradle "migration" process: simplify
> the gradle structure and files, and document this.
>
> I know we already have a Jira about the documentation part, but I would
> like to "polish" and use a clean structure for the Gradle resources. As
> already quickly discussed, I think that having one gradle file per tasks
> in the .gradle directory would be helpful.
>
> The goal is really to simplify the contribution.
>
> Do you agree if I add a Jira about "Gradle polish" ?
> Thoughts ?
>
> Regards
> JB
>
> On 07/04/2018 04:52, Scott Wegner wrote:
> > Here's an end-of-day update on migration work:
> >
> > * Snapshot unsigned dailies and signed release builds are working (!!).
> > PR/5048 [1] merges changes from Luke's branch
> >* python precommit failing... will investigate python precommit Monday
> > * All Precommits are gradle only
> > * All Postcommits except performance tests and Java_JDK_Versions_Test
> > use gradle (after PR/5047 [2] merged)
> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be
> > merged before switching
> > * ValidatesRunner_Spark failing consistently; investigating
> >
> > Thanks for another productive day of hacking. I'll pick up again on
> Monday.
> >
> > [1] https://github.com/apache/beam/pull/5048
> > [2] https://github.com/apache/beam/pull/5047
> >
> >
> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
> > mailto:rmannibu...@gmail.com>> wrote:
> >
> > Why building a zip per runner which its stack and just pointing out
> > on that zip and let beam lazy load the runner:
> >
> > --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or
> > the fromSystemProperties() if it gets merged a day ;))
> >
> > Le 6 avr. 2018 20:21, "Kenneth Knowles"  > > a écrit :
> >
> > I'm working on finding a solution for launching the Nexmark
> > suite with each runner. This doesn't have to be done via Gradle,
> > but we anyhow need built artifacts that don't require user
> > classpath intervention.
> >
> > It looks to me like the examples are also missing this - they
> > have separate configuration e.g. sparkRunnerPreCommit but that
> > is overspecified compared to a free-form launching of a main()
> > program with a runner profile.
> >
> > On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  > > wrote:
> >
> > Romain, are you talking about the profiles that exist as
> > part of the archetype examples?
> >
> > If so, then those still exist and haven't been changed. If
> > not, can you provide a link to the profile in a pom file to
> > be clearer?
> >
> > On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau
> > mailto:rmannibu...@gmail.com>>
> wrote:
> >
> > Hi Scott,
> >
> > is it right that 2 doesn't handle the hierachy anymore
> > and that it doesn't handle profiles for runners as it is
> > currently with maven?
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau  | Blog
> > 

Re: Gradle Status [April 6]

[Lukasz Cwik]

JB, learning the build system in a project is hopefully avoided by most
users if the contribution guide can clearly explain what users need to do.
But for everyone else who wants to change a dependency version or add a
dependency it should be as simple as copy/paste (which I believe it already
is). Note that people who want to do anything more complicated like add new
jenkins job configurations for new integration test targets needs to learn
how the build system works, how maven profiles work, how to plumb the
required set of attributes from Jenkins into the test target via the build
system.

Kenn, I have to disagree with a lot in 1 and 2. For users who are
unfamiliar with Maven, we didn't setup the Maven build in such a way that
anyone unfamiliar with Maven knew what was going on or try to use concepts
unnatural to Maven in an attempt to make it seem easier. I believe we
should stick with Gradle/Groovy conventions. Users who are not familiar
with how Gradle/Groovy works will either ask the community or look at
StackOverflow and doing things like passing the project around into
functions is extremely uncommon compared to using the current context and
applying closures to it. For users who are familiar with Gradle, the builds
should be natural Gradle.

*- If you are going to refer to an identifier, it should have an explicit
point of origin (not strings smashed together in a loop)*
I assume your referring to how the examples java precommit is done. It is
the only case of it to my knowledge and extra hands to migrate it to be
like how the validates runner tests run would be useful. Filed
https://issues.apache.org/jira/browse/BEAM-4033


On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles  wrote:

> Huge +1 to the idea of investing in simplification and polish of the
> Gradle files before considering the migration complete.
>
> 1. build.gradle files should be as close to straight-line configuration as
> possible:
>  - You should be able to understand a module's build easily, locally,
> without knowing Groovy
>  - As little Groovy programming as possible, focused on providing
> well-defined utilities
>
> 2. Explicit > implicit, especially for config files
>  - Ditto about being able to understand a module's build locally
>  - Minimize globally inherited config, in favor of explicitly calling
> utilities
>  - Passing current project to a utility is better than trying to make
> something the "closure"
>  - If you are going to refer to an identifier, it should have an explicit
> point of origin (not strings smashed together in a loop)
>
> When in doubt, a little redundancy to improve readability is worth it in
> this context.
>
> Kenn
>
> On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré 
> wrote:
>
>> Hi all,
>>
>> I did multiple gradle build since last week and I would like to share
>> one of my concern: it's about the communities.
>>
>> If I think our users won't see any change for them due to Gradle build
>> (I think that most of our users will still use Maven with artifacts
>> provided by Gradle), I'm more concerned by the dev community and the
>> contribution.
>>
>> Maven is well known and straight forward for a large part of potential
>> contributors. I think we have to keep in mind that we still have to grow
>> up our contributors community.
>>
>> Today, maybe I'm wrong, but I have the feeling that gradle build is not
>> straight forward (build.gradle includes build_rules.gradle, gathering
>> all taks all together).
>>
>> I would like to add a task in the gradle "migration" process: simplify
>> the gradle structure and files, and document this.
>>
>> I know we already have a Jira about the documentation part, but I would
>> like to "polish" and use a clean structure for the Gradle resources. As
>> already quickly discussed, I think that having one gradle file per tasks
>> in the .gradle directory would be helpful.
>>
>> The goal is really to simplify the contribution.
>>
>> Do you agree if I add a Jira about "Gradle polish" ?
>> Thoughts ?
>>
>> Regards
>> JB
>>
>> On 07/04/2018 04:52, Scott Wegner wrote:
>> > Here's an end-of-day update on migration work:
>> >
>> > * Snapshot unsigned dailies and signed release builds are working (!!).
>> > PR/5048 [1] merges changes from Luke's branch
>> >* python precommit failing... will investigate python precommit
>> Monday
>> > * All Precommits are gradle only
>> > * All Postcommits except performance tests and Java_JDK_Versions_Test
>> > use gradle (after PR/5047 [2] merged)
>> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be
>> > merged before switching
>> > * ValidatesRunner_Spark failing consistently; investigating
>> >
>> > Thanks for another productive day of hacking. I'll pick up again on
>> Monday.
>> >
>> > [1] https://github.com/apache/beam/pull/5048
>> > [2] https://github.com/apache/beam/pull/5047
>> >
>> >
>> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
>> > mailto:rmannibu...@gmail.com>> wrote:
>> >
>> > Why buildin

Re: Gradle Status [April 6]

[Lukasz Cwik]

Fixed up prior e-mail.

On Mon, Apr 9, 2018 at 1:50 PM Lukasz Cwik  wrote:

> JB, learning the build system in a project is hopefully avoided by most
> users if the contribution guide can clearly explain what users need to do.
> But for everyone else who wants to change a dependency version or add a
> dependency it should be as simple as copy/paste (which I believe it already
> is). Note that people who want to do anything more complicated like add new
> jenkins job configurations for new integration test targets needs to learn
> how the build system works, how to plumb the required set of attributes
> from Jenkins into the test target via the build system.
>
> Kenn, I have to disagree with a lot in 1 and 2. For users who are
> unfamiliar with Maven, we didn't setup the Maven build in such a way that
> anyone unfamiliar with Maven knew what was going on or try to use concepts
> unnatural to Maven in an attempt to make it seem easier. I believe we
> should stick with Gradle/Groovy conventions. Users who are not familiar
> with how Gradle/Groovy works will either ask the community or look at
> StackOverflow and doing things like passing the project around into
> functions is extremely uncommon compared to using the current context and
> applying closures to it. For users who are familiar with Gradle, the builds
> should be natural Gradle.
>
> *- If you are going to refer to an identifier, it should have an explicit
> point of origin (not strings smashed together in a loop)*
> I assume your referring to how the examples java precommit is done. It is
> the only case of it to my knowledge and extra hands to migrate it to be
> like how the validates runner tests run would be useful. Filed
> https://issues.apache.org/jira/browse/BEAM-4033
>
>
> On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles  wrote:
>
>> Huge +1 to the idea of investing in simplification and polish of the
>> Gradle files before considering the migration complete.
>>
>> 1. build.gradle files should be as close to straight-line configuration
>> as possible:
>>  - You should be able to understand a module's build easily, locally,
>> without knowing Groovy
>>  - As little Groovy programming as possible, focused on providing
>> well-defined utilities
>>
>> 2. Explicit > implicit, especially for config files
>>  - Ditto about being able to understand a module's build locally
>>  - Minimize globally inherited config, in favor of explicitly calling
>> utilities
>>  - Passing current project to a utility is better than trying to make
>> something the "closure"
>>  - If you are going to refer to an identifier, it should have an explicit
>> point of origin (not strings smashed together in a loop)
>>
>> When in doubt, a little redundancy to improve readability is worth it in
>> this context.
>>
>> Kenn
>>
>> On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré 
>> wrote:
>>
>>> Hi all,
>>>
>>> I did multiple gradle build since last week and I would like to share
>>> one of my concern: it's about the communities.
>>>
>>> If I think our users won't see any change for them due to Gradle build
>>> (I think that most of our users will still use Maven with artifacts
>>> provided by Gradle), I'm more concerned by the dev community and the
>>> contribution.
>>>
>>> Maven is well known and straight forward for a large part of potential
>>> contributors. I think we have to keep in mind that we still have to grow
>>> up our contributors community.
>>>
>>> Today, maybe I'm wrong, but I have the feeling that gradle build is not
>>> straight forward (build.gradle includes build_rules.gradle, gathering
>>> all taks all together).
>>>
>>> I would like to add a task in the gradle "migration" process: simplify
>>> the gradle structure and files, and document this.
>>>
>>> I know we already have a Jira about the documentation part, but I would
>>> like to "polish" and use a clean structure for the Gradle resources. As
>>> already quickly discussed, I think that having one gradle file per tasks
>>> in the .gradle directory would be helpful.
>>>
>>> The goal is really to simplify the contribution.
>>>
>>> Do you agree if I add a Jira about "Gradle polish" ?
>>> Thoughts ?
>>>
>>> Regards
>>> JB
>>>
>>> On 07/04/2018 04:52, Scott Wegner wrote:
>>> > Here's an end-of-day update on migration work:
>>> >
>>> > * Snapshot unsigned dailies and signed release builds are working (!!).
>>> > PR/5048 [1] merges changes from Luke's branch
>>> >* python precommit failing... will investigate python precommit
>>> Monday
>>> > * All Precommits are gradle only
>>> > * All Postcommits except performance tests and Java_JDK_Versions_Test
>>> > use gradle (after PR/5047 [2] merged)
>>> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be
>>> > merged before switching
>>> > * ValidatesRunner_Spark failing consistently; investigating
>>> >
>>> > Thanks for another productive day of hacking. I'll pick up again on
>>> Monday.
>>> >
>>> > [1] https://github.com/apache/beam/pull/5048

Re: Gradle Status [April 6]

[Kenneth Knowles]

My phrasing sounded a bit too much like adding a blocking condition. I
absolutely don't want to do that. Our maven build was far from clear, and
had lots of tech debt and spooky action at a distance,

I just want to emphatically agree with JB's sentiment that we have an
opportunity to improve it before it ossifies in the current state. Maybe we
won't really have time, and that's OK too and we can just improve as we go.

Luke - I know we have different standards of clarity but I doubt we'll have
specific disagreements on cleanups to the build as they happen.* I was
mostly listing elementary programming concepts that apply to our build and
tend to get forgotten in configs & build files. Being "natural gradle" is
also important. Let's see how it goes.

Kenn

*FWIW I did know that we have some string-munged identifiers already, but
mostly I am speaking from experience beyond & before Beam too. I'll likely
tidy the examples build.gradle as I finish up nexmark in a similar way.

On Mon, Apr 9, 2018 at 10:50 AM Lukasz Cwik  wrote:

> JB, learning the build system in a project is hopefully avoided by most
> users if the contribution guide can clearly explain what users need to do.
> But for everyone else who wants to change a dependency version or add a
> dependency it should be as simple as copy/paste (which I believe it already
> is). Note that people who want to do anything more complicated like add new
> jenkins job configurations for new integration test targets needs to learn
> how the build system works, how maven profiles work, how to plumb the
> required set of attributes from Jenkins into the test target via the build
> system.
>
> Kenn, I have to disagree with a lot in 1 and 2. For users who are
> unfamiliar with Maven, we didn't setup the Maven build in such a way that
> anyone unfamiliar with Maven knew what was going on or try to use concepts
> unnatural to Maven in an attempt to make it seem easier. I believe we
> should stick with Gradle/Groovy conventions. Users who are not familiar
> with how Gradle/Groovy works will either ask the community or look at
> StackOverflow and doing things like passing the project around into
> functions is extremely uncommon compared to using the current context and
> applying closures to it. For users who are familiar with Gradle, the builds
> should be natural Gradle.
>
> *- If you are going to refer to an identifier, it should have an explicit
> point of origin (not strings smashed together in a loop)*
> I assume your referring to how the examples java precommit is done. It is
> the only case of it to my knowledge and extra hands to migrate it to be
> like how the validates runner tests run would be useful. Filed
> https://issues.apache.org/jira/browse/BEAM-4033
>
>
> On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles  wrote:
>
>> Huge +1 to the idea of investing in simplification and polish of the
>> Gradle files before considering the migration complete.
>>
>> 1. build.gradle files should be as close to straight-line configuration
>> as possible:
>>  - You should be able to understand a module's build easily, locally,
>> without knowing Groovy
>>  - As little Groovy programming as possible, focused on providing
>> well-defined utilities
>>
>> 2. Explicit > implicit, especially for config files
>>  - Ditto about being able to understand a module's build locally
>>  - Minimize globally inherited config, in favor of explicitly calling
>> utilities
>>  - Passing current project to a utility is better than trying to make
>> something the "closure"
>>  - If you are going to refer to an identifier, it should have an explicit
>> point of origin (not strings smashed together in a loop)
>>
>> When in doubt, a little redundancy to improve readability is worth it in
>> this context.
>>
>> Kenn
>>
>> On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré 
>> wrote:
>>
>>> Hi all,
>>>
>>> I did multiple gradle build since last week and I would like to share
>>> one of my concern: it's about the communities.
>>>
>>> If I think our users won't see any change for them due to Gradle build
>>> (I think that most of our users will still use Maven with artifacts
>>> provided by Gradle), I'm more concerned by the dev community and the
>>> contribution.
>>>
>>> Maven is well known and straight forward for a large part of potential
>>> contributors. I think we have to keep in mind that we still have to grow
>>> up our contributors community.
>>>
>>> Today, maybe I'm wrong, but I have the feeling that gradle build is not
>>> straight forward (build.gradle includes build_rules.gradle, gathering
>>> all taks all together).
>>>
>>> I would like to add a task in the gradle "migration" process: simplify
>>> the gradle structure and files, and document this.
>>>
>>> I know we already have a Jira about the documentation part, but I would
>>> like to "polish" and use a clean structure for the Gradle resources. As
>>> already quickly discussed, I think that having one gradle file per tasks
>>> in the .

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

+1 gradle not being mainstream - and even more in beam scope than other
java scope - it must stay simple. I know that personally it takes me x2 or
3 to say "ok ill fork" a project using gradle, in particular with custom
scripts and not only parent/module scripts.

If needed we can write custom plugins to simply it and hide the complexity
and weird task naming can move to "standard" names with options/parameters
maybe?

Le 9 avr. 2018 20:27, "Kenneth Knowles"  a écrit :

> My phrasing sounded a bit too much like adding a blocking condition. I
> absolutely don't want to do that. Our maven build was far from clear, and
> had lots of tech debt and spooky action at a distance,
>
> I just want to emphatically agree with JB's sentiment that we have an
> opportunity to improve it before it ossifies in the current state. Maybe we
> won't really have time, and that's OK too and we can just improve as we go.
>
> Luke - I know we have different standards of clarity but I doubt we'll
> have specific disagreements on cleanups to the build as they happen.* I was
> mostly listing elementary programming concepts that apply to our build and
> tend to get forgotten in configs & build files. Being "natural gradle" is
> also important. Let's see how it goes.
>
> Kenn
>
> *FWIW I did know that we have some string-munged identifiers already, but
> mostly I am speaking from experience beyond & before Beam too. I'll likely
> tidy the examples build.gradle as I finish up nexmark in a similar way.
>
> On Mon, Apr 9, 2018 at 10:50 AM Lukasz Cwik  wrote:
>
>> JB, learning the build system in a project is hopefully avoided by most
>> users if the contribution guide can clearly explain what users need to do.
>> But for everyone else who wants to change a dependency version or add a
>> dependency it should be as simple as copy/paste (which I believe it already
>> is). Note that people who want to do anything more complicated like add new
>> jenkins job configurations for new integration test targets needs to learn
>> how the build system works, how maven profiles work, how to plumb the
>> required set of attributes from Jenkins into the test target via the build
>> system.
>>
>> Kenn, I have to disagree with a lot in 1 and 2. For users who are
>> unfamiliar with Maven, we didn't setup the Maven build in such a way that
>> anyone unfamiliar with Maven knew what was going on or try to use concepts
>> unnatural to Maven in an attempt to make it seem easier. I believe we
>> should stick with Gradle/Groovy conventions. Users who are not familiar
>> with how Gradle/Groovy works will either ask the community or look at
>> StackOverflow and doing things like passing the project around into
>> functions is extremely uncommon compared to using the current context and
>> applying closures to it. For users who are familiar with Gradle, the builds
>> should be natural Gradle.
>>
>> *- If you are going to refer to an identifier, it should have an explicit
>> point of origin (not strings smashed together in a loop)*
>> I assume your referring to how the examples java precommit is done. It is
>> the only case of it to my knowledge and extra hands to migrate it to be
>> like how the validates runner tests run would be useful. Filed
>> https://issues.apache.org/jira/browse/BEAM-4033
>>
>>
>> On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles  wrote:
>>
>>> Huge +1 to the idea of investing in simplification and polish of the
>>> Gradle files before considering the migration complete.
>>>
>>> 1. build.gradle files should be as close to straight-line configuration
>>> as possible:
>>>  - You should be able to understand a module's build easily, locally,
>>> without knowing Groovy
>>>  - As little Groovy programming as possible, focused on providing
>>> well-defined utilities
>>>
>>> 2. Explicit > implicit, especially for config files
>>>  - Ditto about being able to understand a module's build locally
>>>  - Minimize globally inherited config, in favor of explicitly calling
>>> utilities
>>>  - Passing current project to a utility is better than trying to make
>>> something the "closure"
>>>  - If you are going to refer to an identifier, it should have an
>>> explicit point of origin (not strings smashed together in a loop)
>>>
>>> When in doubt, a little redundancy to improve readability is worth it in
>>> this context.
>>>
>>> Kenn
>>>
>>> On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré 
>>> wrote:
>>>
 Hi all,

 I did multiple gradle build since last week and I would like to share
 one of my concern: it's about the communities.

 If I think our users won't see any change for them due to Gradle build
 (I think that most of our users will still use Maven with artifacts
 provided by Gradle), I'm more concerned by the dev community and the
 contribution.

 Maven is well known and straight forward for a large part of potential
 contributors. I think we have to keep in mind that we still have to grow
 up our c

Re: Gradle Status [April 6]

[Lukasz Cwik]

Romain, can you clarify by the weird task naming? (Give some examples using
our current project and Gradle and what you would have expected.)

On Mon, Apr 9, 2018 at 2:49 PM Romain Manni-Bucau 
wrote:

> +1 gradle not being mainstream - and even more in beam scope than other
> java scope - it must stay simple. I know that personally it takes me x2 or
> 3 to say "ok ill fork" a project using gradle, in particular with custom
> scripts and not only parent/module scripts.
>
> If needed we can write custom plugins to simply it and hide the complexity
> and weird task naming can move to "standard" names with options/parameters
> maybe?
>
> Le 9 avr. 2018 20:27, "Kenneth Knowles"  a écrit :
>
>> My phrasing sounded a bit too much like adding a blocking condition. I
>> absolutely don't want to do that. Our maven build was far from clear, and
>> had lots of tech debt and spooky action at a distance,
>>
>> I just want to emphatically agree with JB's sentiment that we have an
>> opportunity to improve it before it ossifies in the current state. Maybe we
>> won't really have time, and that's OK too and we can just improve as we go.
>>
>> Luke - I know we have different standards of clarity but I doubt we'll
>> have specific disagreements on cleanups to the build as they happen.* I was
>> mostly listing elementary programming concepts that apply to our build and
>> tend to get forgotten in configs & build files. Being "natural gradle" is
>> also important. Let's see how it goes.
>>
>> Kenn
>>
>> *FWIW I did know that we have some string-munged identifiers already, but
>> mostly I am speaking from experience beyond & before Beam too. I'll likely
>> tidy the examples build.gradle as I finish up nexmark in a similar way.
>>
>> On Mon, Apr 9, 2018 at 10:50 AM Lukasz Cwik  wrote:
>>
>>> JB, learning the build system in a project is hopefully avoided by most
>>> users if the contribution guide can clearly explain what users need to do.
>>> But for everyone else who wants to change a dependency version or add a
>>> dependency it should be as simple as copy/paste (which I believe it already
>>> is). Note that people who want to do anything more complicated like add new
>>> jenkins job configurations for new integration test targets needs to learn
>>> how the build system works, how maven profiles work, how to plumb the
>>> required set of attributes from Jenkins into the test target via the build
>>> system.
>>>
>>> Kenn, I have to disagree with a lot in 1 and 2. For users who are
>>> unfamiliar with Maven, we didn't setup the Maven build in such a way that
>>> anyone unfamiliar with Maven knew what was going on or try to use concepts
>>> unnatural to Maven in an attempt to make it seem easier. I believe we
>>> should stick with Gradle/Groovy conventions. Users who are not familiar
>>> with how Gradle/Groovy works will either ask the community or look at
>>> StackOverflow and doing things like passing the project around into
>>> functions is extremely uncommon compared to using the current context and
>>> applying closures to it. For users who are familiar with Gradle, the builds
>>> should be natural Gradle.
>>>
>>> *- If you are going to refer to an identifier, it should have an
>>> explicit point of origin (not strings smashed together in a loop)*
>>> I assume your referring to how the examples java precommit is done. It
>>> is the only case of it to my knowledge and extra hands to migrate it to be
>>> like how the validates runner tests run would be useful. Filed
>>> https://issues.apache.org/jira/browse/BEAM-4033
>>>
>>>
>>> On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles  wrote:
>>>
 Huge +1 to the idea of investing in simplification and polish of the
 Gradle files before considering the migration complete.

 1. build.gradle files should be as close to straight-line configuration
 as possible:
  - You should be able to understand a module's build easily, locally,
 without knowing Groovy
  - As little Groovy programming as possible, focused on providing
 well-defined utilities

 2. Explicit > implicit, especially for config files
  - Ditto about being able to understand a module's build locally
  - Minimize globally inherited config, in favor of explicitly calling
 utilities
  - Passing current project to a utility is better than trying to make
 something the "closure"
  - If you are going to refer to an identifier, it should have an
 explicit point of origin (not strings smashed together in a loop)

 When in doubt, a little redundancy to improve readability is worth it
 in this context.

 Kenn

 On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré 
 wrote:

> Hi all,
>
> I did multiple gradle build since last week and I would like to share
> one of my concern: it's about the communities.
>
> If I think our users won't see any change for them due to Gradle build
> (I think that most of our

Re: Gradle Status [April 6]

[Jean-Baptiste Onofré]

Hi Luke,

let me take a concrete example.

I'm developing a Beam extension. Most of the time, I will ask myself:

1. How do I do the build.gradle and include my extension in Beam project
2. How do my extension is compile (the equivalent of 
maven-compiler-plugin) and how to deal with dependency and scope

3. How my extension is packaged
4. How my test are executed

Of course, we can document this when we are on the happy path.
Now, imagine, dependency is not correctly resolved, rat is failing, 
findbugs is complaining, etc. I have to check the "core" Beam gradle rules.
Today, I think it's not easy at all to find the corresponding tasks and 
configuration.


I saw several projects storing the tasks in .gradle (like Apache Geode 
for instance).


My point is that I think it's a right timing to polish/cleanup our 
build. Else, I'm afraid we will never do that in the future.


Regards
JB

On 09/04/2018 19:50, Lukasz Cwik wrote:
JB, learning the build system in a project is hopefully avoided by most 
users if the contribution guide can clearly explain what users need to 
do. But for everyone else who wants to change a dependency version or 
add a dependency it should be as simple as copy/paste (which I believe 
it already is). Note that people who want to do anything more 
complicated like add new jenkins job configurations for new integration 
test targets needs to learn how the build system works, how maven 
profiles work, how to plumb the required set of attributes from Jenkins 
into the test target via the build system.


Kenn, I have to disagree with a lot in 1 and 2. For users who are 
unfamiliar with Maven, we didn't setup the Maven build in such a way 
that anyone unfamiliar with Maven knew what was going on or try to use 
concepts unnatural to Maven in an attempt to make it seem easier. I 
believe we should stick with Gradle/Groovy conventions. Users who are 
not familiar with how Gradle/Groovy works will either ask the community 
or look at StackOverflow and doing things like passing the project 
around into functions is extremely uncommon compared to using the 
current context and applying closures to it. For users who are familiar 
with Gradle, the builds should be natural Gradle.


*- If you are going to refer to an identifier, it should have an 
explicit point of origin (not strings smashed together in a loop)*
I assume your referring to how the examples java precommit is done. It 
is the only case of it to my knowledge and extra hands to migrate it to 
be like how the validates runner tests run would be useful. Filed 
https://issues.apache.org/jira/browse/BEAM-4033



On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles > wrote:


Huge +1 to the idea of investing in simplification and polish of the
Gradle files before considering the migration complete.

1. build.gradle files should be as close to straight-line
configuration as possible:
  - You should be able to understand a module's build easily,
locally, without knowing Groovy
  - As little Groovy programming as possible, focused on providing
well-defined utilities

2. Explicit > implicit, especially for config files
  - Ditto about being able to understand a module's build locally
  - Minimize globally inherited config, in favor of explicitly
calling utilities
  - Passing current project to a utility is better than trying to
make something the "closure"
  - If you are going to refer to an identifier, it should have an
explicit point of origin (not strings smashed together in a loop)

When in doubt, a little redundancy to improve readability is worth
it in this context.

Kenn

On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré mailto:j...@nanthrax.net>> wrote:

Hi all,

I did multiple gradle build since last week and I would like to
share
one of my concern: it's about the communities.

If I think our users won't see any change for them due to Gradle
build
(I think that most of our users will still use Maven with artifacts
provided by Gradle), I'm more concerned by the dev community and the
contribution.

Maven is well known and straight forward for a large part of
potential
contributors. I think we have to keep in mind that we still have
to grow
up our contributors community.

Today, maybe I'm wrong, but I have the feeling that gradle build
is not
straight forward (build.gradle includes build_rules.gradle,
gathering
all taks all together).

I would like to add a task in the gradle "migration" process:
simplify
the gradle structure and files, and document this.

I know we already have a Jira about the documentation part, but
I would
like to "polish" and use a clean structure for the Gradle
resources. As
already quickly discussed, I th

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

Le 9 avr. 2018 21:04, "Lukasz Cwik"  a écrit :

Romain, can you clarify by the weird task naming? (Give some examples using
our current project and Gradle and what you would have expected.)


Sure

https://github.com/apache/beam/blob/fb6ba3bfd43605a0f4944828b5f19e75840fa7aa/examples/java/build.gradle#L78
for instance

It is way more natural to have one generic command you can select a runner
(including a csv value) than N commands for this kind of things.

Gradle allows to define custom tasks and "scopes". This must not be used at
Xtrem until justifiable. We now have a lot of configurations not making
much sense too until you scripted yourself the script or know beam well.
This doesnt enable contribs :(.

Hope it is clearer


On Mon, Apr 9, 2018 at 2:49 PM Romain Manni-Bucau 
wrote:

> +1 gradle not being mainstream - and even more in beam scope than other
> java scope - it must stay simple. I know that personally it takes me x2 or
> 3 to say "ok ill fork" a project using gradle, in particular with custom
> scripts and not only parent/module scripts.
>
> If needed we can write custom plugins to simply it and hide the complexity
> and weird task naming can move to "standard" names with options/parameters
> maybe?
>
> Le 9 avr. 2018 20:27, "Kenneth Knowles"  a écrit :
>
>> My phrasing sounded a bit too much like adding a blocking condition. I
>> absolutely don't want to do that. Our maven build was far from clear, and
>> had lots of tech debt and spooky action at a distance,
>>
>> I just want to emphatically agree with JB's sentiment that we have an
>> opportunity to improve it before it ossifies in the current state. Maybe we
>> won't really have time, and that's OK too and we can just improve as we go.
>>
>> Luke - I know we have different standards of clarity but I doubt we'll
>> have specific disagreements on cleanups to the build as they happen.* I was
>> mostly listing elementary programming concepts that apply to our build and
>> tend to get forgotten in configs & build files. Being "natural gradle" is
>> also important. Let's see how it goes.
>>
>> Kenn
>>
>> *FWIW I did know that we have some string-munged identifiers already, but
>> mostly I am speaking from experience beyond & before Beam too. I'll likely
>> tidy the examples build.gradle as I finish up nexmark in a similar way.
>>
>> On Mon, Apr 9, 2018 at 10:50 AM Lukasz Cwik  wrote:
>>
>>> JB, learning the build system in a project is hopefully avoided by most
>>> users if the contribution guide can clearly explain what users need to do.
>>> But for everyone else who wants to change a dependency version or add a
>>> dependency it should be as simple as copy/paste (which I believe it already
>>> is). Note that people who want to do anything more complicated like add new
>>> jenkins job configurations for new integration test targets needs to learn
>>> how the build system works, how maven profiles work, how to plumb the
>>> required set of attributes from Jenkins into the test target via the build
>>> system.
>>>
>>> Kenn, I have to disagree with a lot in 1 and 2. For users who are
>>> unfamiliar with Maven, we didn't setup the Maven build in such a way that
>>> anyone unfamiliar with Maven knew what was going on or try to use concepts
>>> unnatural to Maven in an attempt to make it seem easier. I believe we
>>> should stick with Gradle/Groovy conventions. Users who are not familiar
>>> with how Gradle/Groovy works will either ask the community or look at
>>> StackOverflow and doing things like passing the project around into
>>> functions is extremely uncommon compared to using the current context and
>>> applying closures to it. For users who are familiar with Gradle, the builds
>>> should be natural Gradle.
>>>
>>> *- If you are going to refer to an identifier, it should have an
>>> explicit point of origin (not strings smashed together in a loop)*
>>> I assume your referring to how the examples java precommit is done. It
>>> is the only case of it to my knowledge and extra hands to migrate it to be
>>> like how the validates runner tests run would be useful. Filed
>>> https://issues.apache.org/jira/browse/BEAM-4033
>>>
>>>
>>> On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles  wrote:
>>>
 Huge +1 to the idea of investing in simplification and polish of the
 Gradle files before considering the migration complete.

 1. build.gradle files should be as close to straight-line configuration
 as possible:
  - You should be able to understand a module's build easily, locally,
 without knowing Groovy
  - As little Groovy programming as possible, focused on providing
 well-defined utilities

 2. Explicit > implicit, especially for config files
  - Ditto about being able to understand a module's build locally
  - Minimize globally inherited config, in favor of explicitly calling
 utilities
  - Passing current project to a utility is better than trying to make
 something the "closure"
  - 

Re: Gradle Status [April 6]

[Lukasz Cwik]

Simplifying examples/java precommit is encompassed in BEAM-4033. Both you
and Kenn pointed to the same thing. To my knowledge it is the only place
where we have such a loop that generates tasks.

On Mon, Apr 9, 2018 at 3:37 PM Romain Manni-Bucau 
wrote:

>
>
> Le 9 avr. 2018 21:04, "Lukasz Cwik"  a écrit :
>
> Romain, can you clarify by the weird task naming? (Give some examples
> using our current project and Gradle and what you would have expected.)
>
>
> Sure
>
>
> https://github.com/apache/beam/blob/fb6ba3bfd43605a0f4944828b5f19e75840fa7aa/examples/java/build.gradle#L78
> for instance
>
> It is way more natural to have one generic command you can select a runner
> (including a csv value) than N commands for this kind of things.
>
> Gradle allows to define custom tasks and "scopes". This must not be used
> at Xtrem until justifiable. We now have a lot of configurations not making
> much sense too until you scripted yourself the script or know beam well.
> This doesnt enable contribs :(.
>
> Hope it is clearer
>
>
> On Mon, Apr 9, 2018 at 2:49 PM Romain Manni-Bucau 
> wrote:
>
>> +1 gradle not being mainstream - and even more in beam scope than other
>> java scope - it must stay simple. I know that personally it takes me x2 or
>> 3 to say "ok ill fork" a project using gradle, in particular with custom
>> scripts and not only parent/module scripts.
>>
>> If needed we can write custom plugins to simply it and hide the
>> complexity and weird task naming can move to "standard" names with
>> options/parameters maybe?
>>
>> Le 9 avr. 2018 20:27, "Kenneth Knowles"  a écrit :
>>
>>> My phrasing sounded a bit too much like adding a blocking condition. I
>>> absolutely don't want to do that. Our maven build was far from clear, and
>>> had lots of tech debt and spooky action at a distance,
>>>
>>> I just want to emphatically agree with JB's sentiment that we have an
>>> opportunity to improve it before it ossifies in the current state. Maybe we
>>> won't really have time, and that's OK too and we can just improve as we go.
>>>
>>> Luke - I know we have different standards of clarity but I doubt we'll
>>> have specific disagreements on cleanups to the build as they happen.* I was
>>> mostly listing elementary programming concepts that apply to our build and
>>> tend to get forgotten in configs & build files. Being "natural gradle" is
>>> also important. Let's see how it goes.
>>>
>>> Kenn
>>>
>>> *FWIW I did know that we have some string-munged identifiers already,
>>> but mostly I am speaking from experience beyond & before Beam too. I'll
>>> likely tidy the examples build.gradle as I finish up nexmark in a similar
>>> way.
>>>
>>> On Mon, Apr 9, 2018 at 10:50 AM Lukasz Cwik  wrote:
>>>
 JB, learning the build system in a project is hopefully avoided by most
 users if the contribution guide can clearly explain what users need to do.
 But for everyone else who wants to change a dependency version or add a
 dependency it should be as simple as copy/paste (which I believe it already
 is). Note that people who want to do anything more complicated like add new
 jenkins job configurations for new integration test targets needs to learn
 how the build system works, how maven profiles work, how to plumb the
 required set of attributes from Jenkins into the test target via the build
 system.

 Kenn, I have to disagree with a lot in 1 and 2. For users who are
 unfamiliar with Maven, we didn't setup the Maven build in such a way that
 anyone unfamiliar with Maven knew what was going on or try to use concepts
 unnatural to Maven in an attempt to make it seem easier. I believe we
 should stick with Gradle/Groovy conventions. Users who are not familiar
 with how Gradle/Groovy works will either ask the community or look at
 StackOverflow and doing things like passing the project around into
 functions is extremely uncommon compared to using the current context and
 applying closures to it. For users who are familiar with Gradle, the builds
 should be natural Gradle.

 *- If you are going to refer to an identifier, it should have an
 explicit point of origin (not strings smashed together in a loop)*
 I assume your referring to how the examples java precommit is done. It
 is the only case of it to my knowledge and extra hands to migrate it to be
 like how the validates runner tests run would be useful. Filed
 https://issues.apache.org/jira/browse/BEAM-4033


 On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles  wrote:

> Huge +1 to the idea of investing in simplification and polish of the
> Gradle files before considering the migration complete.
>
> 1. build.gradle files should be as close to straight-line
> configuration as possible:
>  - You should be able to understand a module's build easily, locally,
> without knowing Groovy
>  - As little Groovy programming as possible, focuse

Re: Gradle Status [April 6]

[Etienne Chauchot]

As a gradle beginner, I could not agree more ! 
+1
Etienne
Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :
> Hi all,
> 
> I did multiple gradle build since last week and I would like to share 
> one of my concern: it's about the communities.
> 
> If I think our users won't see any change for them due to Gradle build 
> (I think that most of our users will still use Maven with artifacts 
> provided by Gradle), I'm more concerned by the dev community and the 
> contribution.
> 
> Maven is well known and straight forward for a large part of potential 
> contributors. I think we have to keep in mind that we still have to grow 
> up our contributors community.
> 
> Today, maybe I'm wrong, but I have the feeling that gradle build is not 
> straight forward (build.gradle includes build_rules.gradle, gathering 
> all taks all together).
> 
> I would like to add a task in the gradle "migration" process: simplify 
> the gradle structure and files, and document this.
> 
> I know we already have a Jira about the documentation part, but I would 
> like to "polish" and use a clean structure for the Gradle resources. As 
> already quickly discussed, I think that having one gradle file per tasks 
> in the .gradle directory would be helpful.
> 
> The goal is really to simplify the contribution.
> 
> Do you agree if I add a Jira about "Gradle polish" ?
> Thoughts ?
> 
> Regards
> JB
> 
> On 07/04/2018 04:52, Scott Wegner wrote:
> > 
> > Here's an end-of-day update on migration work:
> > 
> > * Snapshot unsigned dailies and signed release builds are working (!!). 
> > PR/5048 [1] merges changes from Luke's branch
> >    * python precommit failing... will investigate python precommit Monday
> > * All Precommits are gradle only
> > * All Postcommits except performance tests and Java_JDK_Versions_Test  
> > use gradle (after PR/5047 [2] merged)
> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be 
> > merged before switching
> > * ValidatesRunner_Spark failing consistently; investigating
> > 
> > Thanks for another productive day of hacking. I'll pick up again on Monday.
> > 
> > [1] https://github.com/apache/beam/pull/5048
> > [2] https://github.com/apache/beam/pull/5047
> > 
> > 
> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau 
> > > wrote:
> > 
> > Why building a zip per runner which its stack and just pointing out
> > on that zip and let beam lazy load the runner:
> > 
> > --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or
> > the fromSystemProperties() if it gets merged a day ;))
> > 
> > Le 6 avr. 2018 20:21, "Kenneth Knowles"  > k...@google.com>> a écrit :
> > 
> > I'm working on finding a solution for launching the Nexmark
> > suite with each runner. This doesn't have to be done via Gradle,
> > but we anyhow need built artifacts that don't require user
> > classpath intervention.
> > 
> > It looks to me like the examples are also missing this - they
> > have separate configuration e.g. sparkRunnerPreCommit but that
> > is overspecified compared to a free-form launching of a main()
> > program with a runner profile.
> > 
> > On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  > lc...@google.com>> wrote:
> > 
> > Romain, are you talking about the profiles that exist as
> > part of the archetype examples?
> > 
> > If so, then those still exist and haven't been changed. If
> > not, can you provide a link to the profile in a pom file to
> > be clearer?
> > 
> > On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau
> > > wrote:
> > 
> > Hi Scott,
> > 
> > is it right that 2 doesn't handle the hierachy anymore
> > and that it doesn't handle profiles for runners as it is
> > currently with maven?
> > 
> > 
> > Romain Manni-Bucau
> > @rmannibucau  | Blog
> >  | Old Blog
> >  | Github
> >  | LinkedIn
> >  | Book
> > 
> > 
> > 
> > 2018-04-06 18:32 GMT+02:00 Scott Wegner
> > >:
> > 
> > I wanted to start a thread to summarize the current
> > state of Gradle migration. We've made lots of good
> > progress so far this week. Here's the status from
> > what I can tell-- please add or correct anything I
> > missed:
> > 
> > * Release artifacts can be built and published for
> > Snapshot and officlal releases

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

What's the plan to make idea supporting gradle on beam project? Do we
import the workaround mentionned in
https://youtrack.jetbrains.com/issue/IDEA-175172?
For the ones who didn't see this issue in action: idea will compile in
out/ instead of build/ and you will just miss all the resources you
need like some SPI registration which are used by all our registrar =>
no way to run tests in idea without hacking the configuration quite
deeply :(

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-10 10:08 GMT+02:00 Etienne Chauchot :
> As a gradle beginner, I could not agree more !
> +1
> Etienne
> Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :
>
> Hi all,
>
> I did multiple gradle build since last week and I would like to share
> one of my concern: it's about the communities.
>
> If I think our users won't see any change for them due to Gradle build
> (I think that most of our users will still use Maven with artifacts
> provided by Gradle), I'm more concerned by the dev community and the
> contribution.
>
> Maven is well known and straight forward for a large part of potential
> contributors. I think we have to keep in mind that we still have to grow
> up our contributors community.
>
> Today, maybe I'm wrong, but I have the feeling that gradle build is not
> straight forward (build.gradle includes build_rules.gradle, gathering
> all taks all together).
>
> I would like to add a task in the gradle "migration" process: simplify
> the gradle structure and files, and document this.
>
> I know we already have a Jira about the documentation part, but I would
> like to "polish" and use a clean structure for the Gradle resources. As
> already quickly discussed, I think that having one gradle file per tasks
> in the .gradle directory would be helpful.
>
> The goal is really to simplify the contribution.
>
> Do you agree if I add a Jira about "Gradle polish" ?
> Thoughts ?
>
> Regards
> JB
>
> On 07/04/2018 04:52, Scott Wegner wrote:
>
> Here's an end-of-day update on migration work:
>
> * Snapshot unsigned dailies and signed release builds are working (!!).
> PR/5048 [1] merges changes from Luke's branch
>* python precommit failing... will investigate python precommit Monday
> * All Precommits are gradle only
> * All Postcommits except performance tests and Java_JDK_Versions_Test
> use gradle (after PR/5047 [2] merged)
> * Nightly snapshot release using gradle is ready; needs PR/5048 to be
> merged before switching
> * ValidatesRunner_Spark failing consistently; investigating
>
> Thanks for another productive day of hacking. I'll pick up again on Monday.
>
> [1] https://github.com/apache/beam/pull/5048
> [2] https://github.com/apache/beam/pull/5047
>
>
> On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
> mailto:rmannibu...@gmail.com>> wrote:
>
> Why building a zip per runner which its stack and just pointing out
> on that zip and let beam lazy load the runner:
>
> --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or
> the fromSystemProperties() if it gets merged a day ;))
>
> Le 6 avr. 2018 20:21, "Kenneth Knowles"  > a écrit :
>
> I'm working on finding a solution for launching the Nexmark
> suite with each runner. This doesn't have to be done via Gradle,
> but we anyhow need built artifacts that don't require user
> classpath intervention.
>
> It looks to me like the examples are also missing this - they
> have separate configuration e.g. sparkRunnerPreCommit but that
> is overspecified compared to a free-form launching of a main()
> program with a runner profile.
>
> On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  > wrote:
>
> Romain, are you talking about the profiles that exist as
> part of the archetype examples?
>
> If so, then those still exist and haven't been changed. If
> not, can you provide a link to the profile in a pom file to
> be clearer?
>
> On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau
> mailto:rmannibu...@gmail.com>> wrote:
>
> Hi Scott,
>
> is it right that 2 doesn't handle the hierachy anymore
> and that it doesn't handle profiles for runners as it is
> currently with maven?
>
>
> Romain Manni-Bucau
> @rmannibucau  | Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
>
> 
>
> 2018-04-06 18:32 GMT+02:00 Scott Wegner
> mailto:sweg...@google.com>>:

Re: Gradle Status [April 6]

[Lukasz Cwik]

I have found that the simplest setup is to delegate the build/test actions
to Gradle. This allows you to run unit tests very easily and since its in
the same manner that Gradle would have, you know that if its passing it
will pass on the command line and on Jenkins. Here is one site that
discusses how to set this up:
http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html


On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau 
wrote:

> What's the plan to make idea supporting gradle on beam project? Do we
> import the workaround mentionned in
> https://youtrack.jetbrains.com/issue/IDEA-175172?
> For the ones who didn't see this issue in action: idea will compile in
> out/ instead of build/ and you will just miss all the resources you
> need like some SPI registration which are used by all our registrar =>
> no way to run tests in idea without hacking the configuration quite
> deeply :(
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
>
> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot :
> > As a gradle beginner, I could not agree more !
> > +1
> > Etienne
> > Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :
> >
> > Hi all,
> >
> > I did multiple gradle build since last week and I would like to share
> > one of my concern: it's about the communities.
> >
> > If I think our users won't see any change for them due to Gradle build
> > (I think that most of our users will still use Maven with artifacts
> > provided by Gradle), I'm more concerned by the dev community and the
> > contribution.
> >
> > Maven is well known and straight forward for a large part of potential
> > contributors. I think we have to keep in mind that we still have to grow
> > up our contributors community.
> >
> > Today, maybe I'm wrong, but I have the feeling that gradle build is not
> > straight forward (build.gradle includes build_rules.gradle, gathering
> > all taks all together).
> >
> > I would like to add a task in the gradle "migration" process: simplify
> > the gradle structure and files, and document this.
> >
> > I know we already have a Jira about the documentation part, but I would
> > like to "polish" and use a clean structure for the Gradle resources. As
> > already quickly discussed, I think that having one gradle file per tasks
> > in the .gradle directory would be helpful.
> >
> > The goal is really to simplify the contribution.
> >
> > Do you agree if I add a Jira about "Gradle polish" ?
> > Thoughts ?
> >
> > Regards
> > JB
> >
> > On 07/04/2018 04:52, Scott Wegner wrote:
> >
> > Here's an end-of-day update on migration work:
> >
> > * Snapshot unsigned dailies and signed release builds are working (!!).
> > PR/5048 [1] merges changes from Luke's branch
> >* python precommit failing... will investigate python precommit Monday
> > * All Precommits are gradle only
> > * All Postcommits except performance tests and Java_JDK_Versions_Test
> > use gradle (after PR/5047 [2] merged)
> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be
> > merged before switching
> > * ValidatesRunner_Spark failing consistently; investigating
> >
> > Thanks for another productive day of hacking. I'll pick up again on
> Monday.
> >
> > [1] https://github.com/apache/beam/pull/5048
> > [2] https://github.com/apache/beam/pull/5047
> >
> >
> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
> > mailto:rmannibu...@gmail.com>> wrote:
> >
> > Why building a zip per runner which its stack and just pointing out
> > on that zip and let beam lazy load the runner:
> >
> > --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or
> > the fromSystemProperties() if it gets merged a day ;))
> >
> > Le 6 avr. 2018 20:21, "Kenneth Knowles"  > > a écrit :
> >
> > I'm working on finding a solution for launching the Nexmark
> > suite with each runner. This doesn't have to be done via Gradle,
> > but we anyhow need built artifacts that don't require user
> > classpath intervention.
> >
> > It looks to me like the examples are also missing this - they
> > have separate configuration e.g. sparkRunnerPreCommit but that
> > is overspecified compared to a free-form launching of a main()
> > program with a runner profile.
> >
> > On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  > > wrote:
> >
> > Romain, are you talking about the profiles that exist as
> > part of the archetype examples?
> >
> > If so, then those still exist and haven't been changed. If
> > not, can you provide a link to the profile in a pom file to
> > be clearer?
> >
> > On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau
> > mailto:rmannibu...@gmail.com>>
> wrote:
> >
> > Hi Scott,
> >
> > is it right that 2 doesn't handle the hierac

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

You dont have issue due to the build setup with that option. I get:

avr. 10, 2018 3:20:10 PM
org.apache.beam.runners.direct.DirectTransformExecutor run
GRAVE: Error occurred within
org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
com.google.common.util.concurrent.ExecutionError:
java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy

?

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-10 15:13 GMT+02:00 Lukasz Cwik :
> I have found that the simplest setup is to delegate the build/test actions
> to Gradle. This allows you to run unit tests very easily and since its in
> the same manner that Gradle would have, you know that if its passing it will
> pass on the command line and on Jenkins. Here is one site that discusses how
> to set this up:
> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
>
>
> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau 
> wrote:
>>
>> What's the plan to make idea supporting gradle on beam project? Do we
>> import the workaround mentionned in
>> https://youtrack.jetbrains.com/issue/IDEA-175172?
>> For the ones who didn't see this issue in action: idea will compile in
>> out/ instead of build/ and you will just miss all the resources you
>> need like some SPI registration which are used by all our registrar =>
>> no way to run tests in idea without hacking the configuration quite
>> deeply :(
>>
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>
>>
>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot :
>> > As a gradle beginner, I could not agree more !
>> > +1
>> > Etienne
>> > Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :
>> >
>> > Hi all,
>> >
>> > I did multiple gradle build since last week and I would like to share
>> > one of my concern: it's about the communities.
>> >
>> > If I think our users won't see any change for them due to Gradle build
>> > (I think that most of our users will still use Maven with artifacts
>> > provided by Gradle), I'm more concerned by the dev community and the
>> > contribution.
>> >
>> > Maven is well known and straight forward for a large part of potential
>> > contributors. I think we have to keep in mind that we still have to grow
>> > up our contributors community.
>> >
>> > Today, maybe I'm wrong, but I have the feeling that gradle build is not
>> > straight forward (build.gradle includes build_rules.gradle, gathering
>> > all taks all together).
>> >
>> > I would like to add a task in the gradle "migration" process: simplify
>> > the gradle structure and files, and document this.
>> >
>> > I know we already have a Jira about the documentation part, but I would
>> > like to "polish" and use a clean structure for the Gradle resources. As
>> > already quickly discussed, I think that having one gradle file per tasks
>> > in the .gradle directory would be helpful.
>> >
>> > The goal is really to simplify the contribution.
>> >
>> > Do you agree if I add a Jira about "Gradle polish" ?
>> > Thoughts ?
>> >
>> > Regards
>> > JB
>> >
>> > On 07/04/2018 04:52, Scott Wegner wrote:
>> >
>> > Here's an end-of-day update on migration work:
>> >
>> > * Snapshot unsigned dailies and signed release builds are working (!!).
>> > PR/5048 [1] merges changes from Luke's branch
>> >* python precommit failing... will investigate python precommit
>> > Monday
>> > * All Precommits are gradle only
>> > * All Postcommits except performance tests and Java_JDK_Versions_Test
>> > use gradle (after PR/5047 [2] merged)
>> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be
>> > merged before switching
>> > * ValidatesRunner_Spark failing consistently; investigating
>> >
>> > Thanks for another productive day of hacking. I'll pick up again on
>> > Monday.
>> >
>> > [1] https://github.com/apache/beam/pull/5048
>> > [2] https://github.com/apache/beam/pull/5047
>> >
>> >
>> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
>> > mailto:rmannibu...@gmail.com>> wrote:
>> >
>> > Why building a zip per runner which its stack and just pointing out
>> > on that zip and let beam lazy load the runner:
>> >
>> > --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or
>> > the fromSystemProperties() if it gets merged a day ;))
>> >
>> > Le 6 avr. 2018 20:21, "Kenneth Knowles" > > > a écrit :
>> >
>> > I'm working on finding a solution for launching the Nexmark
>> > suite with each runner. This doesn't have to be done via Gradle,
>> > but we anyhow need built artifacts that don't require user
>> > classpath intervention.
>> >
>> > It looks to me like the examples are also missing this - they
>> > have separate configuration e.g. sparkRunnerPreCommit but that
>> > is overspecified compared to a free-form launching of a main()
>> > program with a runner profile.
>> >
>> > On Fri, Apr 6, 2018 

Re: Gradle Status [April 6]

[Lukasz Cwik]

Romain, I haven't seen that error. At the very top of your test execution
log it gives you the tasks that it is running, for example:
6:41:33 AM: Executing tasks ':beam-sdks-java-core:cleanTest
:beam-sdks-java-core:test --tests
"org.apache.beam.sdk.coders.AvroCoderTest.testAvroCoderEncoding"'...

What task did it fail for you for?

On Tue, Apr 10, 2018 at 9:21 AM Romain Manni-Bucau 
wrote:

> You dont have issue due to the build setup with that option. I get:
>
> avr. 10, 2018 3:20:10 PM
> org.apache.beam.runners.direct.DirectTransformExecutor run
> GRAVE: Error occurred within
> org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
> com.google.common.util.concurrent.ExecutionError:
> java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
>
> ?
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
>
> 2018-04-10 15:13 GMT+02:00 Lukasz Cwik :
> > I have found that the simplest setup is to delegate the build/test
> actions
> > to Gradle. This allows you to run unit tests very easily and since its in
> > the same manner that Gradle would have, you know that if its passing it
> will
> > pass on the command line and on Jenkins. Here is one site that discusses
> how
> > to set this up:
> >
> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
> >
> >
> > On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >>
> >> What's the plan to make idea supporting gradle on beam project? Do we
> >> import the workaround mentionned in
> >> https://youtrack.jetbrains.com/issue/IDEA-175172?
> >> For the ones who didn't see this issue in action: idea will compile in
> >> out/ instead of build/ and you will just miss all the resources you
> >> need like some SPI registration which are used by all our registrar =>
> >> no way to run tests in idea without hacking the configuration quite
> >> deeply :(
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >>
> >>
> >> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot :
> >> > As a gradle beginner, I could not agree more !
> >> > +1
> >> > Etienne
> >> > Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :
> >> >
> >> > Hi all,
> >> >
> >> > I did multiple gradle build since last week and I would like to share
> >> > one of my concern: it's about the communities.
> >> >
> >> > If I think our users won't see any change for them due to Gradle build
> >> > (I think that most of our users will still use Maven with artifacts
> >> > provided by Gradle), I'm more concerned by the dev community and the
> >> > contribution.
> >> >
> >> > Maven is well known and straight forward for a large part of potential
> >> > contributors. I think we have to keep in mind that we still have to
> grow
> >> > up our contributors community.
> >> >
> >> > Today, maybe I'm wrong, but I have the feeling that gradle build is
> not
> >> > straight forward (build.gradle includes build_rules.gradle, gathering
> >> > all taks all together).
> >> >
> >> > I would like to add a task in the gradle "migration" process: simplify
> >> > the gradle structure and files, and document this.
> >> >
> >> > I know we already have a Jira about the documentation part, but I
> would
> >> > like to "polish" and use a clean structure for the Gradle resources.
> As
> >> > already quickly discussed, I think that having one gradle file per
> tasks
> >> > in the .gradle directory would be helpful.
> >> >
> >> > The goal is really to simplify the contribution.
> >> >
> >> > Do you agree if I add a Jira about "Gradle polish" ?
> >> > Thoughts ?
> >> >
> >> > Regards
> >> > JB
> >> >
> >> > On 07/04/2018 04:52, Scott Wegner wrote:
> >> >
> >> > Here's an end-of-day update on migration work:
> >> >
> >> > * Snapshot unsigned dailies and signed release builds are working
> (!!).
> >> > PR/5048 [1] merges changes from Luke's branch
> >> >* python precommit failing... will investigate python precommit
> >> > Monday
> >> > * All Precommits are gradle only
> >> > * All Postcommits except performance tests and Java_JDK_Versions_Test
> >> > use gradle (after PR/5047 [2] merged)
> >> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be
> >> > merged before switching
> >> > * ValidatesRunner_Spark failing consistently; investigating
> >> >
> >> > Thanks for another productive day of hacking. I'll pick up again on
> >> > Monday.
> >> >
> >> > [1] https://github.com/apache/beam/pull/5048
> >> > [2] https://github.com/apache/beam/pull/5047
> >> >
> >> >
> >> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
> >> > mailto:rmannibu...@gmail.com>> wrote:
> >> >
> >> > Why building a zip per runner which its stack and just pointing
> out
> >> > on that zip and let beam lazy load the runner:
> >> >
> >> > --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=...
> (or
> >> > the fromSystemProperties() if it gets merged a day ;))
> >> >
> >> > 

Re: Gradle Status [April 6]

[Jean-Baptiste Onofré]

That's a very important issue for contribution. Up to now, I used Maven 
for setup IntelliJ (and it works just fine). If we remove the pom.xml, 
we have to support Eclipse and IntelliJ "smoothly".


Let me try in IntelliJ.

Regards
JB

On 10/04/2018 15:21, Romain Manni-Bucau wrote:

You dont have issue due to the build setup with that option. I get:

avr. 10, 2018 3:20:10 PM
org.apache.beam.runners.direct.DirectTransformExecutor run
GRAVE: Error occurred within
org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
com.google.common.util.concurrent.ExecutionError:
java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy

?

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-10 15:13 GMT+02:00 Lukasz Cwik :

I have found that the simplest setup is to delegate the build/test actions
to Gradle. This allows you to run unit tests very easily and since its in
the same manner that Gradle would have, you know that if its passing it will
pass on the command line and on Jenkins. Here is one site that discusses how
to set this up:
http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html


On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau 
wrote:


What's the plan to make idea supporting gradle on beam project? Do we
import the workaround mentionned in
https://youtrack.jetbrains.com/issue/IDEA-175172?
For the ones who didn't see this issue in action: idea will compile in
out/ instead of build/ and you will just miss all the resources you
need like some SPI registration which are used by all our registrar =>
no way to run tests in idea without hacking the configuration quite
deeply :(

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-10 10:08 GMT+02:00 Etienne Chauchot :

As a gradle beginner, I could not agree more !
+1
Etienne
Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :

Hi all,

I did multiple gradle build since last week and I would like to share
one of my concern: it's about the communities.

If I think our users won't see any change for them due to Gradle build
(I think that most of our users will still use Maven with artifacts
provided by Gradle), I'm more concerned by the dev community and the
contribution.

Maven is well known and straight forward for a large part of potential
contributors. I think we have to keep in mind that we still have to grow
up our contributors community.

Today, maybe I'm wrong, but I have the feeling that gradle build is not
straight forward (build.gradle includes build_rules.gradle, gathering
all taks all together).

I would like to add a task in the gradle "migration" process: simplify
the gradle structure and files, and document this.

I know we already have a Jira about the documentation part, but I would
like to "polish" and use a clean structure for the Gradle resources. As
already quickly discussed, I think that having one gradle file per tasks
in the .gradle directory would be helpful.

The goal is really to simplify the contribution.

Do you agree if I add a Jira about "Gradle polish" ?
Thoughts ?

Regards
JB

On 07/04/2018 04:52, Scott Wegner wrote:

Here's an end-of-day update on migration work:

* Snapshot unsigned dailies and signed release builds are working (!!).
PR/5048 [1] merges changes from Luke's branch
* python precommit failing... will investigate python precommit
Monday
* All Precommits are gradle only
* All Postcommits except performance tests and Java_JDK_Versions_Test
use gradle (after PR/5047 [2] merged)
* Nightly snapshot release using gradle is ready; needs PR/5048 to be
merged before switching
* ValidatesRunner_Spark failing consistently; investigating

Thanks for another productive day of hacking. I'll pick up again on
Monday.

[1] https://github.com/apache/beam/pull/5048
[2] https://github.com/apache/beam/pull/5047


On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
mailto:rmannibu...@gmail.com>> wrote:

 Why building a zip per runner which its stack and just pointing out
 on that zip and let beam lazy load the runner:

 --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or
 the fromSystemProperties() if it gets merged a day ;))

 Le 6 avr. 2018 20:21, "Kenneth Knowles" mailto:k...@google.com>> a écrit :

 I'm working on finding a solution for launching the Nexmark
 suite with each runner. This doesn't have to be done via Gradle,
 but we anyhow need built artifacts that don't require user
 classpath intervention.

 It looks to me like the examples are also missing this - they
 have separate configuration e.g. sparkRunnerPreCommit but that
 is overspecified compared to a free-form launching of a main()
 program with a runner profile.

 On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik mailto:lc...@google.com>> wrote:

 Romain, are you talking about the profiles that exist as
 part of the arche

Re: Gradle Status [April 6]

[Lukasz Cwik]

beam-site PR/414 updates the instructions for using Intellij and how to
import a module:

1. Create an empty IntelliJ project outside of the Beam source tree.
2. Under Project Structure > Project, select a Project SDK.
3. Under Project Structure > Modules, click the + sign to add a module and
   select "Import Module".
1. Select the directory containing the Beam source tree.
2. Tick the "Import module from external model" button and select Gradle
   from the list.
3. Tick the following boxes.
   * Use auto-import
   * Create separate module per source set
   * Store generated project files externally
   * Use default gradle wrapper
4. Delegate build actions to Gradle by going to Settings > Build, Execution,
   Deployment > Build Tools > Gradle and checking "Delegate IDE build/run
   actions to gradle".

On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré 
wrote:

> That's a very important issue for contribution. Up to now, I used Maven
> for setup IntelliJ (and it works just fine). If we remove the pom.xml,
> we have to support Eclipse and IntelliJ "smoothly".
>
> Let me try in IntelliJ.
>
> Regards
> JB
>
> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
> > You dont have issue due to the build setup with that option. I get:
> >
> > avr. 10, 2018 3:20:10 PM
> > org.apache.beam.runners.direct.DirectTransformExecutor run
> > GRAVE: Error occurred within
> > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
> > com.google.common.util.concurrent.ExecutionError:
> > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
> >
> > ?
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> >
> > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik :
> >> I have found that the simplest setup is to delegate the build/test
> actions
> >> to Gradle. This allows you to run unit tests very easily and since its
> in
> >> the same manner that Gradle would have, you know that if its passing it
> will
> >> pass on the command line and on Jenkins. Here is one site that
> discusses how
> >> to set this up:
> >>
> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
> >>
> >>
> >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau <
> rmannibu...@gmail.com>
> >> wrote:
> >>>
> >>> What's the plan to make idea supporting gradle on beam project? Do we
> >>> import the workaround mentionned in
> >>> https://youtrack.jetbrains.com/issue/IDEA-175172?
> >>> For the ones who didn't see this issue in action: idea will compile in
> >>> out/ instead of build/ and you will just miss all the resources you
> >>> need like some SPI registration which are used by all our registrar =>
> >>> no way to run tests in idea without hacking the configuration quite
> >>> deeply :(
> >>>
> >>> Romain Manni-Bucau
> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >>>
> >>>
> >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot :
>  As a gradle beginner, I could not agree more !
>  +1
>  Etienne
>  Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :
> 
>  Hi all,
> 
>  I did multiple gradle build since last week and I would like to share
>  one of my concern: it's about the communities.
> 
>  If I think our users won't see any change for them due to Gradle build
>  (I think that most of our users will still use Maven with artifacts
>  provided by Gradle), I'm more concerned by the dev community and the
>  contribution.
> 
>  Maven is well known and straight forward for a large part of potential
>  contributors. I think we have to keep in mind that we still have to
> grow
>  up our contributors community.
> 
>  Today, maybe I'm wrong, but I have the feeling that gradle build is
> not
>  straight forward (build.gradle includes build_rules.gradle, gathering
>  all taks all together).
> 
>  I would like to add a task in the gradle "migration" process: simplify
>  the gradle structure and files, and document this.
> 
>  I know we already have a Jira about the documentation part, but I
> would
>  like to "polish" and use a clean structure for the Gradle resources.
> As
>  already quickly discussed, I think that having one gradle file per
> tasks
>  in the .gradle directory would be helpful.
> 
>  The goal is really to simplify the contribution.
> 
>  Do you agree if I add a Jira about "Gradle polish" ?
>  Thoughts ?
> 
>  Regards
>  JB
> 
>  On 07/04/2018 04:52, Scott Wegner wrote:
> 
>  Here's an end-of-day update on migration work:
> 
>  * Snapshot unsigned dailies and signed release builds are working
> (!!).
>  PR/5048 [1] merges changes from Luke's branch
>  * python precommit failing... will investigate python precommit
>  Monday
>  * All Precommits are gradle only
>  * All Postcommits except performance tests and Java_JDK_Vers

Re: Gradle Status [April 6]

[Jean-Baptiste Onofré]

It's what I did, I'm trying a complete reload now (maybe this step failed).

On 10/04/2018 16:38, Lukasz Cwik wrote:
beam-site PR/414 updates the instructions for using Intellij and how to 
import a module:


1. Create an empty IntelliJ project outside of the Beam source tree.
2. Under Project Structure > Project, select a Project SDK.
3. Under Project Structure > Modules, click the + sign to add a module and
    select "Import Module".
     1. Select the directory containing the Beam source tree.
     2. Tick the "Import module from external model" button and select 
Gradle

        from the list.
     3. Tick the following boxes.
        * Use auto-import
        * Create separate module per source set
        * Store generated project files externally
        * Use default gradle wrapper
4. Delegate build actions to Gradle by going to Settings > Build, Execution,
    Deployment > Build Tools > Gradle and checking "Delegate IDE build/run
    actions to gradle".

On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré > wrote:


That's a very important issue for contribution. Up to now, I used Maven
for setup IntelliJ (and it works just fine). If we remove the pom.xml,
we have to support Eclipse and IntelliJ "smoothly".

Let me try in IntelliJ.

Regards
JB

On 10/04/2018 15:21, Romain Manni-Bucau wrote:
 > You dont have issue due to the build setup with that option. I get:
 >
 > avr. 10, 2018 3:20:10 PM
 > org.apache.beam.runners.direct.DirectTransformExecutor run
 > GRAVE: Error occurred within
 > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
 > com.google.common.util.concurrent.ExecutionError:
 > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
 >
 > ?
 >
 > Romain Manni-Bucau
 > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
 >
 >
 > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik mailto:lc...@google.com>>:
 >> I have found that the simplest setup is to delegate the
build/test actions
 >> to Gradle. This allows you to run unit tests very easily and
since its in
 >> the same manner that Gradle would have, you know that if its
passing it will
 >> pass on the command line and on Jenkins. Here is one site that
discusses how
 >> to set this up:
 >>

http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
 >>
 >>
 >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau
mailto:rmannibu...@gmail.com>>
 >> wrote:
 >>>
 >>> What's the plan to make idea supporting gradle on beam project?
Do we
 >>> import the workaround mentionned in
 >>> https://youtrack.jetbrains.com/issue/IDEA-175172?
 >>> For the ones who didn't see this issue in action: idea will
compile in
 >>> out/ instead of build/ and you will just miss all the resources you
 >>> need like some SPI registration which are used by all our
registrar =>
 >>> no way to run tests in idea without hacking the configuration quite
 >>> deeply :(
 >>>
 >>> Romain Manni-Bucau
 >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
 >>>
 >>>
 >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot
mailto:echauc...@apache.org>>:
  As a gradle beginner, I could not agree more !
  +1
  Etienne
  Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a
écrit :
 
  Hi all,
 
  I did multiple gradle build since last week and I would like
to share
  one of my concern: it's about the communities.
 
  If I think our users won't see any change for them due to
Gradle build
  (I think that most of our users will still use Maven with
artifacts
  provided by Gradle), I'm more concerned by the dev community
and the
  contribution.
 
  Maven is well known and straight forward for a large part of
potential
  contributors. I think we have to keep in mind that we still
have to grow
  up our contributors community.
 
  Today, maybe I'm wrong, but I have the feeling that gradle
build is not
  straight forward (build.gradle includes build_rules.gradle,
gathering
  all taks all together).
 
  I would like to add a task in the gradle "migration" process:
simplify
  the gradle structure and files, and document this.
 
  I know we already have a Jira about the documentation part,
but I would
  like to "polish" and use a clean structure for the Gradle
resources. As
  already quickly discussed, I think that having one gradle file
per tasks
  in the .gradle directory would be helpful.
 
  The goal is really to simplify the contribution.
 
  Do you agree if I add a Jira

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

side note: do NOT use auto-import until you are sure you can, it locks
regularly on beam (pby too big for idea?) and makes idea ready to be
killed :(

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré :
> It's what I did, I'm trying a complete reload now (maybe this step failed).
>
> On 10/04/2018 16:38, Lukasz Cwik wrote:
>>
>> beam-site PR/414 updates the instructions for using Intellij and how to
>> import a module:
>>
>> 1. Create an empty IntelliJ project outside of the Beam source tree.
>> 2. Under Project Structure > Project, select a Project SDK.
>> 3. Under Project Structure > Modules, click the + sign to add a module and
>> select "Import Module".
>>  1. Select the directory containing the Beam source tree.
>>  2. Tick the "Import module from external model" button and select
>> Gradle
>> from the list.
>>  3. Tick the following boxes.
>> * Use auto-import
>> * Create separate module per source set
>> * Store generated project files externally
>> * Use default gradle wrapper
>> 4. Delegate build actions to Gradle by going to Settings > Build,
>> Execution,
>> Deployment > Build Tools > Gradle and checking "Delegate IDE build/run
>> actions to gradle".
>>
>> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré > > wrote:
>>
>> That's a very important issue for contribution. Up to now, I used
>> Maven
>> for setup IntelliJ (and it works just fine). If we remove the pom.xml,
>> we have to support Eclipse and IntelliJ "smoothly".
>>
>> Let me try in IntelliJ.
>>
>> Regards
>> JB
>>
>> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
>>  > You dont have issue due to the build setup with that option. I get:
>>  >
>>  > avr. 10, 2018 3:20:10 PM
>>  > org.apache.beam.runners.direct.DirectTransformExecutor run
>>  > GRAVE: Error occurred within
>>  > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
>>  > com.google.common.util.concurrent.ExecutionError:
>>  > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
>>  >
>>  > ?
>>  >
>>  > Romain Manni-Bucau
>>  > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>  >
>>  >
>>  > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik > >:
>>  >> I have found that the simplest setup is to delegate the
>> build/test actions
>>  >> to Gradle. This allows you to run unit tests very easily and
>> since its in
>>  >> the same manner that Gradle would have, you know that if its
>> passing it will
>>  >> pass on the command line and on Jenkins. Here is one site that
>> discusses how
>>  >> to set this up:
>>  >>
>>
>> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
>>  >>
>>  >>
>>  >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau
>> mailto:rmannibu...@gmail.com>>
>>  >> wrote:
>>  >>>
>>  >>> What's the plan to make idea supporting gradle on beam project?
>> Do we
>>  >>> import the workaround mentionned in
>>  >>> https://youtrack.jetbrains.com/issue/IDEA-175172?
>>  >>> For the ones who didn't see this issue in action: idea will
>> compile in
>>  >>> out/ instead of build/ and you will just miss all the resources
>> you
>>  >>> need like some SPI registration which are used by all our
>> registrar =>
>>  >>> no way to run tests in idea without hacking the configuration
>> quite
>>  >>> deeply :(
>>  >>>
>>  >>> Romain Manni-Bucau
>>  >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>  >>>
>>  >>>
>>  >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot
>> mailto:echauc...@apache.org>>:
>>
>>   As a gradle beginner, I could not agree more !
>>   +1
>>   Etienne
>>   Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a
>> écrit :
>>  
>>   Hi all,
>>  
>>   I did multiple gradle build since last week and I would like
>> to share
>>   one of my concern: it's about the communities.
>>  
>>   If I think our users won't see any change for them due to
>> Gradle build
>>   (I think that most of our users will still use Maven with
>> artifacts
>>   provided by Gradle), I'm more concerned by the dev community
>> and the
>>   contribution.
>>  
>>   Maven is well known and straight forward for a large part of
>> potential
>>   contributors. I think we have to keep in mind that we still
>> have to grow
>>   up our contributors community.
>>  
>>   Today, maybe I'm wrong, but I have the feeling that gradle
>> build is not
>>   straight forward (build.gradle includes build_rules.gradle,
>> g

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

Ok, didn't find a way to make it working properly (only workaround
with direct commands and no good idea integration for debugging). I'm
back with maven, if anyone knows how to properly solve it let's do it.
If not I think JB point is to consider more than any other criteria.

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-10 16:41 GMT+02:00 Romain Manni-Bucau :
> side note: do NOT use auto-import until you are sure you can, it locks
> regularly on beam (pby too big for idea?) and makes idea ready to be
> killed :(
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
>
> 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré :
>> It's what I did, I'm trying a complete reload now (maybe this step failed).
>>
>> On 10/04/2018 16:38, Lukasz Cwik wrote:
>>>
>>> beam-site PR/414 updates the instructions for using Intellij and how to
>>> import a module:
>>>
>>> 1. Create an empty IntelliJ project outside of the Beam source tree.
>>> 2. Under Project Structure > Project, select a Project SDK.
>>> 3. Under Project Structure > Modules, click the + sign to add a module and
>>> select "Import Module".
>>>  1. Select the directory containing the Beam source tree.
>>>  2. Tick the "Import module from external model" button and select
>>> Gradle
>>> from the list.
>>>  3. Tick the following boxes.
>>> * Use auto-import
>>> * Create separate module per source set
>>> * Store generated project files externally
>>> * Use default gradle wrapper
>>> 4. Delegate build actions to Gradle by going to Settings > Build,
>>> Execution,
>>> Deployment > Build Tools > Gradle and checking "Delegate IDE build/run
>>> actions to gradle".
>>>
>>> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré >> > wrote:
>>>
>>> That's a very important issue for contribution. Up to now, I used
>>> Maven
>>> for setup IntelliJ (and it works just fine). If we remove the pom.xml,
>>> we have to support Eclipse and IntelliJ "smoothly".
>>>
>>> Let me try in IntelliJ.
>>>
>>> Regards
>>> JB
>>>
>>> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
>>>  > You dont have issue due to the build setup with that option. I get:
>>>  >
>>>  > avr. 10, 2018 3:20:10 PM
>>>  > org.apache.beam.runners.direct.DirectTransformExecutor run
>>>  > GRAVE: Error occurred within
>>>  > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
>>>  > com.google.common.util.concurrent.ExecutionError:
>>>  > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
>>>  >
>>>  > ?
>>>  >
>>>  > Romain Manni-Bucau
>>>  > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>  >
>>>  >
>>>  > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik >> >:
>>>  >> I have found that the simplest setup is to delegate the
>>> build/test actions
>>>  >> to Gradle. This allows you to run unit tests very easily and
>>> since its in
>>>  >> the same manner that Gradle would have, you know that if its
>>> passing it will
>>>  >> pass on the command line and on Jenkins. Here is one site that
>>> discusses how
>>>  >> to set this up:
>>>  >>
>>>
>>> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
>>>  >>
>>>  >>
>>>  >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau
>>> mailto:rmannibu...@gmail.com>>
>>>  >> wrote:
>>>  >>>
>>>  >>> What's the plan to make idea supporting gradle on beam project?
>>> Do we
>>>  >>> import the workaround mentionned in
>>>  >>> https://youtrack.jetbrains.com/issue/IDEA-175172?
>>>  >>> For the ones who didn't see this issue in action: idea will
>>> compile in
>>>  >>> out/ instead of build/ and you will just miss all the resources
>>> you
>>>  >>> need like some SPI registration which are used by all our
>>> registrar =>
>>>  >>> no way to run tests in idea without hacking the configuration
>>> quite
>>>  >>> deeply :(
>>>  >>>
>>>  >>> Romain Manni-Bucau
>>>  >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>  >>>
>>>  >>>
>>>  >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot
>>> mailto:echauc...@apache.org>>:
>>>
>>>   As a gradle beginner, I could not agree more !
>>>   +1
>>>   Etienne
>>>   Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a
>>> écrit :
>>>  
>>>   Hi all,
>>>  
>>>   I did multiple gradle build since last week and I would like
>>> to share
>>>   one of my concern: it's about the communities.
>>>  
>>>   If I think our users won't see any change for them due to
>>> Gradle build
>>>   (I think that most of our users will still use Maven with
>>> artifacts
>>>  >>