Re: Gradle Status: Migrated!

[Henning Rohde]

JB - for your comparison, please also omit cross-compiling all the Go
examples because they are only built using Gradle.




On Tue, May 1, 2018 at 8:59 AM Jean-Baptiste Onofré  wrote:

> Thanks for the update Kenn, that makes sense.
>
> I'm checking the artifacts generated by Gradle right now.
>
> Regards
> JB
>
> On 01/05/2018 17:42, Kenneth Knowles wrote:
> > Raw execution time for tasks from clean is not the only thing to test. I
> > would say it is not even important. Try these from clean:
> >
> >   - Gradle: ./gradlew :beam-sdks-java-io-mongodb:test && ./gradlew
> > :beam-sdks-java-io-mongodb:test
> >   - Maven: mvn -pl sdks/java/io/mongodb test -am && mvn -pl
> > sdks/java/io/mongodb test -am
> >
> > Quick run on my laptop:
> >
> >   - Gradle: 66s (65s then 1s)
> >   - Maven: 317s (173s then 144s)
> >
> > Of course, the mvn command runs a bunch of useless executions AND it is
> > incorrect because it isn't using built jars. That's part of the point -
> > there is no way to do what you want with mvn. Let's try to make a
> > command that avoids useless work and builds the jars:
> >
> >   - Maven:  (mvn -pl sdks/java/io/mongodb install -DskipTests -am && mvn
> > -pl sdks/java/io/mongodb test) && (each time)
> >
> > That takes 102s the first time and 64s the second time. And that is
> > about the realistic workflow for someone trying to get something done.
> > Even if we touch a file Gradle finishes in 20s. So the best case for mvn
> > is this head-to-head:
> >
> >   - Gradle: 65s + 20s + 20s + 20s + 20s + ...
> >   - Maven: 102s + 64s + 64s + 64s + 64s + ...
> >
> > Kenn
> >
> >
> > On Tue, May 1, 2018 at 8:09 AM Jean-Baptiste Onofré  > > wrote:
> >
> > Thanks, for me, Maven 3.5.2 takes quite the same time than Gradle
> > (using
> > the wrapper). It's maybe related to my environment.
> >
> > Anyway, I'm doing a complete build review both in term of building
> > time,
> > and equivalence (artifacts publishing, test, plugin execution).
> >
> > I will provide an update soon.
> >
> > Regards
> > JB
> >
> > On 01/05/2018 16:57, Reuven Lax wrote:
> >  > Luke did gather data which showed that on our Jenkins executors
> the
> >  > Gradle build was much faster than the Maven build. Also right now
> we
> >  > have incremental builds turned off, but once we're confident
> > enough to
> >  > enable them (at least for local development) that will often drop
> > build
> >  > times a lot.
> >  >
> >  > On Tue, May 1, 2018 at 4:01 AM Jean-Baptiste Onofré
> > 
> >  > >> wrote:
> >  >
> >  > By the way, I'm curious: did someone evaluate the build time
> gap
> >  > between Maven
> >  > and Gradle ? One of the main reason to migrate to Gradle was
> > the inc
> >  > build and
> >  > build time. The builds I have launched are quite the same in
> >  > duration. I will do
> >  > deeper tests to evaluate the gap.
> >  >
> >  > Regards
> >  > JB
> >  >
> >  > On 05/01/2018 12:48 PM, Łukasz Gajowy wrote:
> >  >  > Hi Scott,
> >  >  >
> >  >  > thanks for the update! Just a clarification about IO
> > performance
> >  > tests: those
> >  >  > were fully migrated in Beam and all task necessary for
> running
> >  > them are there
> >  >  > but Jenkins jobs still run mvn commands. This is due the
> > fact that
> >  >  > PerfkitBenchmarker code (which is invoked by Jenkins and
> >  > constructs the commands
> >  >  > by itself) was not updated yet. This should be finished
> before
> >  > fully dropping mvn.
> >  >  >
> >  >  > More on that topic here, in
> >  >  > comments: https://issues.apache.org/jira/browse/BEAM-3942
> >  >  > PR changing the commands to gradle is waiting for PerfKit
> > devs review
> >  >  > here:
> >  >
> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648
> >  >  >
> >  >  > Best regards,
> >  >  >
> >  >  > 2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau
> >  > 
> > >
> >  >  >  >   >  >  >  >
> >  >  > Hi Scott
> >  >  >
> >  >  > While
> >  >
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
> >  >  >
> >  >
> >   <
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057> is
> >  >  > open, gradle is a concurrent 

Re: Gradle Status: Migrated!

[Jean-Baptiste Onofré]

Thanks for the update Kenn, that makes sense.

I'm checking the artifacts generated by Gradle right now.

Regards
JB

On 01/05/2018 17:42, Kenneth Knowles wrote:
Raw execution time for tasks from clean is not the only thing to test. I 
would say it is not even important. Try these from clean:


  - Gradle: ./gradlew :beam-sdks-java-io-mongodb:test && ./gradlew 
:beam-sdks-java-io-mongodb:test
  - Maven: mvn -pl sdks/java/io/mongodb test -am && mvn -pl 
sdks/java/io/mongodb test -am


Quick run on my laptop:

  - Gradle: 66s (65s then 1s)
  - Maven: 317s (173s then 144s)

Of course, the mvn command runs a bunch of useless executions AND it is 
incorrect because it isn't using built jars. That's part of the point - 
there is no way to do what you want with mvn. Let's try to make a 
command that avoids useless work and builds the jars:


  - Maven:  (mvn -pl sdks/java/io/mongodb install -DskipTests -am && mvn 
-pl sdks/java/io/mongodb test) && (each time)


That takes 102s the first time and 64s the second time. And that is 
about the realistic workflow for someone trying to get something done. 
Even if we touch a file Gradle finishes in 20s. So the best case for mvn 
is this head-to-head:


  - Gradle: 65s + 20s + 20s + 20s + 20s + ...
  - Maven: 102s + 64s + 64s + 64s + 64s + ...

Kenn


On Tue, May 1, 2018 at 8:09 AM Jean-Baptiste Onofré > wrote:


Thanks, for me, Maven 3.5.2 takes quite the same time than Gradle
(using
the wrapper). It's maybe related to my environment.

Anyway, I'm doing a complete build review both in term of building
time,
and equivalence (artifacts publishing, test, plugin execution).

I will provide an update soon.

Regards
JB

On 01/05/2018 16:57, Reuven Lax wrote:
 > Luke did gather data which showed that on our Jenkins executors the
 > Gradle build was much faster than the Maven build. Also right now we
 > have incremental builds turned off, but once we're confident
enough to
 > enable them (at least for local development) that will often drop
build
 > times a lot.
 >
 > On Tue, May 1, 2018 at 4:01 AM Jean-Baptiste Onofré

 > >> wrote:
 >
 >     By the way, I'm curious: did someone evaluate the build time gap
 >     between Maven
 >     and Gradle ? One of the main reason to migrate to Gradle was
the inc
 >     build and
 >     build time. The builds I have launched are quite the same in
 >     duration. I will do
 >     deeper tests to evaluate the gap.
 >
 >     Regards
 >     JB
 >
 >     On 05/01/2018 12:48 PM, Łukasz Gajowy wrote:
 >      > Hi Scott,
 >      >
 >      > thanks for the update! Just a clarification about IO
performance
 >     tests: those
 >      > were fully migrated in Beam and all task necessary for running
 >     them are there
 >      > but Jenkins jobs still run mvn commands. This is due the
fact that
 >      > PerfkitBenchmarker code (which is invoked by Jenkins and
 >     constructs the commands
 >      > by itself) was not updated yet. This should be finished before
 >     fully dropping mvn.
 >      >
 >      > More on that topic here, in
 >      > comments: https://issues.apache.org/jira/browse/BEAM-3942
 >      > PR changing the commands to gradle is waiting for PerfKit
devs review
 >      > here:
 > https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648
 >      >
 >      > Best regards,
 >      >
 >      > 2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau
 >     
>
 >      >        >
 >      >     Hi Scott
 >      >
 >      >     While
 > https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
 >      >
 > 
   is

 >      >     open, gradle is a concurrent of maven but maven must
stay the
 >     default build
 >      >     tool cause gradle breaks users.
 >      >
 >      >
 >      >     Le 1 mai 2018 01:59, "Scott Wegner"

 >     >
 >      >     
     écrit :
 >      >
 >      >         Many many of you have been hacking diligently on the
 >     Gradle build, and
 >      >         I'm happy to announce that we now 

Re: Gradle Status: Migrated!

[Kenneth Knowles]

Raw execution time for tasks from clean is not the only thing to test. I
would say it is not even important. Try these from clean:

 - Gradle: ./gradlew :beam-sdks-java-io-mongodb:test && ./gradlew :
beam-sdks-java-io-mongodb:test
 - Maven: mvn -pl sdks/java/io/mongodb test -am && mvn -pl
sdks/java/io/mongodb test -am

Quick run on my laptop:

 - Gradle: 66s (65s then 1s)
 - Maven: 317s (173s then 144s)

Of course, the mvn command runs a bunch of useless executions AND it is
incorrect because it isn't using built jars. That's part of the point -
there is no way to do what you want with mvn. Let's try to make a command
that avoids useless work and builds the jars:

 - Maven:  (mvn -pl sdks/java/io/mongodb install -DskipTests -am && mvn -pl
sdks/java/io/mongodb test) && (each time)

That takes 102s the first time and 64s the second time. And that is about
the realistic workflow for someone trying to get something done. Even if we
touch a file Gradle finishes in 20s. So the best case for mvn is this
head-to-head:

 - Gradle: 65s + 20s + 20s + 20s + 20s + ...
 - Maven: 102s + 64s + 64s + 64s + 64s + ...

Kenn


On Tue, May 1, 2018 at 8:09 AM Jean-Baptiste Onofré  wrote:

> Thanks, for me, Maven 3.5.2 takes quite the same time than Gradle (using
> the wrapper). It's maybe related to my environment.
>
> Anyway, I'm doing a complete build review both in term of building time,
> and equivalence (artifacts publishing, test, plugin execution).
>
> I will provide an update soon.
>
> Regards
> JB
>
> On 01/05/2018 16:57, Reuven Lax wrote:
> > Luke did gather data which showed that on our Jenkins executors the
> > Gradle build was much faster than the Maven build. Also right now we
> > have incremental builds turned off, but once we're confident enough to
> > enable them (at least for local development) that will often drop build
> > times a lot.
> >
> > On Tue, May 1, 2018 at 4:01 AM Jean-Baptiste Onofré  > > wrote:
> >
> > By the way, I'm curious: did someone evaluate the build time gap
> > between Maven
> > and Gradle ? One of the main reason to migrate to Gradle was the inc
> > build and
> > build time. The builds I have launched are quite the same in
> > duration. I will do
> > deeper tests to evaluate the gap.
> >
> > Regards
> > JB
> >
> > On 05/01/2018 12:48 PM, Łukasz Gajowy wrote:
> >  > Hi Scott,
> >  >
> >  > thanks for the update! Just a clarification about IO performance
> > tests: those
> >  > were fully migrated in Beam and all task necessary for running
> > them are there
> >  > but Jenkins jobs still run mvn commands. This is due the fact that
> >  > PerfkitBenchmarker code (which is invoked by Jenkins and
> > constructs the commands
> >  > by itself) was not updated yet. This should be finished before
> > fully dropping mvn.
> >  >
> >  > More on that topic here, in
> >  > comments: https://issues.apache.org/jira/browse/BEAM-3942
> >  > PR changing the commands to gradle is waiting for PerfKit devs
> review
> >  > here:
> > https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648
> >  >
> >  > Best regards,
> >  >
> >  > 2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau
> > 
> >  > >>:
> >  >
> >  > Hi Scott
> >  >
> >  > While
> >
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
> >  >
> >   <
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057> is
> >  > open, gradle is a concurrent of maven but maven must stay the
> > default build
> >  > tool cause gradle breaks users.
> >  >
> >  >
> >  > Le 1 mai 2018 01:59, "Scott Wegner"  > 
> >  > >> a
> > écrit :
> >  >
> >  > Many many of you have been hacking diligently on the
> > Gradle build, and
> >  > I'm happy to announce that we now have a
> > fully-functioning Gradle build!
> >  > There's been a ton of progress since our last update [1]:
> >  >
> >  > * Improved nightly snapshot release [2]
> >  > * Improve runner quickstarts [5] [11]
> >  > * Python post-commit ported to Gradle [3]
> >  > * Update performance testing framework for Gradle [4] [12]
> >  > * Generate javadocs from Gradle [6]
> >  > * Update to latest Gradle version [7] [21]
> >  > * Updated documentation [8] [22]
> >  > * Tune CI build resource usage for Jenkins [9] [19]
> >  > * Improve shading of test jars [10] [13] [14]
> >  > * Add 'errorprone' and 'spotless' static analysis 

Re: Gradle Status: Migrated!

[Jean-Baptiste Onofré]

Thanks, for me, Maven 3.5.2 takes quite the same time than Gradle (using 
the wrapper). It's maybe related to my environment.


Anyway, I'm doing a complete build review both in term of building time, 
and equivalence (artifacts publishing, test, plugin execution).


I will provide an update soon.

Regards
JB

On 01/05/2018 16:57, Reuven Lax wrote:
Luke did gather data which showed that on our Jenkins executors the 
Gradle build was much faster than the Maven build. Also right now we 
have incremental builds turned off, but once we're confident enough to 
enable them (at least for local development) that will often drop build 
times a lot.


On Tue, May 1, 2018 at 4:01 AM Jean-Baptiste Onofré > wrote:


By the way, I'm curious: did someone evaluate the build time gap
between Maven
and Gradle ? One of the main reason to migrate to Gradle was the inc
build and
build time. The builds I have launched are quite the same in
duration. I will do
deeper tests to evaluate the gap.

Regards
JB

On 05/01/2018 12:48 PM, Łukasz Gajowy wrote:
 > Hi Scott,
 >
 > thanks for the update! Just a clarification about IO performance
tests: those
 > were fully migrated in Beam and all task necessary for running
them are there
 > but Jenkins jobs still run mvn commands. This is due the fact that
 > PerfkitBenchmarker code (which is invoked by Jenkins and
constructs the commands
 > by itself) was not updated yet. This should be finished before
fully dropping mvn.
 >
 > More on that topic here, in
 > comments: https://issues.apache.org/jira/browse/BEAM-3942
 > PR changing the commands to gradle is waiting for PerfKit devs review
 > here:
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648
 >
 > Best regards,
 >
 > 2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau

 > >>:
 >
 >     Hi Scott
 >
 >     While
https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
 >   
   is

 >     open, gradle is a concurrent of maven but maven must stay the
default build
 >     tool cause gradle breaks users.
 >
 >
 >     Le 1 mai 2018 01:59, "Scott Wegner" 
 >     >> a
écrit :
 >
 >         Many many of you have been hacking diligently on the
Gradle build, and
 >         I'm happy to announce that we now have a
fully-functioning Gradle build!
 >         There's been a ton of progress since our last update [1]:
 >
 >         * Improved nightly snapshot release [2]
 >         * Improve runner quickstarts [5] [11]
 >         * Python post-commit ported to Gradle [3]
 >         * Update performance testing framework for Gradle [4] [12]
 >         * Generate javadocs from Gradle [6]
 >         * Update to latest Gradle version [7] [21]
 >         * Updated documentation [8] [22]
 >         * Tune CI build resource usage for Jenkins [9] [19]
 >         * Improve shading of test jars [10] [13] [14]
 >         * Add 'errorprone' and 'spotless' static analysis [15] [24]
 >         * Improve IntelliJ project generation [16] [17]
 >         * Reduce number of ValidatesRunner tests [18]
 >         * Update release documentation for Gradle [20]
 >         * Update docker build scripts for Gradle [23]
 >
 >         The build process and Jenkins environment have stabilized
and we've
 >         resolved migration blockers. The final step is to use
Gradle to produce
 >         an official release. The release documentation has been
updated for
 >         Gradle and I recommend we use these docs for the 2.5.0
release. Assuming
 >         the release goes well, we can declare the migration fully
validated and
 >         stop supporting dual build systems.
 >
 >         During the migration we identified a number of
opportunities to improve
 >         the build even further. Feel free to grab one of the
items off of the
 >         JIRA: BEAM-4045 [24]
 >
 >         Thanks again to all those that contributed. This has
truly been a
 >         community effort!
 >
 >         [1]

https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
 >   
  

 >         [2] https://github.com/apache/beam/pull/5142
 >         

Re: Gradle Status: Migrated!

[Reuven Lax]

Luke did gather data which showed that on our Jenkins executors the Gradle
build was much faster than the Maven build. Also right now we have
incremental builds turned off, but once we're confident enough to enable
them (at least for local development) that will often drop build times a
lot.

On Tue, May 1, 2018 at 4:01 AM Jean-Baptiste Onofré  wrote:

> By the way, I'm curious: did someone evaluate the build time gap between
> Maven
> and Gradle ? One of the main reason to migrate to Gradle was the inc build
> and
> build time. The builds I have launched are quite the same in duration. I
> will do
> deeper tests to evaluate the gap.
>
> Regards
> JB
>
> On 05/01/2018 12:48 PM, Łukasz Gajowy wrote:
> > Hi Scott,
> >
> > thanks for the update! Just a clarification about IO performance tests:
> those
> > were fully migrated in Beam and all task necessary for running them are
> there
> > but Jenkins jobs still run mvn commands. This is due the fact that
> > PerfkitBenchmarker code (which is invoked by Jenkins and constructs the
> commands
> > by itself) was not updated yet. This should be finished before fully
> dropping mvn.
> >
> > More on that topic here, in
> > comments: https://issues.apache.org/jira/browse/BEAM-3942
> > PR changing the commands to gradle is waiting for PerfKit devs review
> > here:
> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648
> >
> > Best regards,
> >
> > 2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau  > >:
> >
> > Hi Scott
> >
> > While
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
> > <
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057> is
> > open, gradle is a concurrent of maven but maven must stay the
> default build
> > tool cause gradle breaks users.
> >
> >
> > Le 1 mai 2018 01:59, "Scott Wegner"  > > a écrit :
> >
> > Many many of you have been hacking diligently on the Gradle
> build, and
> > I'm happy to announce that we now have a fully-functioning
> Gradle build!
> > There's been a ton of progress since our last update [1]:
> >
> > * Improved nightly snapshot release [2]
> > * Improve runner quickstarts [5] [11]
> > * Python post-commit ported to Gradle [3]
> > * Update performance testing framework for Gradle [4] [12]
> > * Generate javadocs from Gradle [6]
> > * Update to latest Gradle version [7] [21]
> > * Updated documentation [8] [22]
> > * Tune CI build resource usage for Jenkins [9] [19]
> > * Improve shading of test jars [10] [13] [14]
> > * Add 'errorprone' and 'spotless' static analysis [15] [24]
> > * Improve IntelliJ project generation [16] [17]
> > * Reduce number of ValidatesRunner tests [18]
> > * Update release documentation for Gradle [20]
> > * Update docker build scripts for Gradle [23]
> >
> > The build process and Jenkins environment have stabilized and
> we've
> > resolved migration blockers. The final step is to use Gradle to
> produce
> > an official release. The release documentation has been updated
> for
> > Gradle and I recommend we use these docs for the 2.5.0 release.
> Assuming
> > the release goes well, we can declare the migration fully
> validated and
> > stop supporting dual build systems.
> >
> > During the migration we identified a number of opportunities to
> improve
> > the build even further. Feel free to grab one of the items off
> of the
> > JIRA: BEAM-4045 [24]
> >
> > Thanks again to all those that contributed. This has truly been a
> > community effort!
> >
> > [1]
> https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
> > <
> https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
> >
> > [2] https://github.com/apache/beam/pull/5142
> > 
> > [3] https://github.com/apache/beam/pull/5146
> > 
> > [4] https://github.com/apache/beam/pull/5003
> > 
> > [5] https://github.com/apache/beam/pull/5151
> > 
> > [6] https://github.com/apache/beam/pull/5121
> > 
> > [7] https://github.com/apache/beam/pull/5104
> > 
> > [8] https://github.com/apache/beam/pull/5183
> > 
> > [9] https://github.com/apache/beam/pull/5171
> > 

Re: Gradle Status: Migrated!

[Jean-Baptiste Onofré]

By the way, I'm curious: did someone evaluate the build time gap between Maven
and Gradle ? One of the main reason to migrate to Gradle was the inc build and
build time. The builds I have launched are quite the same in duration. I will do
deeper tests to evaluate the gap.

Regards
JB

On 05/01/2018 12:48 PM, Łukasz Gajowy wrote:
> Hi Scott, 
> 
> thanks for the update! Just a clarification about IO performance tests: those
> were fully migrated in Beam and all task necessary for running them are there
> but Jenkins jobs still run mvn commands. This is due the fact that
> PerfkitBenchmarker code (which is invoked by Jenkins and constructs the 
> commands
> by itself) was not updated yet. This should be finished before fully dropping 
> mvn. 
> 
> More on that topic here, in
> comments: https://issues.apache.org/jira/browse/BEAM-3942
> PR changing the commands to gradle is waiting for PerfKit devs review
> here: https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648
> 
> Best regards,
> 
> 2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau  >:
> 
> Hi Scott
> 
> While 
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
>  is
> open, gradle is a concurrent of maven but maven must stay the default 
> build
> tool cause gradle breaks users.
> 
> 
> Le 1 mai 2018 01:59, "Scott Wegner"  > a écrit :
> 
> Many many of you have been hacking diligently on the Gradle build, and
> I'm happy to announce that we now have a fully-functioning Gradle 
> build!
> There's been a ton of progress since our last update [1]:
> 
> * Improved nightly snapshot release [2]
> * Improve runner quickstarts [5] [11]
> * Python post-commit ported to Gradle [3]
> * Update performance testing framework for Gradle [4] [12]
> * Generate javadocs from Gradle [6]
> * Update to latest Gradle version [7] [21]
> * Updated documentation [8] [22]
> * Tune CI build resource usage for Jenkins [9] [19]
> * Improve shading of test jars [10] [13] [14]
> * Add 'errorprone' and 'spotless' static analysis [15] [24]
> * Improve IntelliJ project generation [16] [17]
> * Reduce number of ValidatesRunner tests [18]
> * Update release documentation for Gradle [20]
> * Update docker build scripts for Gradle [23]
> 
> The build process and Jenkins environment have stabilized and we've
> resolved migration blockers. The final step is to use Gradle to 
> produce
> an official release. The release documentation has been updated for
> Gradle and I recommend we use these docs for the 2.5.0 release. 
> Assuming
> the release goes well, we can declare the migration fully validated 
> and
> stop supporting dual build systems.
> 
> During the migration we identified a number of opportunities to 
> improve
> the build even further. Feel free to grab one of the items off of the
> JIRA: BEAM-4045 [24]
> 
> Thanks again to all those that contributed. This has truly been a
> community effort!
> 
> [1] 
> https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
> 
> 
> [2] https://github.com/apache/beam/pull/5142
>  
> [3] https://github.com/apache/beam/pull/5146
>  
> [4] https://github.com/apache/beam/pull/5003
>  
> [5] https://github.com/apache/beam/pull/5151
>  
> [6] https://github.com/apache/beam/pull/5121
>  
> [7] https://github.com/apache/beam/pull/5104
>  
> [8] https://github.com/apache/beam/pull/5183
>  
> [9] https://github.com/apache/beam/pull/5171
>  
> [10] https://github.com/apache/beam/pull/5117
>  
> [11] https://github.com/apache/beam/pull/5200
>  
> [12] https://github.com/apache/beam/pull/5051
>  
> [13] https://github.com/apache/beam/pull/4740
>  
> [14] https://github.com/apache/beam/pull/4702
> 

Re: Gradle Status: Migrated!

[Łukasz Gajowy]

Hi Scott,

thanks for the update! Just a clarification about IO performance tests:
those were fully migrated in Beam and all task necessary for running them
are there but Jenkins jobs still run mvn commands. This is due the fact
that PerfkitBenchmarker code (which is invoked by Jenkins and constructs
the commands by itself) was not updated yet. This should be finished before
fully dropping mvn.

More on that topic here, in comments:
https://issues.apache.org/jira/browse/BEAM-3942
PR changing the commands to gradle is waiting for PerfKit devs review here:
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1648

Best regards,

2018-05-01 9:17 GMT+02:00 Romain Manni-Bucau :

> Hi Scott
>
> While https://issues.apache.org/jira/plugins/servlet/
> mobile#issue/BEAM-4057 is open, gradle is a concurrent of maven but maven
> must stay the default build tool cause gradle breaks users.
>
>
> Le 1 mai 2018 01:59, "Scott Wegner"  a écrit :
>
>> Many many of you have been hacking diligently on the Gradle build, and
>> I'm happy to announce that we now have a fully-functioning Gradle build!
>> There's been a ton of progress since our last update [1]:
>>
>> * Improved nightly snapshot release [2]
>> * Improve runner quickstarts [5] [11]
>> * Python post-commit ported to Gradle [3]
>> * Update performance testing framework for Gradle [4] [12]
>> * Generate javadocs from Gradle [6]
>> * Update to latest Gradle version [7] [21]
>> * Updated documentation [8] [22]
>> * Tune CI build resource usage for Jenkins [9] [19]
>> * Improve shading of test jars [10] [13] [14]
>> * Add 'errorprone' and 'spotless' static analysis [15] [24]
>> * Improve IntelliJ project generation [16] [17]
>> * Reduce number of ValidatesRunner tests [18]
>> * Update release documentation for Gradle [20]
>> * Update docker build scripts for Gradle [23]
>>
>> The build process and Jenkins environment have stabilized and we've
>> resolved migration blockers. The final step is to use Gradle to produce an
>> official release. The release documentation has been updated for Gradle and
>> I recommend we use these docs for the 2.5.0 release. Assuming the release
>> goes well, we can declare the migration fully validated and stop supporting
>> dual build systems.
>>
>> During the migration we identified a number of opportunities to improve
>> the build even further. Feel free to grab one of the items off of the JIRA:
>> BEAM-4045 [24]
>>
>> Thanks again to all those that contributed. This has truly been a
>> community effort!
>>
>> [1] https://lists.apache.org/thread.html/5f6bae323acc1b05096
>> 2e68ec310613e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
>> [2] https://github.com/apache/beam/pull/5142
>> [3] https://github.com/apache/beam/pull/5146
>> [4] https://github.com/apache/beam/pull/5003
>> [5] https://github.com/apache/beam/pull/5151
>> [6] https://github.com/apache/beam/pull/5121
>> [7] https://github.com/apache/beam/pull/5104
>> [8] https://github.com/apache/beam/pull/5183
>> [9] https://github.com/apache/beam/pull/5171
>> [10] https://github.com/apache/beam/pull/5117
>> [11] https://github.com/apache/beam/pull/5200
>> [12] https://github.com/apache/beam/pull/5051
>> [13] https://github.com/apache/beam/pull/4740
>> [14] https://github.com/apache/beam/pull/4702
>> [15] https://github.com/apache/beam/pull/4701
>> [16] https://github.com/apache/beam/pull/4626
>> [17] https://github.com/apache/beam/pull/4625
>> [18] https://github.com/apache/beam/pull/5193
>> [19] https://github.com/apache/beam/pull/5222
>> [20] https://github.com/apache/beam/pull/5187
>> [21] https://github.com/apache/beam/pull/5217
>> [22] https://github.com/apache/beam/pull/5115
>> [23] https://github.com/apache/beam/pull/5252
>> [24] https://github.com/apache/beam/pull/5161
>> [25] https://issues.apache.org/jira/browse/BEAM-4045
>> --
>>
>>
>> Got feedback? http://go/swegner-feedback
>>
>

Re: Gradle Status: Migrated!

[Romain Manni-Bucau]

Hi Scott

While https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-4057
is open, gradle is a concurrent of maven but maven must stay the default
build tool cause gradle breaks users.


Le 1 mai 2018 01:59, "Scott Wegner"  a écrit :

> Many many of you have been hacking diligently on the Gradle build, and I'm
> happy to announce that we now have a fully-functioning Gradle build!
> There's been a ton of progress since our last update [1]:
>
> * Improved nightly snapshot release [2]
> * Improve runner quickstarts [5] [11]
> * Python post-commit ported to Gradle [3]
> * Update performance testing framework for Gradle [4] [12]
> * Generate javadocs from Gradle [6]
> * Update to latest Gradle version [7] [21]
> * Updated documentation [8] [22]
> * Tune CI build resource usage for Jenkins [9] [19]
> * Improve shading of test jars [10] [13] [14]
> * Add 'errorprone' and 'spotless' static analysis [15] [24]
> * Improve IntelliJ project generation [16] [17]
> * Reduce number of ValidatesRunner tests [18]
> * Update release documentation for Gradle [20]
> * Update docker build scripts for Gradle [23]
>
> The build process and Jenkins environment have stabilized and we've
> resolved migration blockers. The final step is to use Gradle to produce an
> official release. The release documentation has been updated for Gradle and
> I recommend we use these docs for the 2.5.0 release. Assuming the release
> goes well, we can declare the migration fully validated and stop supporting
> dual build systems.
>
> During the migration we identified a number of opportunities to improve
> the build even further. Feel free to grab one of the items off of the JIRA:
> BEAM-4045 [24]
>
> Thanks again to all those that contributed. This has truly been a
> community effort!
>
> [1] https://lists.apache.org/thread.html/5f6bae323acc1b050962e68ec31061
> 3e0121b05bc5c42915c536fb59@%3Cdev.beam.apache.org%3E
> [2] https://github.com/apache/beam/pull/5142
> [3] https://github.com/apache/beam/pull/5146
> [4] https://github.com/apache/beam/pull/5003
> [5] https://github.com/apache/beam/pull/5151
> [6] https://github.com/apache/beam/pull/5121
> [7] https://github.com/apache/beam/pull/5104
> [8] https://github.com/apache/beam/pull/5183
> [9] https://github.com/apache/beam/pull/5171
> [10] https://github.com/apache/beam/pull/5117
> [11] https://github.com/apache/beam/pull/5200
> [12] https://github.com/apache/beam/pull/5051
> [13] https://github.com/apache/beam/pull/4740
> [14] https://github.com/apache/beam/pull/4702
> [15] https://github.com/apache/beam/pull/4701
> [16] https://github.com/apache/beam/pull/4626
> [17] https://github.com/apache/beam/pull/4625
> [18] https://github.com/apache/beam/pull/5193
> [19] https://github.com/apache/beam/pull/5222
> [20] https://github.com/apache/beam/pull/5187
> [21] https://github.com/apache/beam/pull/5217
> [22] https://github.com/apache/beam/pull/5115
> [23] https://github.com/apache/beam/pull/5252
> [24] https://github.com/apache/beam/pull/5161
> [25] https://issues.apache.org/jira/browse/BEAM-4045
> --
>
>
> Got feedback? http://go/swegner-feedback
>

Re: Gradle Status [April 11]

[Nathan Fisher]

Great thanks for that! Makes perfect sense.

On Wed, Apr 11, 2018 at 7:59 PM, Kenneth Knowles  wrote:

> There are plenty of technical hurdles since Bazel is best suited to an
> isolated monorepo. We could probably have overcome those problems with
> effort. I got through a few of them in the short sprint I did. Ultimately
> it was a community-driven decision: more people (as in "more than just me"
> :-) jumped on board and got excited about building out the Gradle build.
>
> Kenn
>
> On Wed, Apr 11, 2018 at 3:17 PM Nathan Fisher 
> wrote:
>
>> Is there a document or similar outlining the decision behind Gradle over
>> Bazel?
>>
>> For purely intellectual curiosity I’m curious what tradeoffs/benefits
>> were considered when evaluating the two or if it was more a matter of
>> community/contributor familiarity.
>>
>> I found this thread which seems to imply issues around dependency
>> management;
>>
>>
>> https://lists.apache.org/thread.html/bba0a89f2561fb2f7150a8381c1eb3923fa46299f3b35ff1304d7c46@%3Cdev.beam.apache.org%3E
>>
>> On Wed, Apr 11, 2018 at 3:21 PM, Kenneth Knowles  wrote:
>>
>>> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
>>> Netty has been introduced since yesterday. Etienne mentioned he has worked
>>> toward setting up periodic runs on all runners, so this should help get us
>>> towards that. We'll probably prefer to build standalone fat jars for
>>> selected runners and use those, which is pending unknown issues in the
>>> shadow config leaving out required dependencies.
>>>
>>> Kenn
>>>
>>> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner 
>>> wrote:
>>>
 Thanks everyone for the continued effort towards the Gradle migration.
 As a high-level summary of our progress since Friday: we have a viable
 build, with a number of minor issues that we're still working out. Please
 take a look at the new documentation in our contribution guide and log any
 bugs that you find.

 Here's a more detailed view of improvements from just the past few
 days..

 Release artifacts:
 *  Pom.xml generation logic now in master [1]
 * Nightly snapshots are now produced using Gradle [2]
 * Excluded modules propagated to dependencies when generating * pom.xml
 * Artifact JARs are properly shaded [3]
 * Working on fixing dependency scopes in generated pom [4]
 PreCommits / Postcommits:
 * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
 [7] [8] [9]
 * Jenkins results now include JUnit test results [10] and build scan
 for easier debugging [11]
 * Spark ValidatesRunner PostCommit passes [12] [13]
 * Flink ValidatesRunner PostCommit more reliable [14]
 Documentation / IDE Setup:
 * Contribution Guide [15] is now updated with Gradle commands [16] [17]
 Performance Benchmarks:
 * Working on getting Nexmark benchmarks migrated [18]

 If I missed anything, please add it to this thread.

 We are continuing to use this general roadmap:
 (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
 (b) Postcommits migrated to Gradle
 (c) Migrate documentation from maven to Gradle
 (d) Migrate perfkit suites to use Gradle

 Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
 created a separate issue to track post-migration cleanup items:
 BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.


 [1] https://github.com/apache/beam/pull/5054
 [2] https://github.com/apache/beam/pull/5057
 [3] https://github.com/apache/beam/pull/5087
 [4] https://github.com/apache/beam/pull/5098
 [5] https://github.com/apache/beam/pull/5047
 [6] https://github.com/apache/beam/pull/5088
 [7] https://github.com/apache/beam/pull/5086
 [8] https://github.com/apache/beam/pull/5066
 [9] https://github.com/apache/beam/pull/5059
 [10] https://github.com/apache/beam/pull/5045
 [11] https://github.com/apache/beam/pull/5091
 [12] https://github.com/apache/beam/pull/5093
 [13] https://github.com/apache/beam/pull/5069
 [14] https://github.com/apache/beam/pull/5068
 [15] https://beam.apache.org/contribute/contribution-guide/
 [16] https://github.com/apache/beam-site/pull/412
 [17] https://github.com/apache/beam-site/pull/414
 [18] https://github.com/apache/beam/pull/5051
 [19] https://issues.apache.org/jira/browse/BEAM-3249
 [20] https://issues.apache.org/jira/browse/BEAM-4045

 On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:

> I wanted to start a thread to summarize the current state of Gradle
> migration. We've made lots of good progress so far this week. Here's the
> status from what I can tell-- please add or correct anything I missed:
>
> * Release artifacts can be built and published for 

Re: Gradle Status [April 6]

[Kenneth Knowles]

Caching success/failure of a test is legitimate useful behavior. That said,
I'm not sure how it interacts with Idea.

Kenn

On Sun, Apr 15, 2018 at 12:23 AM Romain Manni-Bucau 
wrote:

> If you dont the diff is empty and gradle runs nothing, no? Saw it with
> gradle 4.6
>
> Le 15 avr. 2018 00:49, "Kenneth Knowles"  a écrit :
>
>> You shouldn't do :module:cleanTest. If that is necessary that's a major
>> bug in the build.
>>
>> Kenn
>>
>> On Fri, Apr 13, 2018 at 11:46 PM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> There is a fake module xxx_test which should have the right classpath
>>> but since idea compilation is messed up you will still have to run
>>> :module:cleanTest :module:test --tests org...MyTest.myMethod, even with
>>> idea which leads to the same latency as the command line :(
>>>
>>> Le 13 avr. 2018 22:23, "Eugene Kirpichov"  a
>>> écrit :
>>>
 While we're talking about running tests in IntelliJ with Gradle...
 Anybody got advice on how to run a single NeedsRunner test in
 sdks-java-core, say, ParDoTest? With Maven, I used to just run the test in
 IntelliJ and specify "runners-direct-java" as the classpath; with Gradle,
 the best I got is to manually run the direct runner's needsRunnerTests task
 with specifying --tests=..., but it takes a long time, and IntelliJ treats
 that as just a gradle task run, not as a test run.

 On Fri, Apr 13, 2018 at 11:14 AM Reuven Lax  wrote:

> Is there a Jira for this 3 second delay? Also you're initial complaint
> was not about the 3 second delay, so it wasn't clear that's what you were
> complaining about.
>
> Reuven
>
> On Fri, Apr 13, 2018 at 4:42 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> When you launch a test with gradle runner it launches gradle which
>> makes loose 3s on a very fast computer and more on a slower (6 on my
>> personal one which is already fast but not as much as my work one). We 
>> are
>> 5 to see that regression at least. So there is a reason to not use the
>> gradle runner if possible cause when you work and need to debug you are
>> just stucked (that is why i switched back to mvn after 15mn, i was 
>> loosing
>> to much time).
>>
>> Switching back to native idea test run would fix it but tests just
>> dont work this way for me whatever setup i do :( - missing resources IIRC
>> in idea out dir.
>>
>> Le 13 avr. 2018 00:07, "Reuven Lax"  a écrit :
>>
>> I also don't quite understand what your question is, and it appears
>> like Dan spent considerable time trying to reproduce your issue. For the
>> record, I have had no issues running tests via Gradle in IntelliJ for the
>> past few weeks.
>>
>> Reuven
>>
>> On Thu, Apr 12, 2018 at 9:47 PM Daniel Oliveira <
>> danolive...@google.com> wrote:
>>
>>> Sorry Romain, I'm not quite sure what you're asking. Can you clarify?
>>>
>>> On Thu, Apr 12, 2018 at 12:22 PM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Well you are the only one to not have the drawbacks to use it so
 maybe dont do it? I know Luke is in holidays but anyone else with the
 knowledge of why we nees that noise compared to idea native 
 tooling/flow?

 Le 12 avr. 2018 20:16, "Daniel Oliveira" 
 a écrit :

> Ah, I did not. Thanks Romain.
>
> I tried it again, restarting in between, and still had no
> differences. Since it seems like there's no reason not to use "Gradle 
> Test
> Runner", I'll mention it in the contributor's guide.
>
> On Thu, Apr 12, 2018 at 10:31 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> @Daniel: did you restart in between? Otherwise it does nothing.
>> One launches JunitCoreRunner from idea and the other a gradle 
>> command.
>>
>> Le 12 avr. 2018 19:24, "Daniel Oliveira" 
>> a écrit :
>>
>>> I think it depends on what exactly switching to "Gradle Test
>>> Runner" from "Platform Test Runner" does. I tried it out on my 
>>> machine and
>>> they seem to act identically to each other. The IntelliJ 
>>> documentation says
>>> it determines what API to use to run the tests
>>> , so maybe
>>> it's usefulness depends on the user's machine, in which case a note 
>>> about
>>> that would be useful. Something like: "If your IDE has trouble 
>>> running
>>> tests via IDEA shortcuts, try the following steps: [...]"
>>>
>>> On Thu, Apr 12, 2018 at 3:29 AM 

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

If you dont the diff is empty and gradle runs nothing, no? Saw it with
gradle 4.6

Le 15 avr. 2018 00:49, "Kenneth Knowles"  a écrit :

> You shouldn't do :module:cleanTest. If that is necessary that's a major
> bug in the build.
>
> Kenn
>
> On Fri, Apr 13, 2018 at 11:46 PM Romain Manni-Bucau 
> wrote:
>
>> There is a fake module xxx_test which should have the right classpath but
>> since idea compilation is messed up you will still have to run
>> :module:cleanTest :module:test --tests org...MyTest.myMethod, even with
>> idea which leads to the same latency as the command line :(
>>
>> Le 13 avr. 2018 22:23, "Eugene Kirpichov"  a
>> écrit :
>>
>>> While we're talking about running tests in IntelliJ with Gradle...
>>> Anybody got advice on how to run a single NeedsRunner test in
>>> sdks-java-core, say, ParDoTest? With Maven, I used to just run the test in
>>> IntelliJ and specify "runners-direct-java" as the classpath; with Gradle,
>>> the best I got is to manually run the direct runner's needsRunnerTests task
>>> with specifying --tests=..., but it takes a long time, and IntelliJ treats
>>> that as just a gradle task run, not as a test run.
>>>
>>> On Fri, Apr 13, 2018 at 11:14 AM Reuven Lax  wrote:
>>>
 Is there a Jira for this 3 second delay? Also you're initial complaint
 was not about the 3 second delay, so it wasn't clear that's what you were
 complaining about.

 Reuven

 On Fri, Apr 13, 2018 at 4:42 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> When you launch a test with gradle runner it launches gradle which
> makes loose 3s on a very fast computer and more on a slower (6 on my
> personal one which is already fast but not as much as my work one). We are
> 5 to see that regression at least. So there is a reason to not use the
> gradle runner if possible cause when you work and need to debug you are
> just stucked (that is why i switched back to mvn after 15mn, i was loosing
> to much time).
>
> Switching back to native idea test run would fix it but tests just
> dont work this way for me whatever setup i do :( - missing resources IIRC
> in idea out dir.
>
> Le 13 avr. 2018 00:07, "Reuven Lax"  a écrit :
>
> I also don't quite understand what your question is, and it appears
> like Dan spent considerable time trying to reproduce your issue. For the
> record, I have had no issues running tests via Gradle in IntelliJ for the
> past few weeks.
>
> Reuven
>
> On Thu, Apr 12, 2018 at 9:47 PM Daniel Oliveira <
> danolive...@google.com> wrote:
>
>> Sorry Romain, I'm not quite sure what you're asking. Can you clarify?
>>
>> On Thu, Apr 12, 2018 at 12:22 PM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Well you are the only one to not have the drawbacks to use it so
>>> maybe dont do it? I know Luke is in holidays but anyone else with the
>>> knowledge of why we nees that noise compared to idea native 
>>> tooling/flow?
>>>
>>> Le 12 avr. 2018 20:16, "Daniel Oliveira"  a
>>> écrit :
>>>
 Ah, I did not. Thanks Romain.

 I tried it again, restarting in between, and still had no
 differences. Since it seems like there's no reason not to use "Gradle 
 Test
 Runner", I'll mention it in the contributor's guide.

 On Thu, Apr 12, 2018 at 10:31 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> @Daniel: did you restart in between? Otherwise it does nothing.
> One launches JunitCoreRunner from idea and the other a gradle command.
>
> Le 12 avr. 2018 19:24, "Daniel Oliveira" 
> a écrit :
>
>> I think it depends on what exactly switching to "Gradle Test
>> Runner" from "Platform Test Runner" does. I tried it out on my 
>> machine and
>> they seem to act identically to each other. The IntelliJ 
>> documentation says
>> it determines what API to use to run the tests
>> , so maybe it's
>> usefulness depends on the user's machine, in which case a note about 
>> that
>> would be useful. Something like: "If your IDE has trouble running 
>> tests via
>> IDEA shortcuts, try the following steps: [...]"
>>
>> On Thu, Apr 12, 2018 at 3:29 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> Daniel, actually I did run it with default IDEA JUnit test
>>> runner. Then, in “Settings > Build, Execution, Deployment > Build 
>>> Tools >
>>> Gradle > Runner" I selected “Gradle Test Runner” in “Run tests 
>>> using”
>>> 

Re: Gradle Status [April 6]

[Kenneth Knowles]

You shouldn't do :module:cleanTest. If that is necessary that's a major bug
in the build.

Kenn

On Fri, Apr 13, 2018 at 11:46 PM Romain Manni-Bucau 
wrote:

> There is a fake module xxx_test which should have the right classpath but
> since idea compilation is messed up you will still have to run
> :module:cleanTest :module:test --tests org...MyTest.myMethod, even with
> idea which leads to the same latency as the command line :(
>
> Le 13 avr. 2018 22:23, "Eugene Kirpichov"  a écrit :
>
>> While we're talking about running tests in IntelliJ with Gradle...
>> Anybody got advice on how to run a single NeedsRunner test in
>> sdks-java-core, say, ParDoTest? With Maven, I used to just run the test in
>> IntelliJ and specify "runners-direct-java" as the classpath; with Gradle,
>> the best I got is to manually run the direct runner's needsRunnerTests task
>> with specifying --tests=..., but it takes a long time, and IntelliJ treats
>> that as just a gradle task run, not as a test run.
>>
>> On Fri, Apr 13, 2018 at 11:14 AM Reuven Lax  wrote:
>>
>>> Is there a Jira for this 3 second delay? Also you're initial complaint
>>> was not about the 3 second delay, so it wasn't clear that's what you were
>>> complaining about.
>>>
>>> Reuven
>>>
>>> On Fri, Apr 13, 2018 at 4:42 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 When you launch a test with gradle runner it launches gradle which
 makes loose 3s on a very fast computer and more on a slower (6 on my
 personal one which is already fast but not as much as my work one). We are
 5 to see that regression at least. So there is a reason to not use the
 gradle runner if possible cause when you work and need to debug you are
 just stucked (that is why i switched back to mvn after 15mn, i was loosing
 to much time).

 Switching back to native idea test run would fix it but tests just dont
 work this way for me whatever setup i do :( - missing resources IIRC in
 idea out dir.

 Le 13 avr. 2018 00:07, "Reuven Lax"  a écrit :

 I also don't quite understand what your question is, and it appears
 like Dan spent considerable time trying to reproduce your issue. For the
 record, I have had no issues running tests via Gradle in IntelliJ for the
 past few weeks.

 Reuven

 On Thu, Apr 12, 2018 at 9:47 PM Daniel Oliveira 
 wrote:

> Sorry Romain, I'm not quite sure what you're asking. Can you clarify?
>
> On Thu, Apr 12, 2018 at 12:22 PM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Well you are the only one to not have the drawbacks to use it so
>> maybe dont do it? I know Luke is in holidays but anyone else with the
>> knowledge of why we nees that noise compared to idea native tooling/flow?
>>
>> Le 12 avr. 2018 20:16, "Daniel Oliveira"  a
>> écrit :
>>
>>> Ah, I did not. Thanks Romain.
>>>
>>> I tried it again, restarting in between, and still had no
>>> differences. Since it seems like there's no reason not to use "Gradle 
>>> Test
>>> Runner", I'll mention it in the contributor's guide.
>>>
>>> On Thu, Apr 12, 2018 at 10:31 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 @Daniel: did you restart in between? Otherwise it does nothing. One
 launches JunitCoreRunner from idea and the other a gradle command.

 Le 12 avr. 2018 19:24, "Daniel Oliveira" 
 a écrit :

> I think it depends on what exactly switching to "Gradle Test
> Runner" from "Platform Test Runner" does. I tried it out on my 
> machine and
> they seem to act identically to each other. The IntelliJ 
> documentation says
> it determines what API to use to run the tests
> , so maybe it's
> usefulness depends on the user's machine, in which case a note about 
> that
> would be useful. Something like: "If your IDE has trouble running 
> tests via
> IDEA shortcuts, try the following steps: [...]"
>
> On Thu, Apr 12, 2018 at 3:29 AM Alexey Romanenko <
> aromanenko@gmail.com> wrote:
>
>> Daniel, actually I did run it with default IDEA JUnit test
>> runner. Then, in “Settings > Build, Execution, Deployment > Build 
>> Tools >
>> Gradle > Runner" I selected “Gradle Test Runner” in “Run tests using”
>> selectbox and it works ok when I run my tests with IDEA shortcuts. 
>> So,
>> probably, we should add this details on
>> https://beam.apache.org/contribute/intellij/ too.
>> What do you think?
>>
>> WBR,
>> Alexey
>>
>> 

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

There is a fake module xxx_test which should have the right classpath but
since idea compilation is messed up you will still have to run
:module:cleanTest :module:test --tests org...MyTest.myMethod, even with
idea which leads to the same latency as the command line :(

Le 13 avr. 2018 22:23, "Eugene Kirpichov"  a écrit :

> While we're talking about running tests in IntelliJ with Gradle...
> Anybody got advice on how to run a single NeedsRunner test in
> sdks-java-core, say, ParDoTest? With Maven, I used to just run the test in
> IntelliJ and specify "runners-direct-java" as the classpath; with Gradle,
> the best I got is to manually run the direct runner's needsRunnerTests task
> with specifying --tests=..., but it takes a long time, and IntelliJ treats
> that as just a gradle task run, not as a test run.
>
> On Fri, Apr 13, 2018 at 11:14 AM Reuven Lax  wrote:
>
>> Is there a Jira for this 3 second delay? Also you're initial complaint
>> was not about the 3 second delay, so it wasn't clear that's what you were
>> complaining about.
>>
>> Reuven
>>
>> On Fri, Apr 13, 2018 at 4:42 AM Romain Manni-Bucau 
>> wrote:
>>
>>> When you launch a test with gradle runner it launches gradle which makes
>>> loose 3s on a very fast computer and more on a slower (6 on my personal one
>>> which is already fast but not as much as my work one). We are 5 to see that
>>> regression at least. So there is a reason to not use the gradle runner if
>>> possible cause when you work and need to debug you are just stucked (that
>>> is why i switched back to mvn after 15mn, i was loosing to much time).
>>>
>>> Switching back to native idea test run would fix it but tests just dont
>>> work this way for me whatever setup i do :( - missing resources IIRC in
>>> idea out dir.
>>>
>>> Le 13 avr. 2018 00:07, "Reuven Lax"  a écrit :
>>>
>>> I also don't quite understand what your question is, and it appears like
>>> Dan spent considerable time trying to reproduce your issue. For the record,
>>> I have had no issues running tests via Gradle in IntelliJ for the past few
>>> weeks.
>>>
>>> Reuven
>>>
>>> On Thu, Apr 12, 2018 at 9:47 PM Daniel Oliveira 
>>> wrote:
>>>
 Sorry Romain, I'm not quite sure what you're asking. Can you clarify?

 On Thu, Apr 12, 2018 at 12:22 PM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Well you are the only one to not have the drawbacks to use it so maybe
> dont do it? I know Luke is in holidays but anyone else with the knowledge
> of why we nees that noise compared to idea native tooling/flow?
>
> Le 12 avr. 2018 20:16, "Daniel Oliveira"  a
> écrit :
>
>> Ah, I did not. Thanks Romain.
>>
>> I tried it again, restarting in between, and still had no
>> differences. Since it seems like there's no reason not to use "Gradle 
>> Test
>> Runner", I'll mention it in the contributor's guide.
>>
>> On Thu, Apr 12, 2018 at 10:31 AM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> @Daniel: did you restart in between? Otherwise it does nothing. One
>>> launches JunitCoreRunner from idea and the other a gradle command.
>>>
>>> Le 12 avr. 2018 19:24, "Daniel Oliveira"  a
>>> écrit :
>>>
 I think it depends on what exactly switching to "Gradle Test
 Runner" from "Platform Test Runner" does. I tried it out on my machine 
 and
 they seem to act identically to each other. The IntelliJ documentation 
 says
 it determines what API to use to run the tests
 , so maybe it's
 usefulness depends on the user's machine, in which case a note about 
 that
 would be useful. Something like: "If your IDE has trouble running 
 tests via
 IDEA shortcuts, try the following steps: [...]"

 On Thu, Apr 12, 2018 at 3:29 AM Alexey Romanenko <
 aromanenko@gmail.com> wrote:

> Daniel, actually I did run it with default IDEA JUnit test runner.
> Then, in “Settings > Build, Execution, Deployment > Build Tools > 
> Gradle >
> Runner" I selected “Gradle Test Runner” in “Run tests using” 
> selectbox and
> it works ok when I run my tests with IDEA shortcuts. So, probably, we
> should add this details on https://beam.apache.org/
> contribute/intellij/ too.
> What do you think?
>
> WBR,
> Alexey
>
> On 11 Apr 2018, at 21:17, Daniel Oliveira 
> wrote:
>
> Alexey, are you referring to tests run with "./gradlew
> :beam-runners-direct-java:needsRunnerTests"? That command works
> fine for me in both versions of IDEA, but I believe 

Re: Gradle Status [April 6]

[Reuven Lax]

Is there a Jira for this 3 second delay? Also you're initial complaint was
not about the 3 second delay, so it wasn't clear that's what you were
complaining about.

Reuven

On Fri, Apr 13, 2018 at 4:42 AM Romain Manni-Bucau 
wrote:

> When you launch a test with gradle runner it launches gradle which makes
> loose 3s on a very fast computer and more on a slower (6 on my personal one
> which is already fast but not as much as my work one). We are 5 to see that
> regression at least. So there is a reason to not use the gradle runner if
> possible cause when you work and need to debug you are just stucked (that
> is why i switched back to mvn after 15mn, i was loosing to much time).
>
> Switching back to native idea test run would fix it but tests just dont
> work this way for me whatever setup i do :( - missing resources IIRC in
> idea out dir.
>
> Le 13 avr. 2018 00:07, "Reuven Lax"  a écrit :
>
> I also don't quite understand what your question is, and it appears like
> Dan spent considerable time trying to reproduce your issue. For the record,
> I have had no issues running tests via Gradle in IntelliJ for the past few
> weeks.
>
> Reuven
>
> On Thu, Apr 12, 2018 at 9:47 PM Daniel Oliveira 
> wrote:
>
>> Sorry Romain, I'm not quite sure what you're asking. Can you clarify?
>>
>> On Thu, Apr 12, 2018 at 12:22 PM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Well you are the only one to not have the drawbacks to use it so maybe
>>> dont do it? I know Luke is in holidays but anyone else with the knowledge
>>> of why we nees that noise compared to idea native tooling/flow?
>>>
>>> Le 12 avr. 2018 20:16, "Daniel Oliveira"  a
>>> écrit :
>>>
 Ah, I did not. Thanks Romain.

 I tried it again, restarting in between, and still had no differences.
 Since it seems like there's no reason not to use "Gradle Test Runner", I'll
 mention it in the contributor's guide.

 On Thu, Apr 12, 2018 at 10:31 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> @Daniel: did you restart in between? Otherwise it does nothing. One
> launches JunitCoreRunner from idea and the other a gradle command.
>
> Le 12 avr. 2018 19:24, "Daniel Oliveira"  a
> écrit :
>
>> I think it depends on what exactly switching to "Gradle Test Runner"
>> from "Platform Test Runner" does. I tried it out on my machine and they
>> seem to act identically to each other. The IntelliJ documentation says it
>> determines what API to use to run the tests
>> , so maybe it's
>> usefulness depends on the user's machine, in which case a note about that
>> would be useful. Something like: "If your IDE has trouble running tests 
>> via
>> IDEA shortcuts, try the following steps: [...]"
>>
>> On Thu, Apr 12, 2018 at 3:29 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> Daniel, actually I did run it with default IDEA JUnit test runner.
>>> Then, in “Settings > Build, Execution, Deployment > Build Tools > 
>>> Gradle >
>>> Runner" I selected “Gradle Test Runner” in “Run tests using” selectbox 
>>> and
>>> it works ok when I run my tests with IDEA shortcuts. So, probably, we
>>> should add this details on
>>> https://beam.apache.org/contribute/intellij/ too.
>>> What do you think?
>>>
>>> WBR,
>>> Alexey
>>>
>>> On 11 Apr 2018, at 21:17, Daniel Oliveira 
>>> wrote:
>>>
>>> Alexey, are you referring to tests run with "./gradlew
>>> :beam-runners-direct-java:needsRunnerTests"? That command works fine 
>>> for me
>>> in both versions of IDEA, but I believe the same tests fail if you run 
>>> them
>>> directly through "./gradlew test".
>>>
>>> However, I am having issues with a bunch of validatesRunner tests,
>>> mostly be caused by
>>> :beam-runners-google-cloud-dataflow-java:validatesRunner. Not sure if 
>>> it's
>>> because of a code change or a gradle config, I'll keep looking into it.
>>>
>>> On Wed, Apr 11, 2018 at 11:01 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 I got tests running rrconfiguring gradle (which was setup for
 another project but seems beam didnt like it) but latency is still 
 "high"
 using gradle runner for tests (like Etienne said ~3s on an i7 with 16G 
 vs a
 few ms with default idea test runner, would be great to solve that).

 I also find the integration quite fishy cause configurations are
 customs so idea is kind of forced to propose your modukes 3 times at 
 least
 when you select the classpath (x_test being generally the working one).

 Also the false positive you get if 

Re: Gradle Status [April 6]

[Daniel Oliveira]

Ah, I see. I was attempting to replicate Alexey's issues with DirectRunner
tests, so I wasn't checking for that 3s overhead. A quick test for me shows
that I also get that several second overhead when running tests with
Gradle, ranging from 3-6 sec. However, on my machine this delay is
constant, regardless of whether I use the Gradle Runner or Platform Runner.

I think getting more info from someone more knowledgeable about Gradle is a
good idea; I'm still very new to Gradle, which is why my ability to help
with the Gradle effort is mainly limited to helping with documentation.

On Thu, Apr 12, 2018 at 9:41 PM Romain Manni-Bucau 
wrote:

> When you launch a test with gradle runner it launches gradle which makes
> loose 3s on a very fast computer and more on a slower (6 on my personal one
> which is already fast but not as much as my work one). We are 5 to see that
> regression at least. So there is a reason to not use the gradle runner if
> possible cause when you work and need to debug you are just stucked (that
> is why i switched back to mvn after 15mn, i was loosing to much time).
>
> Switching back to native idea test run would fix it but tests just dont
> work this way for me whatever setup i do :( - missing resources IIRC in
> idea out dir.
>
> Le 13 avr. 2018 00:07, "Reuven Lax"  a écrit :
>
> I also don't quite understand what your question is, and it appears like
> Dan spent considerable time trying to reproduce your issue. For the record,
> I have had no issues running tests via Gradle in IntelliJ for the past few
> weeks.
>
> Reuven
>
> On Thu, Apr 12, 2018 at 9:47 PM Daniel Oliveira 
> wrote:
>
>> Sorry Romain, I'm not quite sure what you're asking. Can you clarify?
>>
>> On Thu, Apr 12, 2018 at 12:22 PM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Well you are the only one to not have the drawbacks to use it so maybe
>>> dont do it? I know Luke is in holidays but anyone else with the knowledge
>>> of why we nees that noise compared to idea native tooling/flow?
>>>
>>> Le 12 avr. 2018 20:16, "Daniel Oliveira"  a
>>> écrit :
>>>
 Ah, I did not. Thanks Romain.

 I tried it again, restarting in between, and still had no differences.
 Since it seems like there's no reason not to use "Gradle Test Runner", I'll
 mention it in the contributor's guide.

 On Thu, Apr 12, 2018 at 10:31 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> @Daniel: did you restart in between? Otherwise it does nothing. One
> launches JunitCoreRunner from idea and the other a gradle command.
>
> Le 12 avr. 2018 19:24, "Daniel Oliveira"  a
> écrit :
>
>> I think it depends on what exactly switching to "Gradle Test Runner"
>> from "Platform Test Runner" does. I tried it out on my machine and they
>> seem to act identically to each other. The IntelliJ documentation says it
>> determines what API to use to run the tests
>> , so maybe it's
>> usefulness depends on the user's machine, in which case a note about that
>> would be useful. Something like: "If your IDE has trouble running tests 
>> via
>> IDEA shortcuts, try the following steps: [...]"
>>
>> On Thu, Apr 12, 2018 at 3:29 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> Daniel, actually I did run it with default IDEA JUnit test runner.
>>> Then, in “Settings > Build, Execution, Deployment > Build Tools > 
>>> Gradle >
>>> Runner" I selected “Gradle Test Runner” in “Run tests using” selectbox 
>>> and
>>> it works ok when I run my tests with IDEA shortcuts. So, probably, we
>>> should add this details on
>>> https://beam.apache.org/contribute/intellij/ too.
>>> What do you think?
>>>
>>> WBR,
>>> Alexey
>>>
>>> On 11 Apr 2018, at 21:17, Daniel Oliveira 
>>> wrote:
>>>
>>> Alexey, are you referring to tests run with "./gradlew
>>> :beam-runners-direct-java:needsRunnerTests"? That command works fine 
>>> for me
>>> in both versions of IDEA, but I believe the same tests fail if you run 
>>> them
>>> directly through "./gradlew test".
>>>
>>> However, I am having issues with a bunch of validatesRunner tests,
>>> mostly be caused by
>>> :beam-runners-google-cloud-dataflow-java:validatesRunner. Not sure if 
>>> it's
>>> because of a code change or a gradle config, I'll keep looking into it.
>>>
>>> On Wed, Apr 11, 2018 at 11:01 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 I got tests running rrconfiguring gradle (which was setup for
 another project but seems beam didnt like it) but latency is still 
 "high"
 using gradle runner for tests (like Etienne said ~3s on an i7 

Re: Gradle Status [April 6]

[Reuven Lax]

I also don't quite understand what your question is, and it appears like
Dan spent considerable time trying to reproduce your issue. For the record,
I have had no issues running tests via Gradle in IntelliJ for the past few
weeks.

Reuven

On Thu, Apr 12, 2018 at 9:47 PM Daniel Oliveira 
wrote:

> Sorry Romain, I'm not quite sure what you're asking. Can you clarify?
>
> On Thu, Apr 12, 2018 at 12:22 PM Romain Manni-Bucau 
> wrote:
>
>> Well you are the only one to not have the drawbacks to use it so maybe
>> dont do it? I know Luke is in holidays but anyone else with the knowledge
>> of why we nees that noise compared to idea native tooling/flow?
>>
>> Le 12 avr. 2018 20:16, "Daniel Oliveira"  a
>> écrit :
>>
>>> Ah, I did not. Thanks Romain.
>>>
>>> I tried it again, restarting in between, and still had no differences.
>>> Since it seems like there's no reason not to use "Gradle Test Runner", I'll
>>> mention it in the contributor's guide.
>>>
>>> On Thu, Apr 12, 2018 at 10:31 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 @Daniel: did you restart in between? Otherwise it does nothing. One
 launches JunitCoreRunner from idea and the other a gradle command.

 Le 12 avr. 2018 19:24, "Daniel Oliveira"  a
 écrit :

> I think it depends on what exactly switching to "Gradle Test Runner"
> from "Platform Test Runner" does. I tried it out on my machine and they
> seem to act identically to each other. The IntelliJ documentation says it
> determines what API to use to run the tests
> , so maybe it's
> usefulness depends on the user's machine, in which case a note about that
> would be useful. Something like: "If your IDE has trouble running tests 
> via
> IDEA shortcuts, try the following steps: [...]"
>
> On Thu, Apr 12, 2018 at 3:29 AM Alexey Romanenko <
> aromanenko@gmail.com> wrote:
>
>> Daniel, actually I did run it with default IDEA JUnit test runner.
>> Then, in “Settings > Build, Execution, Deployment > Build Tools > Gradle 
>> >
>> Runner" I selected “Gradle Test Runner” in “Run tests using” selectbox 
>> and
>> it works ok when I run my tests with IDEA shortcuts. So, probably, we
>> should add this details on
>> https://beam.apache.org/contribute/intellij/ too.
>> What do you think?
>>
>> WBR,
>> Alexey
>>
>> On 11 Apr 2018, at 21:17, Daniel Oliveira 
>> wrote:
>>
>> Alexey, are you referring to tests run with "./gradlew
>> :beam-runners-direct-java:needsRunnerTests"? That command works fine for 
>> me
>> in both versions of IDEA, but I believe the same tests fail if you run 
>> them
>> directly through "./gradlew test".
>>
>> However, I am having issues with a bunch of validatesRunner tests,
>> mostly be caused by
>> :beam-runners-google-cloud-dataflow-java:validatesRunner. Not sure if 
>> it's
>> because of a code change or a gradle config, I'll keep looking into it.
>>
>> On Wed, Apr 11, 2018 at 11:01 AM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> I got tests running rrconfiguring gradle (which was setup for
>>> another project but seems beam didnt like it) but latency is still 
>>> "high"
>>> using gradle runner for tests (like Etienne said ~3s on an i7 with 16G 
>>> vs a
>>> few ms with default idea test runner, would be great to solve that).
>>>
>>> I also find the integration quite fishy cause configurations are
>>> customs so idea is kind of forced to propose your modukes 3 times at 
>>> least
>>> when you select the classpath (x_test being generally the working one).
>>>
>>> Also the false positive you get if you forget a cleanX is a bit
>>> annoying, maybe we should force a clean for test or at least when there 
>>> is
>>> a --tests to avoid gradle to not run it cause there is no diff.
>>>
>>> So it works but dev productivity is reduced a lot and it became slow
>>> enough to think if you should do a contribution or not - at least for 
>>> me.
>>>
>>> Le 11 avr. 2018 19:37, "Alexey Romanenko" 
>>> a écrit :
>>>
 I’ve managed to import a project as it’s described in documentation
 (starting from empty project) using Idea 2018 and run unit tests
 successfully.
 For some reasons, tests, that use DirectRunner to run a pipeline,
 were failed.

 WBR,
 Alexey

 On 11 Apr 2018, at 19:01, Daniel Oliveira 
 wrote:

 Hi everyone, I was the one who initially wrote the PR with Idea
 instructions . I was

Re: Gradle Status [April 6]

[Daniel Oliveira]

Sorry Romain, I'm not quite sure what you're asking. Can you clarify?

On Thu, Apr 12, 2018 at 12:22 PM Romain Manni-Bucau 
wrote:

> Well you are the only one to not have the drawbacks to use it so maybe
> dont do it? I know Luke is in holidays but anyone else with the knowledge
> of why we nees that noise compared to idea native tooling/flow?
>
> Le 12 avr. 2018 20:16, "Daniel Oliveira"  a
> écrit :
>
>> Ah, I did not. Thanks Romain.
>>
>> I tried it again, restarting in between, and still had no differences.
>> Since it seems like there's no reason not to use "Gradle Test Runner", I'll
>> mention it in the contributor's guide.
>>
>> On Thu, Apr 12, 2018 at 10:31 AM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> @Daniel: did you restart in between? Otherwise it does nothing. One
>>> launches JunitCoreRunner from idea and the other a gradle command.
>>>
>>> Le 12 avr. 2018 19:24, "Daniel Oliveira"  a
>>> écrit :
>>>
 I think it depends on what exactly switching to "Gradle Test Runner"
 from "Platform Test Runner" does. I tried it out on my machine and they
 seem to act identically to each other. The IntelliJ documentation says it
 determines what API to use to run the tests
 , so maybe it's
 usefulness depends on the user's machine, in which case a note about that
 would be useful. Something like: "If your IDE has trouble running tests via
 IDEA shortcuts, try the following steps: [...]"

 On Thu, Apr 12, 2018 at 3:29 AM Alexey Romanenko <
 aromanenko@gmail.com> wrote:

> Daniel, actually I did run it with default IDEA JUnit test runner.
> Then, in “Settings > Build, Execution, Deployment > Build Tools > Gradle >
> Runner" I selected “Gradle Test Runner” in “Run tests using” selectbox and
> it works ok when I run my tests with IDEA shortcuts. So, probably, we
> should add this details on
> https://beam.apache.org/contribute/intellij/ too.
> What do you think?
>
> WBR,
> Alexey
>
> On 11 Apr 2018, at 21:17, Daniel Oliveira 
> wrote:
>
> Alexey, are you referring to tests run with "./gradlew
> :beam-runners-direct-java:needsRunnerTests"? That command works fine for 
> me
> in both versions of IDEA, but I believe the same tests fail if you run 
> them
> directly through "./gradlew test".
>
> However, I am having issues with a bunch of validatesRunner tests,
> mostly be caused by
> :beam-runners-google-cloud-dataflow-java:validatesRunner. Not sure if it's
> because of a code change or a gradle config, I'll keep looking into it.
>
> On Wed, Apr 11, 2018 at 11:01 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> I got tests running rrconfiguring gradle (which was setup for another
>> project but seems beam didnt like it) but latency is still "high" using
>> gradle runner for tests (like Etienne said ~3s on an i7 with 16G vs a few
>> ms with default idea test runner, would be great to solve that).
>>
>> I also find the integration quite fishy cause configurations are
>> customs so idea is kind of forced to propose your modukes 3 times at 
>> least
>> when you select the classpath (x_test being generally the working one).
>>
>> Also the false positive you get if you forget a cleanX is a bit
>> annoying, maybe we should force a clean for test or at least when there 
>> is
>> a --tests to avoid gradle to not run it cause there is no diff.
>>
>> So it works but dev productivity is reduced a lot and it became slow
>> enough to think if you should do a contribution or not - at least for me.
>>
>> Le 11 avr. 2018 19:37, "Alexey Romanenko" 
>> a écrit :
>>
>>> I’ve managed to import a project as it’s described in documentation
>>> (starting from empty project) using Idea 2018 and run unit tests
>>> successfully.
>>> For some reasons, tests, that use DirectRunner to run a pipeline,
>>> were failed.
>>>
>>> WBR,
>>> Alexey
>>>
>>> On 11 Apr 2018, at 19:01, Daniel Oliveira 
>>> wrote:
>>>
>>> Hi everyone, I was the one who initially wrote the PR with Idea
>>> instructions . I was
>>> using 2017.3 as well while writing it so all the instructions were 
>>> tested
>>> on that version. I'll try testing the instructions on 2018 to see if I 
>>> can
>>> reproduce the issues people are having.
>>>
>>> On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik 
>>> wrote:
>>>
 I use 2017.3 and it has been reliable for me. I haven't tried 2018
 yet.

 On Wed, Apr 11, 2018 at 11:30 AM Romain 

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

Well you are the only one to not have the drawbacks to use it so maybe dont
do it? I know Luke is in holidays but anyone else with the knowledge of why
we nees that noise compared to idea native tooling/flow?

Le 12 avr. 2018 20:16, "Daniel Oliveira"  a écrit :

> Ah, I did not. Thanks Romain.
>
> I tried it again, restarting in between, and still had no differences.
> Since it seems like there's no reason not to use "Gradle Test Runner", I'll
> mention it in the contributor's guide.
>
> On Thu, Apr 12, 2018 at 10:31 AM Romain Manni-Bucau 
> wrote:
>
>> @Daniel: did you restart in between? Otherwise it does nothing. One
>> launches JunitCoreRunner from idea and the other a gradle command.
>>
>> Le 12 avr. 2018 19:24, "Daniel Oliveira"  a
>> écrit :
>>
>>> I think it depends on what exactly switching to "Gradle Test Runner"
>>> from "Platform Test Runner" does. I tried it out on my machine and they
>>> seem to act identically to each other. The IntelliJ documentation says it
>>> determines what API to use to run the tests
>>> , so maybe it's
>>> usefulness depends on the user's machine, in which case a note about that
>>> would be useful. Something like: "If your IDE has trouble running tests via
>>> IDEA shortcuts, try the following steps: [...]"
>>>
>>> On Thu, Apr 12, 2018 at 3:29 AM Alexey Romanenko <
>>> aromanenko@gmail.com> wrote:
>>>
 Daniel, actually I did run it with default IDEA JUnit test runner.
 Then, in “Settings > Build, Execution, Deployment > Build Tools > Gradle >
 Runner" I selected “Gradle Test Runner” in “Run tests using” selectbox and
 it works ok when I run my tests with IDEA shortcuts. So, probably, we
 should add this details on https://beam.apache.org/contribute/intellij/
  too.
 What do you think?

 WBR,
 Alexey

 On 11 Apr 2018, at 21:17, Daniel Oliveira 
 wrote:

 Alexey, are you referring to tests run with "./gradlew
 :beam-runners-direct-java:needsRunnerTests"? That command works fine
 for me in both versions of IDEA, but I believe the same tests fail if you
 run them directly through "./gradlew test".

 However, I am having issues with a bunch of validatesRunner tests,
 mostly be caused by 
 :beam-runners-google-cloud-dataflow-java:validatesRunner.
 Not sure if it's because of a code change or a gradle config, I'll keep
 looking into it.

 On Wed, Apr 11, 2018 at 11:01 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> I got tests running rrconfiguring gradle (which was setup for another
> project but seems beam didnt like it) but latency is still "high" using
> gradle runner for tests (like Etienne said ~3s on an i7 with 16G vs a few
> ms with default idea test runner, would be great to solve that).
>
> I also find the integration quite fishy cause configurations are
> customs so idea is kind of forced to propose your modukes 3 times at least
> when you select the classpath (x_test being generally the working one).
>
> Also the false positive you get if you forget a cleanX is a bit
> annoying, maybe we should force a clean for test or at least when there is
> a --tests to avoid gradle to not run it cause there is no diff.
>
> So it works but dev productivity is reduced a lot and it became slow
> enough to think if you should do a contribution or not - at least for me.
>
> Le 11 avr. 2018 19:37, "Alexey Romanenko" 
> a écrit :
>
>> I’ve managed to import a project as it’s described in documentation
>> (starting from empty project) using Idea 2018 and run unit tests
>> successfully.
>> For some reasons, tests, that use DirectRunner to run a pipeline,
>> were failed.
>>
>> WBR,
>> Alexey
>>
>> On 11 Apr 2018, at 19:01, Daniel Oliveira 
>> wrote:
>>
>> Hi everyone, I was the one who initially wrote the PR with Idea
>> instructions . I was
>> using 2017.3 as well while writing it so all the instructions were tested
>> on that version. I'll try testing the instructions on 2018 to see if I 
>> can
>> reproduce the issues people are having.
>>
>> On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik  wrote:
>>
>>> I use 2017.3 and it has been reliable for me. I haven't tried 2018
>>> yet.
>>>
>>> On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Any of you using the idea 2018? the import works for me but then it
 is
 not as smooth as it seems for you. I'm just trying to see if it is a
 procedure thing or a version issue.

 Romain Manni-Bucau

Re: Gradle Status [April 6]

[Daniel Oliveira]

Ah, I did not. Thanks Romain.

I tried it again, restarting in between, and still had no differences.
Since it seems like there's no reason not to use "Gradle Test Runner", I'll
mention it in the contributor's guide.

On Thu, Apr 12, 2018 at 10:31 AM Romain Manni-Bucau 
wrote:

> @Daniel: did you restart in between? Otherwise it does nothing. One
> launches JunitCoreRunner from idea and the other a gradle command.
>
> Le 12 avr. 2018 19:24, "Daniel Oliveira"  a
> écrit :
>
>> I think it depends on what exactly switching to "Gradle Test Runner" from
>> "Platform Test Runner" does. I tried it out on my machine and they seem to
>> act identically to each other. The IntelliJ documentation says it
>> determines what API to use to run the tests
>> , so maybe it's
>> usefulness depends on the user's machine, in which case a note about that
>> would be useful. Something like: "If your IDE has trouble running tests via
>> IDEA shortcuts, try the following steps: [...]"
>>
>> On Thu, Apr 12, 2018 at 3:29 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> Daniel, actually I did run it with default IDEA JUnit test runner. Then,
>>> in “Settings > Build, Execution, Deployment > Build Tools > Gradle >
>>> Runner" I selected “Gradle Test Runner” in “Run tests using” selectbox and
>>> it works ok when I run my tests with IDEA shortcuts. So, probably, we
>>> should add this details on https://beam.apache.org/contribute/intellij/
>>>  too.
>>> What do you think?
>>>
>>> WBR,
>>> Alexey
>>>
>>> On 11 Apr 2018, at 21:17, Daniel Oliveira 
>>> wrote:
>>>
>>> Alexey, are you referring to tests run with "./gradlew
>>> :beam-runners-direct-java:needsRunnerTests"? That command works fine for me
>>> in both versions of IDEA, but I believe the same tests fail if you run them
>>> directly through "./gradlew test".
>>>
>>> However, I am having issues with a bunch of validatesRunner tests,
>>> mostly be caused by
>>> :beam-runners-google-cloud-dataflow-java:validatesRunner. Not sure if it's
>>> because of a code change or a gradle config, I'll keep looking into it.
>>>
>>> On Wed, Apr 11, 2018 at 11:01 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 I got tests running rrconfiguring gradle (which was setup for another
 project but seems beam didnt like it) but latency is still "high" using
 gradle runner for tests (like Etienne said ~3s on an i7 with 16G vs a few
 ms with default idea test runner, would be great to solve that).

 I also find the integration quite fishy cause configurations are
 customs so idea is kind of forced to propose your modukes 3 times at least
 when you select the classpath (x_test being generally the working one).

 Also the false positive you get if you forget a cleanX is a bit
 annoying, maybe we should force a clean for test or at least when there is
 a --tests to avoid gradle to not run it cause there is no diff.

 So it works but dev productivity is reduced a lot and it became slow
 enough to think if you should do a contribution or not - at least for me.

 Le 11 avr. 2018 19:37, "Alexey Romanenko"  a
 écrit :

> I’ve managed to import a project as it’s described in documentation
> (starting from empty project) using Idea 2018 and run unit tests
> successfully.
> For some reasons, tests, that use DirectRunner to run a pipeline, were
> failed.
>
> WBR,
> Alexey
>
> On 11 Apr 2018, at 19:01, Daniel Oliveira 
> wrote:
>
> Hi everyone, I was the one who initially wrote the PR with Idea
> instructions . I was
> using 2017.3 as well while writing it so all the instructions were tested
> on that version. I'll try testing the instructions on 2018 to see if I can
> reproduce the issues people are having.
>
> On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik  wrote:
>
>> I use 2017.3 and it has been reliable for me. I haven't tried 2018
>> yet.
>>
>> On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Any of you using the idea 2018? the import works for me but then it
>>> is
>>> not as smooth as it seems for you. I'm just trying to see if it is a
>>> procedure thing or a version issue.
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>
>>>
>>> 2018-04-11 17:28 GMT+02:00 Kenneth Knowles :
>>> > The only reason I did "empty project then add a module" procedure
>>> was to get
>>> > all the IntelliJ files outside the source tree. IIRC directly
>>> creating from
>>> > existing sources didn't give the necessary configuration options.

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

@Daniel: did you restart in between? Otherwise it does nothing. One
launches JunitCoreRunner from idea and the other a gradle command.

Le 12 avr. 2018 19:24, "Daniel Oliveira"  a écrit :

> I think it depends on what exactly switching to "Gradle Test Runner" from
> "Platform Test Runner" does. I tried it out on my machine and they seem to
> act identically to each other. The IntelliJ documentation says it
> determines what API to use to run the tests
> , so maybe it's
> usefulness depends on the user's machine, in which case a note about that
> would be useful. Something like: "If your IDE has trouble running tests via
> IDEA shortcuts, try the following steps: [...]"
>
> On Thu, Apr 12, 2018 at 3:29 AM Alexey Romanenko 
> wrote:
>
>> Daniel, actually I did run it with default IDEA JUnit test runner. Then,
>> in “Settings > Build, Execution, Deployment > Build Tools > Gradle >
>> Runner" I selected “Gradle Test Runner” in “Run tests using” selectbox and
>> it works ok when I run my tests with IDEA shortcuts. So, probably, we
>> should add this details on https://beam.apache.org/contribute/intellij/
>>  too.
>> What do you think?
>>
>> WBR,
>> Alexey
>>
>> On 11 Apr 2018, at 21:17, Daniel Oliveira  wrote:
>>
>> Alexey, are you referring to tests run with "./gradlew
>> :beam-runners-direct-java:needsRunnerTests"? That command works fine for
>> me in both versions of IDEA, but I believe the same tests fail if you run
>> them directly through "./gradlew test".
>>
>> However, I am having issues with a bunch of validatesRunner tests, mostly
>> be caused by :beam-runners-google-cloud-dataflow-java:validatesRunner.
>> Not sure if it's because of a code change or a gradle config, I'll keep
>> looking into it.
>>
>> On Wed, Apr 11, 2018 at 11:01 AM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> I got tests running rrconfiguring gradle (which was setup for another
>>> project but seems beam didnt like it) but latency is still "high" using
>>> gradle runner for tests (like Etienne said ~3s on an i7 with 16G vs a few
>>> ms with default idea test runner, would be great to solve that).
>>>
>>> I also find the integration quite fishy cause configurations are customs
>>> so idea is kind of forced to propose your modukes 3 times at least when you
>>> select the classpath (x_test being generally the working one).
>>>
>>> Also the false positive you get if you forget a cleanX is a bit
>>> annoying, maybe we should force a clean for test or at least when there is
>>> a --tests to avoid gradle to not run it cause there is no diff.
>>>
>>> So it works but dev productivity is reduced a lot and it became slow
>>> enough to think if you should do a contribution or not - at least for me.
>>>
>>> Le 11 avr. 2018 19:37, "Alexey Romanenko"  a
>>> écrit :
>>>
 I’ve managed to import a project as it’s described in documentation
 (starting from empty project) using Idea 2018 and run unit tests
 successfully.
 For some reasons, tests, that use DirectRunner to run a pipeline, were
 failed.

 WBR,
 Alexey

 On 11 Apr 2018, at 19:01, Daniel Oliveira 
 wrote:

 Hi everyone, I was the one who initially wrote the PR with Idea
 instructions . I was
 using 2017.3 as well while writing it so all the instructions were tested
 on that version. I'll try testing the instructions on 2018 to see if I can
 reproduce the issues people are having.

 On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik  wrote:

> I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet.
>
> On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Any of you using the idea 2018? the import works for me but then it is
>> not as smooth as it seems for you. I'm just trying to see if it is a
>> procedure thing or a version issue.
>>
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>
>>
>> 2018-04-11 17:28 GMT+02:00 Kenneth Knowles :
>> > The only reason I did "empty project then add a module" procedure
>> was to get
>> > all the IntelliJ files outside the source tree. IIRC directly
>> creating from
>> > existing sources didn't give the necessary configuration options.
>> If you
>> > don't care about being able to `git clean` then you can do the
>> shorter
>> > version. Also the particular UI for creation might have improved
>> since I
>> > tried it. I'll do it again.
>> >
>> > On the pom, since it is only generated for machine reading, why
>> care if the
>> > parent is inlined? I actually prefer to avoid coupling with things
>> that you
>> 

Re: Gradle Status [April 6]

[Daniel Oliveira]

I think it depends on what exactly switching to "Gradle Test Runner" from
"Platform Test Runner" does. I tried it out on my machine and they seem to
act identically to each other. The IntelliJ documentation says it
determines what API to use to run the tests
, so maybe it's usefulness
depends on the user's machine, in which case a note about that would be
useful. Something like: "If your IDE has trouble running tests via IDEA
shortcuts, try the following steps: [...]"

On Thu, Apr 12, 2018 at 3:29 AM Alexey Romanenko 
wrote:

> Daniel, actually I did run it with default IDEA JUnit test runner. Then,
> in “Settings > Build, Execution, Deployment > Build Tools > Gradle >
> Runner" I selected “Gradle Test Runner” in “Run tests using” selectbox and
> it works ok when I run my tests with IDEA shortcuts. So, probably, we
> should add this details on https://beam.apache.org/contribute/intellij/
>  too.
> What do you think?
>
> WBR,
> Alexey
>
> On 11 Apr 2018, at 21:17, Daniel Oliveira  wrote:
>
> Alexey, are you referring to tests run with "./gradlew
> :beam-runners-direct-java:needsRunnerTests"? That command works fine for me
> in both versions of IDEA, but I believe the same tests fail if you run them
> directly through "./gradlew test".
>
> However, I am having issues with a bunch of validatesRunner tests, mostly
> be caused by :beam-runners-google-cloud-dataflow-java:validatesRunner. Not
> sure if it's because of a code change or a gradle config, I'll keep looking
> into it.
>
> On Wed, Apr 11, 2018 at 11:01 AM Romain Manni-Bucau 
> wrote:
>
>> I got tests running rrconfiguring gradle (which was setup for another
>> project but seems beam didnt like it) but latency is still "high" using
>> gradle runner for tests (like Etienne said ~3s on an i7 with 16G vs a few
>> ms with default idea test runner, would be great to solve that).
>>
>> I also find the integration quite fishy cause configurations are customs
>> so idea is kind of forced to propose your modukes 3 times at least when you
>> select the classpath (x_test being generally the working one).
>>
>> Also the false positive you get if you forget a cleanX is a bit annoying,
>> maybe we should force a clean for test or at least when there is a --tests
>> to avoid gradle to not run it cause there is no diff.
>>
>> So it works but dev productivity is reduced a lot and it became slow
>> enough to think if you should do a contribution or not - at least for me.
>>
>> Le 11 avr. 2018 19:37, "Alexey Romanenko"  a
>> écrit :
>>
>>> I’ve managed to import a project as it’s described in documentation
>>> (starting from empty project) using Idea 2018 and run unit tests
>>> successfully.
>>> For some reasons, tests, that use DirectRunner to run a pipeline, were
>>> failed.
>>>
>>> WBR,
>>> Alexey
>>>
>>> On 11 Apr 2018, at 19:01, Daniel Oliveira 
>>> wrote:
>>>
>>> Hi everyone, I was the one who initially wrote the PR with Idea
>>> instructions . I was
>>> using 2017.3 as well while writing it so all the instructions were tested
>>> on that version. I'll try testing the instructions on 2018 to see if I can
>>> reproduce the issues people are having.
>>>
>>> On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik  wrote:
>>>
 I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet.

 On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Any of you using the idea 2018? the import works for me but then it is
> not as smooth as it seems for you. I'm just trying to see if it is a
> procedure thing or a version issue.
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
>
> 2018-04-11 17:28 GMT+02:00 Kenneth Knowles :
> > The only reason I did "empty project then add a module" procedure
> was to get
> > all the IntelliJ files outside the source tree. IIRC directly
> creating from
> > existing sources didn't give the necessary configuration options. If
> you
> > don't care about being able to `git clean` then you can do the
> shorter
> > version. Also the particular UI for creation might have improved
> since I
> > tried it. I'll do it again.
> >
> > On the pom, since it is only generated for machine reading, why care
> if the
> > parent is inlined? I actually prefer to avoid coupling with things
> that you
> > have to go and look up. Using inheritance is OK for human edited poms
> > (actually IMO it is still a mistake) but it doesn't change the
> semantics of
> > a shipped pom if they are all immutable, which they should be. It is
> better
> > to have all the info right there. Is there an actually effective
> 

Re: Gradle Status [April 11]

[Romain Manni-Bucau]

just created one https://issues.apache.org/jira/browse/BEAM-4057

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-12 18:15 GMT+02:00 Ahmet Altay :
>> Found another blocker in current artifacts creations: there is not
> pom.xml and pom.properties in META-INF. This is used by tools +
> libraries + integrations so it is quite important to not break it
>
> Romain, is there a JIRA for this issues? If not could you create one please?
>
> On Thu, Apr 12, 2018 at 3:17 AM, Etienne Chauchot 
> wrote:
>>
>> Nice !
>> thanks Kenn
>>
>> Le mercredi 11 avril 2018 à 18:21 +, Kenneth Knowles a écrit :
>>
>> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
>> Netty has been introduced since yesterday. Etienne mentioned he has worked
>> toward setting up periodic runs on all runners, so this should help get us
>> towards that. We'll probably prefer to build standalone fat jars for
>> selected runners and use those, which is pending unknown issues in the
>> shadow config leaving out required dependencies.
>>
>> Kenn
>>
>> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner  wrote:
>>
>> Thanks everyone for the continued effort towards the Gradle migration. As
>> a high-level summary of our progress since Friday: we have a viable build,
>> with a number of minor issues that we're still working out. Please take a
>> look at the new documentation in our contribution guide and log any bugs
>> that you find.
>>
>> Here's a more detailed view of improvements from just the past few days..
>>
>> Release artifacts:
>> *  Pom.xml generation logic now in master [1]
>> * Nightly snapshots are now produced using Gradle [2]
>> * Excluded modules propagated to dependencies when generating * pom.xml
>> * Artifact JARs are properly shaded [3]
>> * Working on fixing dependency scopes in generated pom [4]
>> PreCommits / Postcommits:
>> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
>> [7] [8] [9]
>> * Jenkins results now include JUnit test results [10] and build scan for
>> easier debugging [11]
>> * Spark ValidatesRunner PostCommit passes [12] [13]
>> * Flink ValidatesRunner PostCommit more reliable [14]
>> Documentation / IDE Setup:
>> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
>> Performance Benchmarks:
>> * Working on getting Nexmark benchmarks migrated [18]
>>
>> If I missed anything, please add it to this thread.
>>
>> We are continuing to use this general roadmap:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
>> created a separate issue to track post-migration cleanup items: BEAM-4045
>> [20]. Feel free to grab any unassigned items off of either list.
>>
>>
>> [1] https://github.com/apache/beam/pull/5054
>> [2] https://github.com/apache/beam/pull/5057
>> [3] https://github.com/apache/beam/pull/5087
>> [4] https://github.com/apache/beam/pull/5098
>> [5] https://github.com/apache/beam/pull/5047
>> [6] https://github.com/apache/beam/pull/5088
>> [7] https://github.com/apache/beam/pull/5086
>> [8] https://github.com/apache/beam/pull/5066
>> [9] https://github.com/apache/beam/pull/5059
>> [10] https://github.com/apache/beam/pull/5045
>> [11] https://github.com/apache/beam/pull/5091
>> [12] https://github.com/apache/beam/pull/5093
>> [13] https://github.com/apache/beam/pull/5069
>> [14] https://github.com/apache/beam/pull/5068
>> [15] https://beam.apache.org/contribute/contribution-guide/
>> [16] https://github.com/apache/beam-site/pull/412
>> [17] https://github.com/apache/beam-site/pull/414
>> [18] https://github.com/apache/beam/pull/5051
>> [19] https://issues.apache.org/jira/browse/BEAM-3249
>> [20] https://issues.apache.org/jira/browse/BEAM-4045
>>
>> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:
>>
>> I wanted to start a thread to summarize the current state of Gradle
>> migration. We've made lots of good progress so far this week. Here's the
>> status from what I can tell-- please add or correct anything I missed:
>>
>> * Release artifacts can be built and published for Snapshot and officlal
>> releases [1]
>> * Gradle-generated releases have been validated with the the Apache Beam
>> archetype generation quickstart; still needs additional validation.
>> * Generated release pom files have correct project metadata [2]
>> * The python pre-commits are now working in Gradle [3]
>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>> learn the new system-- please add your own. This will eventually feed into
>> official documentation on the website.
>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>> * Daniel is working on updating documentation to refer to Gradle 

Re: Gradle Status [April 11]

[Ahmet Altay]

> Found another blocker in current artifacts creations: there is not
pom.xml and pom.properties in META-INF. This is used by tools +
libraries + integrations so it is quite important to not break it

Romain, is there a JIRA for this issues? If not could you create one please?

On Thu, Apr 12, 2018 at 3:17 AM, Etienne Chauchot 
wrote:

> Nice !
> thanks Kenn
>
> Le mercredi 11 avril 2018 à 18:21 +, Kenneth Knowles a écrit :
>
> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
> Netty has been introduced since yesterday. Etienne mentioned he has worked
> toward setting up periodic runs on all runners, so this should help get us
> towards that. We'll probably prefer to build standalone fat jars for
> selected runners and use those, which is pending unknown issues in the
> shadow config leaving out required dependencies.
>
> Kenn
>
> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner  wrote:
>
> Thanks everyone for the continued effort towards the Gradle migration. As
> a high-level summary of our progress since Friday: we have a viable build,
> with a number of minor issues that we're still working out. Please take a
> look at the new documentation in our contribution guide and log any bugs
> that you find.
>
> Here's a more detailed view of improvements from just the past few days..
>
> Release artifacts:
> *  Pom.xml generation logic now in master [1]
> * Nightly snapshots are now produced using Gradle [2]
> * Excluded modules propagated to dependencies when generating * pom.xml
> * Artifact JARs are properly shaded [3]
> * Working on fixing dependency scopes in generated pom [4]
> PreCommits / Postcommits:
> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
> [7] [8] [9]
> * Jenkins results now include JUnit test results [10] and build scan for
> easier debugging [11]
> * Spark ValidatesRunner PostCommit passes [12] [13]
> * Flink ValidatesRunner PostCommit more reliable [14]
> Documentation / IDE Setup:
> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
> Performance Benchmarks:
> * Working on getting Nexmark benchmarks migrated [18]
>
> If I missed anything, please add it to this thread.
>
> We are continuing to use this general roadmap:
> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to Gradle
> (d) Migrate perfkit suites to use Gradle
>
> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
> created a separate issue to track post-migration cleanup items:
> BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.
>
>
> [1] https://github.com/apache/beam/pull/5054
> [2] https://github.com/apache/beam/pull/5057
> [3] https://github.com/apache/beam/pull/5087
> [4] https://github.com/apache/beam/pull/5098
> [5] https://github.com/apache/beam/pull/5047
> [6] https://github.com/apache/beam/pull/5088
> [7] https://github.com/apache/beam/pull/5086
> [8] https://github.com/apache/beam/pull/5066
> [9] https://github.com/apache/beam/pull/5059
> [10] https://github.com/apache/beam/pull/5045
> [11] https://github.com/apache/beam/pull/5091
> [12] https://github.com/apache/beam/pull/5093
> [13] https://github.com/apache/beam/pull/5069
> [14] https://github.com/apache/beam/pull/5068
> [15] https://beam.apache.org/contribute/contribution-guide/
> [16] https://github.com/apache/beam-site/pull/412
> [17] https://github.com/apache/beam-site/pull/414
> [18] https://github.com/apache/beam/pull/5051
> [19] https://issues.apache.org/jira/browse/BEAM-3249
> [20] https://issues.apache.org/jira/browse/BEAM-4045
>
> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:
>
> I wanted to start a thread to summarize the current state of Gradle
> migration. We've made lots of good progress so far this week. Here's the
> status from what I can tell-- please add or correct anything I missed:
>
> * Release artifacts can be built and published for Snapshot and officlal
> releases [1]
> * Gradle-generated releases have been validated with the the Apache Beam
> archetype generation quickstart; still needs additional validation.
> * Generated release pom files have correct project metadata [2]
> * The python pre-commits are now working in Gradle [3]
> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
> learn the new system-- please add your own. This will eventually feed into
> official documentation on the website.
> * Łukasz Gajowy is working on migrating performance testing framework [5]
> * Daniel is working on updating documentation to refer to Gradle instead
> of maven
>
> If I missed anything, please add it to this thread.
>
> The general roadmap we're working towards is:
> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to Gradle
> (d) Migrate perfkit 

Re: Gradle Status [April 6]

[Alexey Romanenko]

Daniel, actually I did run it with default IDEA JUnit test runner. Then, in 
“Settings > Build, Execution, Deployment > Build Tools > Gradle > Runner" I 
selected “Gradle Test Runner” in “Run tests using” selectbox and it works ok 
when I run my tests with IDEA shortcuts. So, probably, we should add this 
details on https://beam.apache.org/contribute/intellij/ 
 too.
What do you think?

WBR,
Alexey

> On 11 Apr 2018, at 21:17, Daniel Oliveira  wrote:
> 
> Alexey, are you referring to tests run with "./gradlew 
> :beam-runners-direct-java:needsRunnerTests"? That command works fine for me 
> in both versions of IDEA, but I believe the same tests fail if you run them 
> directly through "./gradlew test".
> 
> However, I am having issues with a bunch of validatesRunner tests, mostly be 
> caused by :beam-runners-google-cloud-dataflow-java:validatesRunner. Not sure 
> if it's because of a code change or a gradle config, I'll keep looking into 
> it.
> 
> On Wed, Apr 11, 2018 at 11:01 AM Romain Manni-Bucau  > wrote:
> I got tests running rrconfiguring gradle (which was setup for another project 
> but seems beam didnt like it) but latency is still "high" using gradle runner 
> for tests (like Etienne said ~3s on an i7 with 16G vs a few ms with default 
> idea test runner, would be great to solve that).
> 
> I also find the integration quite fishy cause configurations are customs so 
> idea is kind of forced to propose your modukes 3 times at least when you 
> select the classpath (x_test being generally the working one).
> 
> Also the false positive you get if you forget a cleanX is a bit annoying, 
> maybe we should force a clean for test or at least when there is a --tests to 
> avoid gradle to not run it cause there is no diff.
> 
> So it works but dev productivity is reduced a lot and it became slow enough 
> to think if you should do a contribution or not - at least for me.
> 
> Le 11 avr. 2018 19:37, "Alexey Romanenko"  > a écrit :
> I’ve managed to import a project as it’s described in documentation (starting 
> from empty project) using Idea 2018 and run unit tests successfully. 
> For some reasons, tests, that use DirectRunner to run a pipeline, were failed.
> 
> WBR,
> Alexey
> 
>> On 11 Apr 2018, at 19:01, Daniel Oliveira > > wrote:
>> 
>> Hi everyone, I was the one who initially wrote the PR with Idea instructions 
>> . I was using 2017.3 as well 
>> while writing it so all the instructions were tested on that version. I'll 
>> try testing the instructions on 2018 to see if I can reproduce the issues 
>> people are having.
>> 
>> On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik > > wrote:
>> I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet.
>> 
>> On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau > > wrote:
>> Any of you using the idea 2018? the import works for me but then it is
>> not as smooth as it seems for you. I'm just trying to see if it is a
>> procedure thing or a version issue.
>> 
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> 
>> 
>> 2018-04-11 17:28 GMT+02:00 Kenneth Knowles > >:
>> > The only reason I did "empty project then add a module" procedure was to 
>> > get
>> > all the IntelliJ files outside the source tree. IIRC directly creating from
>> > existing sources didn't give the necessary configuration options. If you
>> > don't care about being able to `git clean` then you can do the shorter
>> > version. Also the particular UI for creation might have improved since I
>> > tried it. I'll do it again.
>> >
>> > On the pom, since it is only generated for machine reading, why care if the
>> > parent is inlined? I actually prefer to avoid coupling with things that you
>> > have to go and look up. Using inheritance is OK for human edited poms
>> > (actually IMO it is still a mistake) but it doesn't change the semantics of
>> > a shipped pom if they are all immutable, which they should be. It is better
>> > to have all the info right there. Is there an actually effective 
>> > difference?
>> >
>> > Kenn
>> >
>> > On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot > > >
>> > wrote:
>> >>
>> >> Hi all,
>> >> I just tested gradle environment from a fresh source clone with this
>> >> procedure with just a tiny change: I used "new project from existing
>> >> sources" rather than create empty project and then add module.
>> >>
>> >> It works fine and junit runs from intellij also work.  with gradle we pay
>> >> a 2s delay (on my machine) before running the 

Re: Gradle Status [April 11]

[Etienne Chauchot]

Nice !
thanks Kenn
Le mercredi 11 avril 2018 à 18:21 +, Kenneth Knowles a écrit :
> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner + Netty 
> has been introduced since yesterday.
> Etienne mentioned he has worked toward setting up periodic runs on all 
> runners, so this should help get us towards
> that. We'll probably prefer to build standalone fat jars for selected runners 
> and use those, which is pending unknown
> issues in the shadow config leaving out required dependencies.
> 
> Kenn
> 
> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner  wrote:
> > Thanks everyone for the continued effort towards the Gradle migration. As a 
> > high-level summary of our progress since
> > Friday: we have a viable build, with a number of minor issues that we're 
> > still working out. Please take a look at
> > the new documentation in our contribution guide and log any bugs that you 
> > find.
> > 
> > Here's a more detailed view of improvements from just the past few days..
> > 
> > Release artifacts:
> > *  Pom.xml generation logic now in master [1]
> > * Nightly snapshots are now produced using Gradle [2]
> > * Excluded modules propagated to dependencies when generating * pom.xml
> > * Artifact JARs are properly shaded [3]
> > * Working on fixing dependency scopes in generated pom [4]
> > PreCommits / Postcommits:
> > * All PreCommits and PostCommits migrated [5]; working on deflaking [6] [7] 
> > [8] [9]
> > * Jenkins results now include JUnit test results [10] and build scan for 
> > easier debugging [11]
> > * Spark ValidatesRunner PostCommit passes [12] [13]
> > * Flink ValidatesRunner PostCommit more reliable [14] 
> > Documentation / IDE Setup:
> > * Contribution Guide [15] is now updated with Gradle commands [16] [17]
> > Performance Benchmarks:
> > * Working on getting Nexmark benchmarks migrated [18]
> > 
> > If I missed anything, please add it to this thread.
> > 
> > We are continuing to use this general roadmap:
> > (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> > (b) Postcommits migrated to Gradle
> > (c) Migrate documentation from maven to Gradle
> > (d) Migrate perfkit suites to use Gradle
> > 
> > Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has created 
> > a separate issue to track post-migration 
> > cleanup items: BEAM-4045 [20]. Feel free to grab any unassigned items off 
> > of either list.
> > 
> > 
> > [1] https://github.com/apache/beam/pull/5054 
> > [2] https://github.com/apache/beam/pull/5057 
> > [3] https://github.com/apache/beam/pull/5087 
> > [4] https://github.com/apache/beam/pull/5098 
> > [5] https://github.com/apache/beam/pull/5047 
> > [6] https://github.com/apache/beam/pull/5088
> > [7] https://github.com/apache/beam/pull/5086 
> > [8] https://github.com/apache/beam/pull/5066 
> > [9] https://github.com/apache/beam/pull/5059 
> > [10] https://github.com/apache/beam/pull/5045 
> > [11] https://github.com/apache/beam/pull/5091 
> > [12] https://github.com/apache/beam/pull/5093 
> > [13] https://github.com/apache/beam/pull/5069 
> > [14] https://github.com/apache/beam/pull/5068 
> > [15] https://beam.apache.org/contribute/contribution-guide/ 
> > [16] https://github.com/apache/beam-site/pull/412 
> > [17] https://github.com/apache/beam-site/pull/414 
> > [18] https://github.com/apache/beam/pull/5051 
> > [19] https://issues.apache.org/jira/browse/BEAM-3249 
> > [20] https://issues.apache.org/jira/browse/BEAM-4045 
> > 
> > On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:
> > > I wanted to start a thread to summarize the current state of Gradle 
> > > migration. We've made lots of good progress so
> > > far this week. Here's the status from what I can tell-- please add or 
> > > correct anything I missed:
> > > 
> > > * Release artifacts can be built and published for Snapshot and officlal 
> > > releases [1]
> > > * Gradle-generated releases have been validated with the the Apache Beam 
> > > archetype generation quickstart; still
> > > needs additional validation.
> > > * Generated release pom files have correct project metadata [2]
> > > * The python pre-commits are now working in Gradle [3]
> > > * Ismaël has started a collaborative doc of Gradle tips [4] as we all 
> > > learn the new system-- please add your own.
> > > This will eventually feed into official documentation on the website.
> > > * Łukasz Gajowy is working on migrating performance testing framework [5]
> > > * Daniel is working on updating documentation to refer to Gradle instead 
> > > of maven
> > > 
> > > If I missed anything, please add it to this thread.
> > > 
> > > The general roadmap we're working towards is:
> > > (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> > > (b) Postcommits migrated to Gradle
> > > (c) Migrate documentation from maven to Gradle
> > > (d) Migrate perfkit suites to use Gradle
> > > 
> > > For those of you that are hacking: thanks for your 

Re: Gradle Status [April 6]

[Etienne Chauchot]

My test was also on version 2017, 2017.2.7 to be precise
Etienne
Le mercredi 11 avril 2018 à 17:01 +, Daniel Oliveira a écrit :
> Hi everyone, I was the one who initially wrote the PR with Idea instructions. 
> I was using 2017.3 as well while writing
> it so all the instructions were tested on that version. I'll try testing the 
> instructions on 2018 to see if I can
> reproduce the issues people are having.
> 
> On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik  wrote:
> > I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet.
> > 
> > On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau  
> > wrote:
> > > Any of you using the idea 2018? the import works for me but then it is
> > > not as smooth as it seems for you. I'm just trying to see if it is a
> > > procedure thing or a version issue.
> > > 
> > > Romain Manni-Bucau
> > > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> > > 
> > > 
> > > 2018-04-11 17:28 GMT+02:00 Kenneth Knowles :
> > > > The only reason I did "empty project then add a module" procedure was 
> > > > to get
> > > > all the IntelliJ files outside the source tree. IIRC directly creating 
> > > > from
> > > > existing sources didn't give the necessary configuration options. If you
> > > > don't care about being able to `git clean` then you can do the shorter
> > > > version. Also the particular UI for creation might have improved since I
> > > > tried it. I'll do it again.
> > > >
> > > > On the pom, since it is only generated for machine reading, why care if 
> > > > the
> > > > parent is inlined? I actually prefer to avoid coupling with things that 
> > > > you
> > > > have to go and look up. Using inheritance is OK for human edited poms
> > > > (actually IMO it is still a mistake) but it doesn't change the 
> > > > semantics of
> > > > a shipped pom if they are all immutable, which they should be. It is 
> > > > better
> > > > to have all the info right there. Is there an actually effective 
> > > > difference?
> > > >
> > > > Kenn
> > > >
> > > > On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot 
> > > > wrote:
> > > >>
> > > >> Hi all,
> > > >> I just tested gradle environment from a fresh source clone with this
> > > >> procedure with just a tiny change: I used "new project from existing
> > > >> sources" rather than create empty project and then add module.
> > > >>
> > > >> It works fine and junit runs from intellij also work.  with gradle we 
> > > >> pay
> > > >> a 2s delay (on my machine) before running the actual test for each 
> > > >> run. This
> > > >> delay seems quite constant no matter the module: I have run all the 
> > > >> tests at
> > > >> once in  runner-core and a single test in another module with a similar
> > > >> delay.
> > > >>
> > > >> I also tested a debug session from intellij and it runs fine also.
> > > >>
> > > >> I'll do some more testing and keep you posted.
> > > >>
> > > >> I still have some concerns about the potential difficulty for new
> > > >> contributors to have to learn gradle in addition to Beam itself 
> > > >> comparing to
> > > >> maven which is more mainstream for java developers. That could be
> > > >> discouraging for ex for part-time contributors
> > > >>
> > > >> Etienne
> > > >>
> > > >> Le mardi 10 avril 2018 à 14:38 +, Lukasz Cwik a écrit :
> > > >>
> > > >> beam-site PR/414 updates the instructions for using Intellij and how to
> > > >> import a module:
> > > >>
> > > >> 1. Create an empty IntelliJ project outside of the Beam source tree.
> > > >> 2. Under Project Structure > Project, select a Project SDK.
> > > >> 3. Under Project Structure > Modules, click the + sign to add a module 
> > > >> and
> > > >>    select "Import Module".
> > > >>     1. Select the directory containing the Beam source tree.
> > > >>     2. Tick the "Import module from external model" button and select
> > > >> Gradle
> > > >>        from the list.
> > > >>     3. Tick the following boxes.
> > > >>        * Use auto-import
> > > >>        * Create separate module per source set
> > > >>        * Store generated project files externally
> > > >>        * Use default gradle wrapper
> > > >> 4. Delegate build actions to Gradle by going to Settings > Build,
> > > >> Execution,
> > > >>    Deployment > Build Tools > Gradle and checking "Delegate IDE 
> > > >>build/run
> > > >>    actions to gradle".
> > > >>
> > > >> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré 
> > > >> 
> > > >> wrote:
> > > >>
> > > >> That's a very important issue for contribution. Up to now, I used Maven
> > > >> for setup IntelliJ (and it works just fine). If we remove the pom.xml,
> > > >> we have to support Eclipse and IntelliJ "smoothly".
> > > >>
> > > >> Let me try in IntelliJ.
> > > >>
> > > >> Regards
> > > >> JB
> > > >>
> > > >> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
> > > >> > You dont have issue due to the build setup with that 

Re: Gradle Status [April 11]

[Romain Manni-Bucau]

Found another blocker in current artifacts creations: there is not
pom.xml and pom.properties in META-INF. This is used by tools +
libraries + integrations so it is quite important to not break it.

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-12 0:59 GMT+02:00 Kenneth Knowles :
> There are plenty of technical hurdles since Bazel is best suited to an
> isolated monorepo. We could probably have overcome those problems with
> effort. I got through a few of them in the short sprint I did. Ultimately it
> was a community-driven decision: more people (as in "more than just me" :-)
> jumped on board and got excited about building out the Gradle build.
>
> Kenn
>
> On Wed, Apr 11, 2018 at 3:17 PM Nathan Fisher 
> wrote:
>>
>> Is there a document or similar outlining the decision behind Gradle over
>> Bazel?
>>
>> For purely intellectual curiosity I’m curious what tradeoffs/benefits were
>> considered when evaluating the two or if it was more a matter of
>> community/contributor familiarity.
>>
>> I found this thread which seems to imply issues around dependency
>> management;
>>
>>
>> https://lists.apache.org/thread.html/bba0a89f2561fb2f7150a8381c1eb3923fa46299f3b35ff1304d7c46@%3Cdev.beam.apache.org%3E
>>
>> On Wed, Apr 11, 2018 at 3:21 PM, Kenneth Knowles  wrote:
>>>
>>> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
>>> Netty has been introduced since yesterday. Etienne mentioned he has worked
>>> toward setting up periodic runs on all runners, so this should help get us
>>> towards that. We'll probably prefer to build standalone fat jars for
>>> selected runners and use those, which is pending unknown issues in the
>>> shadow config leaving out required dependencies.
>>>
>>> Kenn
>>>
>>> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner  wrote:

 Thanks everyone for the continued effort towards the Gradle migration.
 As a high-level summary of our progress since Friday: we have a viable
 build, with a number of minor issues that we're still working out. Please
 take a look at the new documentation in our contribution guide and log any
 bugs that you find.

 Here's a more detailed view of improvements from just the past few
 days..

 Release artifacts:
 *  Pom.xml generation logic now in master [1]
 * Nightly snapshots are now produced using Gradle [2]
 * Excluded modules propagated to dependencies when generating * pom.xml
 * Artifact JARs are properly shaded [3]
 * Working on fixing dependency scopes in generated pom [4]
 PreCommits / Postcommits:
 * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
 [7] [8] [9]
 * Jenkins results now include JUnit test results [10] and build scan for
 easier debugging [11]
 * Spark ValidatesRunner PostCommit passes [12] [13]
 * Flink ValidatesRunner PostCommit more reliable [14]
 Documentation / IDE Setup:
 * Contribution Guide [15] is now updated with Gradle commands [16] [17]
 Performance Benchmarks:
 * Working on getting Nexmark benchmarks migrated [18]

 If I missed anything, please add it to this thread.

 We are continuing to use this general roadmap:
 (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
 (b) Postcommits migrated to Gradle
 (c) Migrate documentation from maven to Gradle
 (d) Migrate perfkit suites to use Gradle

 Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
 created a separate issue to track post-migration cleanup items: BEAM-4045
 [20]. Feel free to grab any unassigned items off of either list.


 [1] https://github.com/apache/beam/pull/5054
 [2] https://github.com/apache/beam/pull/5057
 [3] https://github.com/apache/beam/pull/5087
 [4] https://github.com/apache/beam/pull/5098
 [5] https://github.com/apache/beam/pull/5047
 [6] https://github.com/apache/beam/pull/5088
 [7] https://github.com/apache/beam/pull/5086
 [8] https://github.com/apache/beam/pull/5066
 [9] https://github.com/apache/beam/pull/5059
 [10] https://github.com/apache/beam/pull/5045
 [11] https://github.com/apache/beam/pull/5091
 [12] https://github.com/apache/beam/pull/5093
 [13] https://github.com/apache/beam/pull/5069
 [14] https://github.com/apache/beam/pull/5068
 [15] https://beam.apache.org/contribute/contribution-guide/
 [16] https://github.com/apache/beam-site/pull/412
 [17] https://github.com/apache/beam-site/pull/414
 [18] https://github.com/apache/beam/pull/5051
 [19] https://issues.apache.org/jira/browse/BEAM-3249
 [20] https://issues.apache.org/jira/browse/BEAM-4045

 On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:
>
> I wanted to start a thread to summarize the current state of Gradle
> 

Re: Gradle Status [April 11]

[Kenneth Knowles]

There are plenty of technical hurdles since Bazel is best suited to an
isolated monorepo. We could probably have overcome those problems with
effort. I got through a few of them in the short sprint I did. Ultimately
it was a community-driven decision: more people (as in "more than just me"
:-) jumped on board and got excited about building out the Gradle build.

Kenn

On Wed, Apr 11, 2018 at 3:17 PM Nathan Fisher 
wrote:

> Is there a document or similar outlining the decision behind Gradle over
> Bazel?
>
> For purely intellectual curiosity I’m curious what tradeoffs/benefits were
> considered when evaluating the two or if it was more a matter of
> community/contributor familiarity.
>
> I found this thread which seems to imply issues around dependency
> management;
>
>
> https://lists.apache.org/thread.html/bba0a89f2561fb2f7150a8381c1eb3923fa46299f3b35ff1304d7c46@%3Cdev.beam.apache.org%3E
>
> On Wed, Apr 11, 2018 at 3:21 PM, Kenneth Knowles  wrote:
>
>> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
>> Netty has been introduced since yesterday. Etienne mentioned he has worked
>> toward setting up periodic runs on all runners, so this should help get us
>> towards that. We'll probably prefer to build standalone fat jars for
>> selected runners and use those, which is pending unknown issues in the
>> shadow config leaving out required dependencies.
>>
>> Kenn
>>
>> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner  wrote:
>>
>>> Thanks everyone for the continued effort towards the Gradle migration.
>>> As a high-level summary of our progress since Friday: we have a viable
>>> build, with a number of minor issues that we're still working out. Please
>>> take a look at the new documentation in our contribution guide and log any
>>> bugs that you find.
>>>
>>> Here's a more detailed view of improvements from just the past few days..
>>>
>>> Release artifacts:
>>> *  Pom.xml generation logic now in master [1]
>>> * Nightly snapshots are now produced using Gradle [2]
>>> * Excluded modules propagated to dependencies when generating * pom.xml
>>> * Artifact JARs are properly shaded [3]
>>> * Working on fixing dependency scopes in generated pom [4]
>>> PreCommits / Postcommits:
>>> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
>>> [7] [8] [9]
>>> * Jenkins results now include JUnit test results [10] and build scan for
>>> easier debugging [11]
>>> * Spark ValidatesRunner PostCommit passes [12] [13]
>>> * Flink ValidatesRunner PostCommit more reliable [14]
>>> Documentation / IDE Setup:
>>> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
>>> Performance Benchmarks:
>>> * Working on getting Nexmark benchmarks migrated [18]
>>>
>>> If I missed anything, please add it to this thread.
>>>
>>> We are continuing to use this general roadmap:
>>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>>> (b) Postcommits migrated to Gradle
>>> (c) Migrate documentation from maven to Gradle
>>> (d) Migrate perfkit suites to use Gradle
>>>
>>> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
>>> created a separate issue to track post-migration cleanup items:
>>> BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.
>>>
>>>
>>> [1] https://github.com/apache/beam/pull/5054
>>> [2] https://github.com/apache/beam/pull/5057
>>> [3] https://github.com/apache/beam/pull/5087
>>> [4] https://github.com/apache/beam/pull/5098
>>> [5] https://github.com/apache/beam/pull/5047
>>> [6] https://github.com/apache/beam/pull/5088
>>> [7] https://github.com/apache/beam/pull/5086
>>> [8] https://github.com/apache/beam/pull/5066
>>> [9] https://github.com/apache/beam/pull/5059
>>> [10] https://github.com/apache/beam/pull/5045
>>> [11] https://github.com/apache/beam/pull/5091
>>> [12] https://github.com/apache/beam/pull/5093
>>> [13] https://github.com/apache/beam/pull/5069
>>> [14] https://github.com/apache/beam/pull/5068
>>> [15] https://beam.apache.org/contribute/contribution-guide/
>>> [16] https://github.com/apache/beam-site/pull/412
>>> [17] https://github.com/apache/beam-site/pull/414
>>> [18] https://github.com/apache/beam/pull/5051
>>> [19] https://issues.apache.org/jira/browse/BEAM-3249
>>> [20] https://issues.apache.org/jira/browse/BEAM-4045
>>>
>>> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:
>>>
 I wanted to start a thread to summarize the current state of Gradle
 migration. We've made lots of good progress so far this week. Here's the
 status from what I can tell-- please add or correct anything I missed:

 * Release artifacts can be built and published for Snapshot and
 officlal releases [1]
 * Gradle-generated releases have been validated with the the Apache
 Beam archetype generation quickstart; still needs additional validation.
 * Generated release pom files have correct project 

Re: Gradle Status [April 11]

[Nathan Fisher]

Is there a document or similar outlining the decision behind Gradle over
Bazel?

For purely intellectual curiosity I’m curious what tradeoffs/benefits were
considered when evaluating the two or if it was more a matter of
community/contributor familiarity.

I found this thread which seems to imply issues around dependency
management;

https://lists.apache.org/thread.html/bba0a89f2561fb2f7150a8381c1eb3923fa46299f3b35ff1304d7c46@%3Cdev.beam.apache.org%3E

On Wed, Apr 11, 2018 at 3:21 PM, Kenneth Knowles  wrote:

> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
> Netty has been introduced since yesterday. Etienne mentioned he has worked
> toward setting up periodic runs on all runners, so this should help get us
> towards that. We'll probably prefer to build standalone fat jars for
> selected runners and use those, which is pending unknown issues in the
> shadow config leaving out required dependencies.
>
> Kenn
>
> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner  wrote:
>
>> Thanks everyone for the continued effort towards the Gradle migration. As
>> a high-level summary of our progress since Friday: we have a viable build,
>> with a number of minor issues that we're still working out. Please take a
>> look at the new documentation in our contribution guide and log any bugs
>> that you find.
>>
>> Here's a more detailed view of improvements from just the past few days..
>>
>> Release artifacts:
>> *  Pom.xml generation logic now in master [1]
>> * Nightly snapshots are now produced using Gradle [2]
>> * Excluded modules propagated to dependencies when generating * pom.xml
>> * Artifact JARs are properly shaded [3]
>> * Working on fixing dependency scopes in generated pom [4]
>> PreCommits / Postcommits:
>> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
>> [7] [8] [9]
>> * Jenkins results now include JUnit test results [10] and build scan for
>> easier debugging [11]
>> * Spark ValidatesRunner PostCommit passes [12] [13]
>> * Flink ValidatesRunner PostCommit more reliable [14]
>> Documentation / IDE Setup:
>> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
>> Performance Benchmarks:
>> * Working on getting Nexmark benchmarks migrated [18]
>>
>> If I missed anything, please add it to this thread.
>>
>> We are continuing to use this general roadmap:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
>> created a separate issue to track post-migration cleanup items:
>> BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.
>>
>>
>> [1] https://github.com/apache/beam/pull/5054
>> [2] https://github.com/apache/beam/pull/5057
>> [3] https://github.com/apache/beam/pull/5087
>> [4] https://github.com/apache/beam/pull/5098
>> [5] https://github.com/apache/beam/pull/5047
>> [6] https://github.com/apache/beam/pull/5088
>> [7] https://github.com/apache/beam/pull/5086
>> [8] https://github.com/apache/beam/pull/5066
>> [9] https://github.com/apache/beam/pull/5059
>> [10] https://github.com/apache/beam/pull/5045
>> [11] https://github.com/apache/beam/pull/5091
>> [12] https://github.com/apache/beam/pull/5093
>> [13] https://github.com/apache/beam/pull/5069
>> [14] https://github.com/apache/beam/pull/5068
>> [15] https://beam.apache.org/contribute/contribution-guide/
>> [16] https://github.com/apache/beam-site/pull/412
>> [17] https://github.com/apache/beam-site/pull/414
>> [18] https://github.com/apache/beam/pull/5051
>> [19] https://issues.apache.org/jira/browse/BEAM-3249
>> [20] https://issues.apache.org/jira/browse/BEAM-4045
>>
>> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:
>>
>>> I wanted to start a thread to summarize the current state of Gradle
>>> migration. We've made lots of good progress so far this week. Here's the
>>> status from what I can tell-- please add or correct anything I missed:
>>>
>>> * Release artifacts can be built and published for Snapshot and officlal
>>> releases [1]
>>> * Gradle-generated releases have been validated with the the Apache Beam
>>> archetype generation quickstart; still needs additional validation.
>>> * Generated release pom files have correct project metadata [2]
>>> * The python pre-commits are now working in Gradle [3]
>>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>>> learn the new system-- please add your own. This will eventually feed into
>>> official documentation on the website.
>>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>>> * Daniel is working on updating documentation to refer to Gradle instead
>>> of maven
>>>
>>> If I missed anything, please add it to this thread.
>>>
>>> The general roadmap we're working towards is:
>>> (a) 

Re: Gradle Status [April 6]

[Lukasz Cwik]

Running Dataflow validates runner tests requires you to have permission to
launch jobs on the 'apache-beam-testing' project.
You'll need to override dataflowProject and dataflowTempRoot with a GCP
project and GCS bucket you have access to.

On Wed, Apr 11, 2018 at 3:17 PM Daniel Oliveira 
wrote:

> Alexey, are you referring to tests run with "./gradlew
> :beam-runners-direct-java:needsRunnerTests"? That command works fine for me
> in both versions of IDEA, but I believe the same tests fail if you run them
> directly through "./gradlew test".
>
> However, I am having issues with a bunch of validatesRunner tests, mostly
> be caused by :beam-runners-google-cloud-dataflow-java:validatesRunner. Not
> sure if it's because of a code change or a gradle config, I'll keep looking
> into it.
>
> On Wed, Apr 11, 2018 at 11:01 AM Romain Manni-Bucau 
> wrote:
>
>> I got tests running rrconfiguring gradle (which was setup for another
>> project but seems beam didnt like it) but latency is still "high" using
>> gradle runner for tests (like Etienne said ~3s on an i7 with 16G vs a few
>> ms with default idea test runner, would be great to solve that).
>>
>> I also find the integration quite fishy cause configurations are customs
>> so idea is kind of forced to propose your modukes 3 times at least when you
>> select the classpath (x_test being generally the working one).
>>
>> Also the false positive you get if you forget a cleanX is a bit annoying,
>> maybe we should force a clean for test or at least when there is a --tests
>> to avoid gradle to not run it cause there is no diff.
>>
>> So it works but dev productivity is reduced a lot and it became slow
>> enough to think if you should do a contribution or not - at least for me.
>>
>> Le 11 avr. 2018 19:37, "Alexey Romanenko"  a
>> écrit :
>>
>>> I’ve managed to import a project as it’s described in documentation
>>> (starting from empty project) using Idea 2018 and run unit tests
>>> successfully.
>>> For some reasons, tests, that use DirectRunner to run a pipeline, were
>>> failed.
>>>
>>> WBR,
>>> Alexey
>>>
>>> On 11 Apr 2018, at 19:01, Daniel Oliveira 
>>> wrote:
>>>
>>> Hi everyone, I was the one who initially wrote the PR with Idea
>>> instructions . I was
>>> using 2017.3 as well while writing it so all the instructions were tested
>>> on that version. I'll try testing the instructions on 2018 to see if I can
>>> reproduce the issues people are having.
>>>
>>> On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik  wrote:
>>>
 I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet.

 On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Any of you using the idea 2018? the import works for me but then it is
> not as smooth as it seems for you. I'm just trying to see if it is a
> procedure thing or a version issue.
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
>
> 2018-04-11 17:28 GMT+02:00 Kenneth Knowles :
> > The only reason I did "empty project then add a module" procedure
> was to get
> > all the IntelliJ files outside the source tree. IIRC directly
> creating from
> > existing sources didn't give the necessary configuration options. If
> you
> > don't care about being able to `git clean` then you can do the
> shorter
> > version. Also the particular UI for creation might have improved
> since I
> > tried it. I'll do it again.
> >
> > On the pom, since it is only generated for machine reading, why care
> if the
> > parent is inlined? I actually prefer to avoid coupling with things
> that you
> > have to go and look up. Using inheritance is OK for human edited poms
> > (actually IMO it is still a mistake) but it doesn't change the
> semantics of
> > a shipped pom if they are all immutable, which they should be. It is
> better
> > to have all the info right there. Is there an actually effective
> difference?
> >
> > Kenn
> >
> > On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot <
> echauc...@apache.org>
> > wrote:
> >>
> >> Hi all,
> >> I just tested gradle environment from a fresh source clone with this
> >> procedure with just a tiny change: I used "new project from existing
> >> sources" rather than create empty project and then add module.
> >>
> >> It works fine and junit runs from intellij also work.  with gradle
> we pay
> >> a 2s delay (on my machine) before running the actual test for each
> run. This
> >> delay seems quite constant no matter the module: I have run all the
> tests at
> >> once in  runner-core and a single test in another module with a
> similar
> >> delay.

Re: Gradle Status [April 6]

[Daniel Oliveira]

Alexey, are you referring to tests run with "./gradlew
:beam-runners-direct-java:needsRunnerTests"? That command works fine for me
in both versions of IDEA, but I believe the same tests fail if you run them
directly through "./gradlew test".

However, I am having issues with a bunch of validatesRunner tests, mostly
be caused by :beam-runners-google-cloud-dataflow-java:validatesRunner. Not
sure if it's because of a code change or a gradle config, I'll keep looking
into it.

On Wed, Apr 11, 2018 at 11:01 AM Romain Manni-Bucau 
wrote:

> I got tests running rrconfiguring gradle (which was setup for another
> project but seems beam didnt like it) but latency is still "high" using
> gradle runner for tests (like Etienne said ~3s on an i7 with 16G vs a few
> ms with default idea test runner, would be great to solve that).
>
> I also find the integration quite fishy cause configurations are customs
> so idea is kind of forced to propose your modukes 3 times at least when you
> select the classpath (x_test being generally the working one).
>
> Also the false positive you get if you forget a cleanX is a bit annoying,
> maybe we should force a clean for test or at least when there is a --tests
> to avoid gradle to not run it cause there is no diff.
>
> So it works but dev productivity is reduced a lot and it became slow
> enough to think if you should do a contribution or not - at least for me.
>
> Le 11 avr. 2018 19:37, "Alexey Romanenko"  a
> écrit :
>
>> I’ve managed to import a project as it’s described in documentation
>> (starting from empty project) using Idea 2018 and run unit tests
>> successfully.
>> For some reasons, tests, that use DirectRunner to run a pipeline, were
>> failed.
>>
>> WBR,
>> Alexey
>>
>> On 11 Apr 2018, at 19:01, Daniel Oliveira  wrote:
>>
>> Hi everyone, I was the one who initially wrote the PR with Idea
>> instructions . I was using
>> 2017.3 as well while writing it so all the instructions were tested on that
>> version. I'll try testing the instructions on 2018 to see if I can
>> reproduce the issues people are having.
>>
>> On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik  wrote:
>>
>>> I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet.
>>>
>>> On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Any of you using the idea 2018? the import works for me but then it is
 not as smooth as it seems for you. I'm just trying to see if it is a
 procedure thing or a version issue.

 Romain Manni-Bucau
 @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


 2018-04-11 17:28 GMT+02:00 Kenneth Knowles :
 > The only reason I did "empty project then add a module" procedure was
 to get
 > all the IntelliJ files outside the source tree. IIRC directly
 creating from
 > existing sources didn't give the necessary configuration options. If
 you
 > don't care about being able to `git clean` then you can do the shorter
 > version. Also the particular UI for creation might have improved
 since I
 > tried it. I'll do it again.
 >
 > On the pom, since it is only generated for machine reading, why care
 if the
 > parent is inlined? I actually prefer to avoid coupling with things
 that you
 > have to go and look up. Using inheritance is OK for human edited poms
 > (actually IMO it is still a mistake) but it doesn't change the
 semantics of
 > a shipped pom if they are all immutable, which they should be. It is
 better
 > to have all the info right there. Is there an actually effective
 difference?
 >
 > Kenn
 >
 > On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot <
 echauc...@apache.org>
 > wrote:
 >>
 >> Hi all,
 >> I just tested gradle environment from a fresh source clone with this
 >> procedure with just a tiny change: I used "new project from existing
 >> sources" rather than create empty project and then add module.
 >>
 >> It works fine and junit runs from intellij also work.  with gradle
 we pay
 >> a 2s delay (on my machine) before running the actual test for each
 run. This
 >> delay seems quite constant no matter the module: I have run all the
 tests at
 >> once in  runner-core and a single test in another module with a
 similar
 >> delay.
 >>
 >> I also tested a debug session from intellij and it runs fine also.
 >>
 >> I'll do some more testing and keep you posted.
 >>
 >> I still have some concerns about the potential difficulty for new
 >> contributors to have to learn gradle in addition to Beam itself
 comparing to
 >> maven which is more mainstream for java developers. That could be
 >> discouraging for ex for part-time contributors

Re: Gradle Status [April 11]

[Kenneth Knowles]

Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
Netty has been introduced since yesterday. Etienne mentioned he has worked
toward setting up periodic runs on all runners, so this should help get us
towards that. We'll probably prefer to build standalone fat jars for
selected runners and use those, which is pending unknown issues in the
shadow config leaving out required dependencies.

Kenn

On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner  wrote:

> Thanks everyone for the continued effort towards the Gradle migration. As
> a high-level summary of our progress since Friday: we have a viable build,
> with a number of minor issues that we're still working out. Please take a
> look at the new documentation in our contribution guide and log any bugs
> that you find.
>
> Here's a more detailed view of improvements from just the past few days..
>
> Release artifacts:
> *  Pom.xml generation logic now in master [1]
> * Nightly snapshots are now produced using Gradle [2]
> * Excluded modules propagated to dependencies when generating * pom.xml
> * Artifact JARs are properly shaded [3]
> * Working on fixing dependency scopes in generated pom [4]
> PreCommits / Postcommits:
> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
> [7] [8] [9]
> * Jenkins results now include JUnit test results [10] and build scan for
> easier debugging [11]
> * Spark ValidatesRunner PostCommit passes [12] [13]
> * Flink ValidatesRunner PostCommit more reliable [14]
> Documentation / IDE Setup:
> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
> Performance Benchmarks:
> * Working on getting Nexmark benchmarks migrated [18]
>
> If I missed anything, please add it to this thread.
>
> We are continuing to use this general roadmap:
> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to Gradle
> (d) Migrate perfkit suites to use Gradle
>
> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
> created a separate issue to track post-migration cleanup items:
> BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.
>
>
> [1] https://github.com/apache/beam/pull/5054
> [2] https://github.com/apache/beam/pull/5057
> [3] https://github.com/apache/beam/pull/5087
> [4] https://github.com/apache/beam/pull/5098
> [5] https://github.com/apache/beam/pull/5047
> [6] https://github.com/apache/beam/pull/5088
> [7] https://github.com/apache/beam/pull/5086
> [8] https://github.com/apache/beam/pull/5066
> [9] https://github.com/apache/beam/pull/5059
> [10] https://github.com/apache/beam/pull/5045
> [11] https://github.com/apache/beam/pull/5091
> [12] https://github.com/apache/beam/pull/5093
> [13] https://github.com/apache/beam/pull/5069
> [14] https://github.com/apache/beam/pull/5068
> [15] https://beam.apache.org/contribute/contribution-guide/
> [16] https://github.com/apache/beam-site/pull/412
> [17] https://github.com/apache/beam-site/pull/414
> [18] https://github.com/apache/beam/pull/5051
> [19] https://issues.apache.org/jira/browse/BEAM-3249
> [20] https://issues.apache.org/jira/browse/BEAM-4045
>
> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner  wrote:
>
>> I wanted to start a thread to summarize the current state of Gradle
>> migration. We've made lots of good progress so far this week. Here's the
>> status from what I can tell-- please add or correct anything I missed:
>>
>> * Release artifacts can be built and published for Snapshot and officlal
>> releases [1]
>> * Gradle-generated releases have been validated with the the Apache Beam
>> archetype generation quickstart; still needs additional validation.
>> * Generated release pom files have correct project metadata [2]
>> * The python pre-commits are now working in Gradle [3]
>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>> learn the new system-- please add your own. This will eventually feed into
>> official documentation on the website.
>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>> * Daniel is working on updating documentation to refer to Gradle instead
>> of maven
>>
>> If I missed anything, please add it to this thread.
>>
>> The general roadmap we're working towards is:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> For those of you that are hacking: thanks for your help so far! Progress
>> is being roughly tracked on the Kanban [6]; please make sure the issues
>> assigned to you are up-to-date. Many of the changes are staged on
>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>
>>
>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>> [3] 

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

I got tests running rrconfiguring gradle (which was setup for another
project but seems beam didnt like it) but latency is still "high" using
gradle runner for tests (like Etienne said ~3s on an i7 with 16G vs a few
ms with default idea test runner, would be great to solve that).

I also find the integration quite fishy cause configurations are customs so
idea is kind of forced to propose your modukes 3 times at least when you
select the classpath (x_test being generally the working one).

Also the false positive you get if you forget a cleanX is a bit annoying,
maybe we should force a clean for test or at least when there is a --tests
to avoid gradle to not run it cause there is no diff.

So it works but dev productivity is reduced a lot and it became slow enough
to think if you should do a contribution or not - at least for me.

Le 11 avr. 2018 19:37, "Alexey Romanenko"  a
écrit :

> I’ve managed to import a project as it’s described in documentation
> (starting from empty project) using Idea 2018 and run unit tests
> successfully.
> For some reasons, tests, that use DirectRunner to run a pipeline, were
> failed.
>
> WBR,
> Alexey
>
> On 11 Apr 2018, at 19:01, Daniel Oliveira  wrote:
>
> Hi everyone, I was the one who initially wrote the PR with Idea
> instructions . I was using
> 2017.3 as well while writing it so all the instructions were tested on that
> version. I'll try testing the instructions on 2018 to see if I can
> reproduce the issues people are having.
>
> On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik  wrote:
>
>> I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet.
>>
>> On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Any of you using the idea 2018? the import works for me but then it is
>>> not as smooth as it seems for you. I'm just trying to see if it is a
>>> procedure thing or a version issue.
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>
>>>
>>> 2018-04-11 17:28 GMT+02:00 Kenneth Knowles :
>>> > The only reason I did "empty project then add a module" procedure was
>>> to get
>>> > all the IntelliJ files outside the source tree. IIRC directly creating
>>> from
>>> > existing sources didn't give the necessary configuration options. If
>>> you
>>> > don't care about being able to `git clean` then you can do the shorter
>>> > version. Also the particular UI for creation might have improved since
>>> I
>>> > tried it. I'll do it again.
>>> >
>>> > On the pom, since it is only generated for machine reading, why care
>>> if the
>>> > parent is inlined? I actually prefer to avoid coupling with things
>>> that you
>>> > have to go and look up. Using inheritance is OK for human edited poms
>>> > (actually IMO it is still a mistake) but it doesn't change the
>>> semantics of
>>> > a shipped pom if they are all immutable, which they should be. It is
>>> better
>>> > to have all the info right there. Is there an actually effective
>>> difference?
>>> >
>>> > Kenn
>>> >
>>> > On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot >> >
>>> > wrote:
>>> >>
>>> >> Hi all,
>>> >> I just tested gradle environment from a fresh source clone with this
>>> >> procedure with just a tiny change: I used "new project from existing
>>> >> sources" rather than create empty project and then add module.
>>> >>
>>> >> It works fine and junit runs from intellij also work.  with gradle we
>>> pay
>>> >> a 2s delay (on my machine) before running the actual test for each
>>> run. This
>>> >> delay seems quite constant no matter the module: I have run all the
>>> tests at
>>> >> once in  runner-core and a single test in another module with a
>>> similar
>>> >> delay.
>>> >>
>>> >> I also tested a debug session from intellij and it runs fine also.
>>> >>
>>> >> I'll do some more testing and keep you posted.
>>> >>
>>> >> I still have some concerns about the potential difficulty for new
>>> >> contributors to have to learn gradle in addition to Beam itself
>>> comparing to
>>> >> maven which is more mainstream for java developers. That could be
>>> >> discouraging for ex for part-time contributors
>>> >>
>>> >> Etienne
>>> >>
>>> >> Le mardi 10 avril 2018 à 14:38 +, Lukasz Cwik a écrit :
>>> >>
>>> >> beam-site PR/414 updates the instructions for using Intellij and how
>>> to
>>> >> import a module:
>>> >>
>>> >> 1. Create an empty IntelliJ project outside of the Beam source tree.
>>> >> 2. Under Project Structure > Project, select a Project SDK.
>>> >> 3. Under Project Structure > Modules, click the + sign to add a
>>> module and
>>> >>select "Import Module".
>>> >> 1. Select the directory containing the Beam source tree.
>>> >> 2. Tick the "Import module from external model" button and select
>>> >> Gradle
>>> >>from the list.
>>> >> 3. 

Re: Gradle Status [April 6]

[Alexey Romanenko]

I’ve managed to import a project as it’s described in documentation (starting 
from empty project) using Idea 2018 and run unit tests successfully. 
For some reasons, tests, that use DirectRunner to run a pipeline, were failed.

WBR,
Alexey

> On 11 Apr 2018, at 19:01, Daniel Oliveira  wrote:
> 
> Hi everyone, I was the one who initially wrote the PR with Idea instructions 
> . I was using 2017.3 as well 
> while writing it so all the instructions were tested on that version. I'll 
> try testing the instructions on 2018 to see if I can reproduce the issues 
> people are having.
> 
> On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik  > wrote:
> I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet.
> 
> On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau  > wrote:
> Any of you using the idea 2018? the import works for me but then it is
> not as smooth as it seems for you. I'm just trying to see if it is a
> procedure thing or a version issue.
> 
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> 
> 
> 2018-04-11 17:28 GMT+02:00 Kenneth Knowles  >:
> > The only reason I did "empty project then add a module" procedure was to get
> > all the IntelliJ files outside the source tree. IIRC directly creating from
> > existing sources didn't give the necessary configuration options. If you
> > don't care about being able to `git clean` then you can do the shorter
> > version. Also the particular UI for creation might have improved since I
> > tried it. I'll do it again.
> >
> > On the pom, since it is only generated for machine reading, why care if the
> > parent is inlined? I actually prefer to avoid coupling with things that you
> > have to go and look up. Using inheritance is OK for human edited poms
> > (actually IMO it is still a mistake) but it doesn't change the semantics of
> > a shipped pom if they are all immutable, which they should be. It is better
> > to have all the info right there. Is there an actually effective difference?
> >
> > Kenn
> >
> > On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot  > >
> > wrote:
> >>
> >> Hi all,
> >> I just tested gradle environment from a fresh source clone with this
> >> procedure with just a tiny change: I used "new project from existing
> >> sources" rather than create empty project and then add module.
> >>
> >> It works fine and junit runs from intellij also work.  with gradle we pay
> >> a 2s delay (on my machine) before running the actual test for each run. 
> >> This
> >> delay seems quite constant no matter the module: I have run all the tests 
> >> at
> >> once in  runner-core and a single test in another module with a similar
> >> delay.
> >>
> >> I also tested a debug session from intellij and it runs fine also.
> >>
> >> I'll do some more testing and keep you posted.
> >>
> >> I still have some concerns about the potential difficulty for new
> >> contributors to have to learn gradle in addition to Beam itself comparing 
> >> to
> >> maven which is more mainstream for java developers. That could be
> >> discouraging for ex for part-time contributors
> >>
> >> Etienne
> >>
> >> Le mardi 10 avril 2018 à 14:38 +, Lukasz Cwik a écrit :
> >>
> >> beam-site PR/414 updates the instructions for using Intellij and how to
> >> import a module:
> >>
> >> 1. Create an empty IntelliJ project outside of the Beam source tree.
> >> 2. Under Project Structure > Project, select a Project SDK.
> >> 3. Under Project Structure > Modules, click the + sign to add a module and
> >>select "Import Module".
> >> 1. Select the directory containing the Beam source tree.
> >> 2. Tick the "Import module from external model" button and select
> >> Gradle
> >>from the list.
> >> 3. Tick the following boxes.
> >>* Use auto-import
> >>* Create separate module per source set
> >>* Store generated project files externally
> >>* Use default gradle wrapper
> >> 4. Delegate build actions to Gradle by going to Settings > Build,
> >> Execution,
> >>Deployment > Build Tools > Gradle and checking "Delegate IDE build/run
> >>actions to gradle".
> >>
> >> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré  >> >
> >> wrote:
> >>
> >> That's a very important issue for contribution. Up to now, I used Maven
> >> for setup IntelliJ (and it works just fine). If we remove the pom.xml,
> >> we have to support Eclipse and IntelliJ "smoothly".
> >>
> >> Let me try in IntelliJ.
> >>
> >> Regards
> >> JB
> >>
> >> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
> >> > You dont have issue due to the build setup with that option. I get:
> >> >
> >> > avr. 10, 2018 3:20:10 PM
> >> > 

Re: Gradle Status [April 6]

[Daniel Oliveira]

Hi everyone, I was the one who initially wrote the PR with Idea instructions
. I was using 2017.3 as well
while writing it so all the instructions were tested on that version. I'll
try testing the instructions on 2018 to see if I can reproduce the issues
people are having.

On Wed, Apr 11, 2018 at 9:51 AM Lukasz Cwik  wrote:

> I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet.
>
> On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau 
> wrote:
>
>> Any of you using the idea 2018? the import works for me but then it is
>> not as smooth as it seems for you. I'm just trying to see if it is a
>> procedure thing or a version issue.
>>
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>
>>
>> 2018-04-11 17:28 GMT+02:00 Kenneth Knowles :
>> > The only reason I did "empty project then add a module" procedure was
>> to get
>> > all the IntelliJ files outside the source tree. IIRC directly creating
>> from
>> > existing sources didn't give the necessary configuration options. If you
>> > don't care about being able to `git clean` then you can do the shorter
>> > version. Also the particular UI for creation might have improved since I
>> > tried it. I'll do it again.
>> >
>> > On the pom, since it is only generated for machine reading, why care if
>> the
>> > parent is inlined? I actually prefer to avoid coupling with things that
>> you
>> > have to go and look up. Using inheritance is OK for human edited poms
>> > (actually IMO it is still a mistake) but it doesn't change the
>> semantics of
>> > a shipped pom if they are all immutable, which they should be. It is
>> better
>> > to have all the info right there. Is there an actually effective
>> difference?
>> >
>> > Kenn
>> >
>> > On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot 
>> > wrote:
>> >>
>> >> Hi all,
>> >> I just tested gradle environment from a fresh source clone with this
>> >> procedure with just a tiny change: I used "new project from existing
>> >> sources" rather than create empty project and then add module.
>> >>
>> >> It works fine and junit runs from intellij also work.  with gradle we
>> pay
>> >> a 2s delay (on my machine) before running the actual test for each
>> run. This
>> >> delay seems quite constant no matter the module: I have run all the
>> tests at
>> >> once in  runner-core and a single test in another module with a similar
>> >> delay.
>> >>
>> >> I also tested a debug session from intellij and it runs fine also.
>> >>
>> >> I'll do some more testing and keep you posted.
>> >>
>> >> I still have some concerns about the potential difficulty for new
>> >> contributors to have to learn gradle in addition to Beam itself
>> comparing to
>> >> maven which is more mainstream for java developers. That could be
>> >> discouraging for ex for part-time contributors
>> >>
>> >> Etienne
>> >>
>> >> Le mardi 10 avril 2018 à 14:38 +, Lukasz Cwik a écrit :
>> >>
>> >> beam-site PR/414 updates the instructions for using Intellij and how to
>> >> import a module:
>> >>
>> >> 1. Create an empty IntelliJ project outside of the Beam source tree.
>> >> 2. Under Project Structure > Project, select a Project SDK.
>> >> 3. Under Project Structure > Modules, click the + sign to add a module
>> and
>> >>select "Import Module".
>> >> 1. Select the directory containing the Beam source tree.
>> >> 2. Tick the "Import module from external model" button and select
>> >> Gradle
>> >>from the list.
>> >> 3. Tick the following boxes.
>> >>* Use auto-import
>> >>* Create separate module per source set
>> >>* Store generated project files externally
>> >>* Use default gradle wrapper
>> >> 4. Delegate build actions to Gradle by going to Settings > Build,
>> >> Execution,
>> >>Deployment > Build Tools > Gradle and checking "Delegate IDE
>> build/run
>> >>actions to gradle".
>> >>
>> >> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré > >
>> >> wrote:
>> >>
>> >> That's a very important issue for contribution. Up to now, I used Maven
>> >> for setup IntelliJ (and it works just fine). If we remove the pom.xml,
>> >> we have to support Eclipse and IntelliJ "smoothly".
>> >>
>> >> Let me try in IntelliJ.
>> >>
>> >> Regards
>> >> JB
>> >>
>> >> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
>> >> > You dont have issue due to the build setup with that option. I get:
>> >> >
>> >> > avr. 10, 2018 3:20:10 PM
>> >> > org.apache.beam.runners.direct.DirectTransformExecutor run
>> >> > GRAVE: Error occurred within
>> >> > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
>> >> > com.google.common.util.concurrent.ExecutionError:
>> >> > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
>> >> >
>> >> > ?
>> >> >
>> >> > Romain Manni-Bucau
>> >> > @rmannibucau |  Blog | Old Blog | Github 

Re: Gradle Status [April 6]

[Lukasz Cwik]

I use 2017.3 and it has been reliable for me. I haven't tried 2018 yet.

On Wed, Apr 11, 2018 at 11:30 AM Romain Manni-Bucau 
wrote:

> Any of you using the idea 2018? the import works for me but then it is
> not as smooth as it seems for you. I'm just trying to see if it is a
> procedure thing or a version issue.
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
>
> 2018-04-11 17:28 GMT+02:00 Kenneth Knowles :
> > The only reason I did "empty project then add a module" procedure was to
> get
> > all the IntelliJ files outside the source tree. IIRC directly creating
> from
> > existing sources didn't give the necessary configuration options. If you
> > don't care about being able to `git clean` then you can do the shorter
> > version. Also the particular UI for creation might have improved since I
> > tried it. I'll do it again.
> >
> > On the pom, since it is only generated for machine reading, why care if
> the
> > parent is inlined? I actually prefer to avoid coupling with things that
> you
> > have to go and look up. Using inheritance is OK for human edited poms
> > (actually IMO it is still a mistake) but it doesn't change the semantics
> of
> > a shipped pom if they are all immutable, which they should be. It is
> better
> > to have all the info right there. Is there an actually effective
> difference?
> >
> > Kenn
> >
> > On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot 
> > wrote:
> >>
> >> Hi all,
> >> I just tested gradle environment from a fresh source clone with this
> >> procedure with just a tiny change: I used "new project from existing
> >> sources" rather than create empty project and then add module.
> >>
> >> It works fine and junit runs from intellij also work.  with gradle we
> pay
> >> a 2s delay (on my machine) before running the actual test for each run.
> This
> >> delay seems quite constant no matter the module: I have run all the
> tests at
> >> once in  runner-core and a single test in another module with a similar
> >> delay.
> >>
> >> I also tested a debug session from intellij and it runs fine also.
> >>
> >> I'll do some more testing and keep you posted.
> >>
> >> I still have some concerns about the potential difficulty for new
> >> contributors to have to learn gradle in addition to Beam itself
> comparing to
> >> maven which is more mainstream for java developers. That could be
> >> discouraging for ex for part-time contributors
> >>
> >> Etienne
> >>
> >> Le mardi 10 avril 2018 à 14:38 +, Lukasz Cwik a écrit :
> >>
> >> beam-site PR/414 updates the instructions for using Intellij and how to
> >> import a module:
> >>
> >> 1. Create an empty IntelliJ project outside of the Beam source tree.
> >> 2. Under Project Structure > Project, select a Project SDK.
> >> 3. Under Project Structure > Modules, click the + sign to add a module
> and
> >>select "Import Module".
> >> 1. Select the directory containing the Beam source tree.
> >> 2. Tick the "Import module from external model" button and select
> >> Gradle
> >>from the list.
> >> 3. Tick the following boxes.
> >>* Use auto-import
> >>* Create separate module per source set
> >>* Store generated project files externally
> >>* Use default gradle wrapper
> >> 4. Delegate build actions to Gradle by going to Settings > Build,
> >> Execution,
> >>Deployment > Build Tools > Gradle and checking "Delegate IDE
> build/run
> >>actions to gradle".
> >>
> >> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré 
> >> wrote:
> >>
> >> That's a very important issue for contribution. Up to now, I used Maven
> >> for setup IntelliJ (and it works just fine). If we remove the pom.xml,
> >> we have to support Eclipse and IntelliJ "smoothly".
> >>
> >> Let me try in IntelliJ.
> >>
> >> Regards
> >> JB
> >>
> >> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
> >> > You dont have issue due to the build setup with that option. I get:
> >> >
> >> > avr. 10, 2018 3:20:10 PM
> >> > org.apache.beam.runners.direct.DirectTransformExecutor run
> >> > GRAVE: Error occurred within
> >> > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
> >> > com.google.common.util.concurrent.ExecutionError:
> >> > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
> >> >
> >> > ?
> >> >
> >> > Romain Manni-Bucau
> >> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >> >
> >> >
> >> > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik :
> >> >> I have found that the simplest setup is to delegate the build/test
> >> >> actions
> >> >> to Gradle. This allows you to run unit tests very easily and since
> its
> >> >> in
> >> >> the same manner that Gradle would have, you know that if its passing
> it
> >> >> will
> >> >> pass on the command line and on Jenkins. Here is one site that
> >> >> discusses how
> >> >> to set this up:
> >> >>
> >> >>
> 

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

Any of you using the idea 2018? the import works for me but then it is
not as smooth as it seems for you. I'm just trying to see if it is a
procedure thing or a version issue.

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-11 17:28 GMT+02:00 Kenneth Knowles :
> The only reason I did "empty project then add a module" procedure was to get
> all the IntelliJ files outside the source tree. IIRC directly creating from
> existing sources didn't give the necessary configuration options. If you
> don't care about being able to `git clean` then you can do the shorter
> version. Also the particular UI for creation might have improved since I
> tried it. I'll do it again.
>
> On the pom, since it is only generated for machine reading, why care if the
> parent is inlined? I actually prefer to avoid coupling with things that you
> have to go and look up. Using inheritance is OK for human edited poms
> (actually IMO it is still a mistake) but it doesn't change the semantics of
> a shipped pom if they are all immutable, which they should be. It is better
> to have all the info right there. Is there an actually effective difference?
>
> Kenn
>
> On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot 
> wrote:
>>
>> Hi all,
>> I just tested gradle environment from a fresh source clone with this
>> procedure with just a tiny change: I used "new project from existing
>> sources" rather than create empty project and then add module.
>>
>> It works fine and junit runs from intellij also work.  with gradle we pay
>> a 2s delay (on my machine) before running the actual test for each run. This
>> delay seems quite constant no matter the module: I have run all the tests at
>> once in  runner-core and a single test in another module with a similar
>> delay.
>>
>> I also tested a debug session from intellij and it runs fine also.
>>
>> I'll do some more testing and keep you posted.
>>
>> I still have some concerns about the potential difficulty for new
>> contributors to have to learn gradle in addition to Beam itself comparing to
>> maven which is more mainstream for java developers. That could be
>> discouraging for ex for part-time contributors
>>
>> Etienne
>>
>> Le mardi 10 avril 2018 à 14:38 +, Lukasz Cwik a écrit :
>>
>> beam-site PR/414 updates the instructions for using Intellij and how to
>> import a module:
>>
>> 1. Create an empty IntelliJ project outside of the Beam source tree.
>> 2. Under Project Structure > Project, select a Project SDK.
>> 3. Under Project Structure > Modules, click the + sign to add a module and
>>select "Import Module".
>> 1. Select the directory containing the Beam source tree.
>> 2. Tick the "Import module from external model" button and select
>> Gradle
>>from the list.
>> 3. Tick the following boxes.
>>* Use auto-import
>>* Create separate module per source set
>>* Store generated project files externally
>>* Use default gradle wrapper
>> 4. Delegate build actions to Gradle by going to Settings > Build,
>> Execution,
>>Deployment > Build Tools > Gradle and checking "Delegate IDE build/run
>>actions to gradle".
>>
>> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré 
>> wrote:
>>
>> That's a very important issue for contribution. Up to now, I used Maven
>> for setup IntelliJ (and it works just fine). If we remove the pom.xml,
>> we have to support Eclipse and IntelliJ "smoothly".
>>
>> Let me try in IntelliJ.
>>
>> Regards
>> JB
>>
>> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
>> > You dont have issue due to the build setup with that option. I get:
>> >
>> > avr. 10, 2018 3:20:10 PM
>> > org.apache.beam.runners.direct.DirectTransformExecutor run
>> > GRAVE: Error occurred within
>> > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
>> > com.google.common.util.concurrent.ExecutionError:
>> > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
>> >
>> > ?
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >
>> >
>> > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik :
>> >> I have found that the simplest setup is to delegate the build/test
>> >> actions
>> >> to Gradle. This allows you to run unit tests very easily and since its
>> >> in
>> >> the same manner that Gradle would have, you know that if its passing it
>> >> will
>> >> pass on the command line and on Jenkins. Here is one site that
>> >> discusses how
>> >> to set this up:
>> >>
>> >> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
>> >>
>> >>
>> >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau
>> >> 
>> >> wrote:
>> >>>
>> >>> What's the plan to make idea supporting gradle on beam project? Do we
>> >>> import the workaround mentionned in
>> >>> https://youtrack.jetbrains.com/issue/IDEA-175172?
>> >>> For the ones who didn't see this 

Re: Gradle Status [April 6]

[Kenneth Knowles]

The only reason I did "empty project then add a module" procedure was to
get all the IntelliJ files outside the source tree. IIRC directly creating
from existing sources didn't give the necessary configuration options. If
you don't care about being able to `git clean` then you can do the shorter
version. Also the particular UI for creation might have improved since I
tried it. I'll do it again.

On the pom, since it is only generated for machine reading, why care if the
parent is inlined? I actually prefer to avoid coupling with things that you
have to go and look up. Using inheritance is OK for human edited poms
(actually IMO it is still a mistake) but it doesn't change the semantics of
a shipped pom if they are all immutable, which they should be. It is better
to have all the info right there. Is there an actually effective difference?

Kenn

On Wed, Apr 11, 2018 at 6:53 AM Etienne Chauchot 
wrote:

> Hi all,
> I just tested gradle environment from a fresh source clone with this
> procedure with just a tiny change: I used "new project from existing
> sources" rather than create empty project and then add module.
>
> It works fine and junit runs from intellij also work.  with gradle we pay
> a 2s delay (on my machine) before running the actual test for each run.
> This delay seems quite constant no matter the module: I have run all the
> tests at once in  runner-core and a single test in another module with a
> similar delay.
>
> I also tested a debug session from intellij and it runs fine also.
>
> I'll do some more testing and keep you posted.
>
> I still have some concerns about the potential difficulty for new
> contributors to have to learn gradle in addition to Beam itself comparing
> to maven which is more mainstream for java developers. That could be
> discouraging for ex for part-time contributors
>
> Etienne
>
> Le mardi 10 avril 2018 à 14:38 +, Lukasz Cwik a écrit :
>
> beam-site PR/414 updates the instructions for using Intellij and how to
> import a module:
>
> 1. Create an empty IntelliJ project outside of the Beam source tree.
> 2. Under Project Structure > Project, select a Project SDK.
> 3. Under Project Structure > Modules, click the + sign to add a module and
>select "Import Module".
> 1. Select the directory containing the Beam source tree.
> 2. Tick the "Import module from external model" button and select
> Gradle
>from the list.
> 3. Tick the following boxes.
>* Use auto-import
>* Create separate module per source set
>* Store generated project files externally
>* Use default gradle wrapper
> 4. Delegate build actions to Gradle by going to Settings > Build,
> Execution,
>Deployment > Build Tools > Gradle and checking "Delegate IDE build/run
>actions to gradle".
>
> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré 
> wrote:
>
> That's a very important issue for contribution. Up to now, I used Maven
> for setup IntelliJ (and it works just fine). If we remove the pom.xml,
> we have to support Eclipse and IntelliJ "smoothly".
>
> Let me try in IntelliJ.
>
> Regards
> JB
>
> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
> > You dont have issue due to the build setup with that option. I get:
> >
> > avr. 10, 2018 3:20:10 PM
> > org.apache.beam.runners.direct.DirectTransformExecutor run
> > GRAVE: Error occurred within
> > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
> > com.google.common.util.concurrent.ExecutionError:
> > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
> >
> > ?
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> >
> > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik :
> >> I have found that the simplest setup is to delegate the build/test
> actions
> >> to Gradle. This allows you to run unit tests very easily and since its
> in
> >> the same manner that Gradle would have, you know that if its passing it
> will
> >> pass on the command line and on Jenkins. Here is one site that
> discusses how
> >> to set this up:
> >>
> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
> >>
> >>
> >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau <
> rmannibu...@gmail.com>
> >> wrote:
> >>>
> >>> What's the plan to make idea supporting gradle on beam project? Do we
> >>> import the workaround mentionned in
> >>> https://youtrack.jetbrains.com/issue/IDEA-175172?
> >>> For the ones who didn't see this issue in action: idea will compile in
> >>> out/ instead of build/ and you will just miss all the resources you
> >>> need like some SPI registration which are used by all our registrar =>
> >>> no way to run tests in idea without hacking the configuration quite
> >>> deeply :(
> >>>
> >>> Romain Manni-Bucau
> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >>>
> >>>
> >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot :
>  

Re: Gradle Status [April 6]

[Etienne Chauchot]

Hi all,
I just tested gradle environment from a fresh source clone with this procedure 
with just a tiny change: I used "new
project from existing sources" rather than create empty project and then add 
module. 
It works fine and junit runs from intellij also work.  with gradle we pay a 2s 
delay (on my machine) before running the
actual test for each run. This delay seems quite constant no matter the module: 
I have run all the tests at once in
 runner-core and a single test in another module with a similar delay. 
I also tested a debug session from intellij and it runs fine also.
I'll do some more testing and keep you posted.
I still have some concerns about the potential difficulty for new contributors 
to have to learn gradle in addition to
Beam itself comparing to maven which is more mainstream for java developers. 
That could be discouraging for ex for part-
time contributors
Etienne
Le mardi 10 avril 2018 à 14:38 +, Lukasz Cwik a écrit :
> beam-site PR/414 updates the instructions for using Intellij and how to 
> import a module:
> 
> 1. Create an empty IntelliJ project outside of the Beam source tree.
> 2. Under Project Structure > Project, select a Project SDK.
> 3. Under Project Structure > Modules, click the + sign to add a module and
>    select "Import Module".
>     1. Select the directory containing the Beam source tree.
>     2. Tick the "Import module from external model" button and select Gradle
>        from the list.
>     3. Tick the following boxes.
>        * Use auto-import
>        * Create separate module per source set
>        * Store generated project files externally
>        * Use default gradle wrapper
> 4. Delegate build actions to Gradle by going to Settings > Build, Execution,
>    Deployment > Build Tools > Gradle and checking "Delegate IDE build/run
>    actions to gradle".
> 
> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré  
> wrote:
> > That's a very important issue for contribution. Up to now, I used Maven
> > for setup IntelliJ (and it works just fine). If we remove the pom.xml,
> > we have to support Eclipse and IntelliJ "smoothly".
> > 
> > Let me try in IntelliJ.
> > 
> > Regards
> > JB
> > 
> > On 10/04/2018 15:21, Romain Manni-Bucau wrote:
> > > You dont have issue due to the build setup with that option. I get:
> > >
> > > avr. 10, 2018 3:20:10 PM
> > > org.apache.beam.runners.direct.DirectTransformExecutor run
> > > GRAVE: Error occurred within
> > > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
> > > com.google.common.util.concurrent.ExecutionError:
> > > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
> > >
> > > ?
> > >
> > > Romain Manni-Bucau
> > > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> > >
> > >
> > > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik :
> > >> I have found that the simplest setup is to delegate the build/test 
> > >> actions
> > >> to Gradle. This allows you to run unit tests very easily and since its in
> > >> the same manner that Gradle would have, you know that if its passing it 
> > >> will
> > >> pass on the command line and on Jenkins. Here is one site that discusses 
> > >> how
> > >> to set this up:
> > >> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
> > >>
> > >>
> > >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau 
> > >> 
> > >> wrote:
> > >>>
> > >>> What's the plan to make idea supporting gradle on beam project? Do we
> > >>> import the workaround mentionned in
> > >>> https://youtrack.jetbrains.com/issue/IDEA-175172?
> > >>> For the ones who didn't see this issue in action: idea will compile in
> > >>> out/ instead of build/ and you will just miss all the resources you
> > >>> need like some SPI registration which are used by all our registrar =>
> > >>> no way to run tests in idea without hacking the configuration quite
> > >>> deeply :(
> > >>>
> > >>> Romain Manni-Bucau
> > >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> > >>>
> > >>>
> > >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot :
> >  As a gradle beginner, I could not agree more !
> >  +1
> >  Etienne
> >  Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :
> > 
> >  Hi all,
> > 
> >  I did multiple gradle build since last week and I would like to share
> >  one of my concern: it's about the communities.
> > 
> >  If I think our users won't see any change for them due to Gradle build
> >  (I think that most of our users will still use Maven with artifacts
> >  provided by Gradle), I'm more concerned by the dev community and the
> >  contribution.
> > 
> >  Maven is well known and straight forward for a large part of potential
> >  contributors. I think we have to keep in mind that we still have to 
> >  grow
> >  up our contributors community.
> > 
> >  Today, maybe I'm 

Re: Gradle Status [April 6]

[Lukasz Cwik]

Romain, it seems like you want us to generate an exact copy of the existing
poms that we have. There is a lot of stuff in there related to
building/testing the Apache Beam project which our users don't need
(integration/test profiles, test scope dependencies, parents, ...). Users
who want to build the project will need to use the same tools as the
community and build from the source tarball.

On Wed, Apr 11, 2018 at 8:48 AM Romain Manni-Bucau 
wrote:

> Hi guys,
>
> checked the snapshot pom today and there are issues:
>
> 1. no parent (so 2 is hard to handle, no properties etc)
> 2. too much noise informations (all the parent data should be in the
> parent)

3. wrong scopes (all is compile) - which is likely a leak of gradle scripts
>
4. missing important configuration (properties and or plugin) like the
> ones defining the java versions




> I didn't check the profiles but it makes already some work ;)
>
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
>
> 2018-04-11 6:50 GMT+02:00 Romain Manni-Bucau :
> >
> >
> > Le 11 avr. 2018 02:30, "Reuven Lax"  a écrit :
> >
> > Actually I always found the right-click to run tests to only sometimes
> work
> > in Maven, especially if there were changes to dependent AutoValue classes
> > where code had to be generated. Too often it would fail, and I would then
> > need to use Maven to rebuild the whole project. It would be cool if
> Gradle
> > could do this more reliably than Maven did.
> >
> >
> > Hmm, i exactly experiment the opposite. Testes with idea 2017 and 2018.
> >
> >
> > Reuven
> >
> > On Tue, Apr 10, 2018 at 8:46 PM Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >>
> >> @jb: what did you change? I re-imported the project like 3 times earlier
> >> today and never got it working acceptably :(
> >>
> >> Personally if importing the project and right click on a test+debug
> works
> >> as good as maven in idea id be happy. I can manage other stuff in a
> console
> >> even if gradle reporting is not that efficient for me for now.
> >>
> >> Le 10 avr. 2018 21:37, "Reuven Lax"  a écrit :
> >>>
> >>> There are a lot of ideas on how to increase usability, but I think
> >>> they'll get lost in the thread. I suggest we try to capture them in
> Jiras.
> >>>
> >>> I suggest we also find out what common use patterns are (people on this
> >>> thread are probably sufficient), as different people will have
> different
> >>> workflows. We can then make sure that all common workflows are
> documented.
> >>> As an example, one task I often do is to run just checkstyle over a
> module
> >>> or the entire project.
> >>>
> >>> Reuven
> >>>
> >>> On Tue, Apr 10, 2018 at 7:18 PM Jean-Baptiste Onofré 
> >>> wrote:
> 
>  FYI, I did a new attempt and it works fine (pretty long). Previous try
>  failed.
> 
>  Regards
>  JB
> 
>  On 10/04/2018 19:52, Kenneth Knowles wrote:
>  > I've been on Idea+Gradle for ~two months, around the time I added
>  > https://github.com/apache/beam/pull/4583 and
>  > https://github.com/apache/beam/pull/4626 to make the import require
>  > zero
>  > user work. I have no fear of deleting my project any time and
>  > re-importing.
>  >
>  > I agree with not having auto-import on. It is just too slow. I can't
>  > remember if it was importing too often due to build outputs or if it
>  > was
>  > just that I was messing with the build.gradle files. Anyhow it
> doesn't
>  > really add much value.
>  >
>  > The gradle runner _is_ able to use submodules and run individual
> tests
>  > methods, and all that.
>  >
>  > Kenn
>  >
>  >
>  > On Tue, Apr 10, 2018 at 9:31 AM Romain Manni-Bucau
>  > > wrote:
>  >
>  > Runner a test doesnt have the right classpath (idea uses out/
>  > instead
>  > of build/) then when you switch on gradle runner the launching
>  > uses
>  > gradle which is not able to use submodules directly but
> reconsider
>  > the
>  > whole project which is quite slow for normal dev iterations
>  > compare to just run the test with the right classpath and a fast
>  > compile step if needed. I lost literally 1h for something simple
>  > with
>  > that tooling, this is way too much to be acceptable on my side
>  > since
>  > I'm sadly not paid to work on beam (one day maybe ;)).
>  >
>  > Romain Manni-Bucau
>  > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>  >
>  >
>  > 2018-04-10 18:27 GMT+02:00 Reuven Lax   > >:
>  >  > Romain,
>  >  >
>  >  > Can you detail what's not working. I switched my IntelliJ
> over
>  > to
>  >   

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

Hi guys,

checked the snapshot pom today and there are issues:

1. no parent (so 2 is hard to handle, no properties etc)
2. too much noise informations (all the parent data should be in the parent)
3. wrong scopes (all is compile) - which is likely a leak of gradle scripts
4. missing important configuration (properties and or plugin) like the
ones defining the java versions

I didn't check the profiles but it makes already some work ;)


Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-11 6:50 GMT+02:00 Romain Manni-Bucau :
>
>
> Le 11 avr. 2018 02:30, "Reuven Lax"  a écrit :
>
> Actually I always found the right-click to run tests to only sometimes work
> in Maven, especially if there were changes to dependent AutoValue classes
> where code had to be generated. Too often it would fail, and I would then
> need to use Maven to rebuild the whole project. It would be cool if Gradle
> could do this more reliably than Maven did.
>
>
> Hmm, i exactly experiment the opposite. Testes with idea 2017 and 2018.
>
>
> Reuven
>
> On Tue, Apr 10, 2018 at 8:46 PM Romain Manni-Bucau 
> wrote:
>>
>> @jb: what did you change? I re-imported the project like 3 times earlier
>> today and never got it working acceptably :(
>>
>> Personally if importing the project and right click on a test+debug works
>> as good as maven in idea id be happy. I can manage other stuff in a console
>> even if gradle reporting is not that efficient for me for now.
>>
>> Le 10 avr. 2018 21:37, "Reuven Lax"  a écrit :
>>>
>>> There are a lot of ideas on how to increase usability, but I think
>>> they'll get lost in the thread. I suggest we try to capture them in Jiras.
>>>
>>> I suggest we also find out what common use patterns are (people on this
>>> thread are probably sufficient), as different people will have different
>>> workflows. We can then make sure that all common workflows are documented.
>>> As an example, one task I often do is to run just checkstyle over a module
>>> or the entire project.
>>>
>>> Reuven
>>>
>>> On Tue, Apr 10, 2018 at 7:18 PM Jean-Baptiste Onofré 
>>> wrote:

 FYI, I did a new attempt and it works fine (pretty long). Previous try
 failed.

 Regards
 JB

 On 10/04/2018 19:52, Kenneth Knowles wrote:
 > I've been on Idea+Gradle for ~two months, around the time I added
 > https://github.com/apache/beam/pull/4583 and
 > https://github.com/apache/beam/pull/4626 to make the import require
 > zero
 > user work. I have no fear of deleting my project any time and
 > re-importing.
 >
 > I agree with not having auto-import on. It is just too slow. I can't
 > remember if it was importing too often due to build outputs or if it
 > was
 > just that I was messing with the build.gradle files. Anyhow it doesn't
 > really add much value.
 >
 > The gradle runner _is_ able to use submodules and run individual tests
 > methods, and all that.
 >
 > Kenn
 >
 >
 > On Tue, Apr 10, 2018 at 9:31 AM Romain Manni-Bucau
 > > wrote:
 >
 > Runner a test doesnt have the right classpath (idea uses out/
 > instead
 > of build/) then when you switch on gradle runner the launching
 > uses
 > gradle which is not able to use submodules directly but reconsider
 > the
 > whole project which is quite slow for normal dev iterations
 > compare to just run the test with the right classpath and a fast
 > compile step if needed. I lost literally 1h for something simple
 > with
 > that tooling, this is way too much to be acceptable on my side
 > since
 > I'm sadly not paid to work on beam (one day maybe ;)).
 >
 > Romain Manni-Bucau
 > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
 >
 >
 > 2018-04-10 18:27 GMT+02:00 Reuven Lax  >:
 >  > Romain,
 >  >
 >  > Can you detail what's not working. I switched my IntelliJ over
 > to
 > Gradle
 >  > about two weeks ago, and haven't had any trouble.
 >  >
 >  > Reuven
 >  >
 >  > On Tue, Apr 10, 2018 at 4:20 PM Romain Manni-Bucau
 > >
 >  > wrote:
 >  >>
 >  >> Ok, didn't find a way to make it working properly (only
 > workaround
 >  >> with direct commands and no good idea integration for
 > debugging). I'm
 >  >> back with maven, if anyone knows how to properly solve it
 > let's
 > do it.
 >  >> If not I think JB point is to consider more than any other
 > criteria.
 >  >>
 >  >> Romain Manni-Bucau
 >  

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

Le 11 avr. 2018 02:30, "Reuven Lax"  a écrit :

Actually I always found the right-click to run tests to only sometimes work
in Maven, especially if there were changes to dependent AutoValue classes
where code had to be generated. Too often it would fail, and I would then
need to use Maven to rebuild the whole project. It would be cool if Gradle
could do this more reliably than Maven did.


Hmm, i exactly experiment the opposite. Testes with idea 2017 and 2018.


Reuven

On Tue, Apr 10, 2018 at 8:46 PM Romain Manni-Bucau 
wrote:

> @jb: what did you change? I re-imported the project like 3 times earlier
> today and never got it working acceptably :(
>
> Personally if importing the project and right click on a test+debug works
> as good as maven in idea id be happy. I can manage other stuff in a console
> even if gradle reporting is not that efficient for me for now.
>
> Le 10 avr. 2018 21:37, "Reuven Lax"  a écrit :
>
>> There are a lot of ideas on how to increase usability, but I think
>> they'll get lost in the thread. I suggest we try to capture them in Jiras.
>>
>> I suggest we also find out what common use patterns are (people on this
>> thread are probably sufficient), as different people will have different
>> workflows. We can then make sure that all common workflows are documented.
>> As an example, one task I often do is to run just checkstyle over a module
>> or the entire project.
>>
>> Reuven
>>
>> On Tue, Apr 10, 2018 at 7:18 PM Jean-Baptiste Onofré 
>> wrote:
>>
>>> FYI, I did a new attempt and it works fine (pretty long). Previous try
>>> failed.
>>>
>>> Regards
>>> JB
>>>
>>> On 10/04/2018 19:52, Kenneth Knowles wrote:
>>> > I've been on Idea+Gradle for ~two months, around the time I added
>>> > https://github.com/apache/beam/pull/4583 and
>>> > https://github.com/apache/beam/pull/4626 to make the import require
>>> zero
>>> > user work. I have no fear of deleting my project any time and
>>> re-importing.
>>> >
>>> > I agree with not having auto-import on. It is just too slow. I can't
>>> > remember if it was importing too often due to build outputs or if it
>>> was
>>> > just that I was messing with the build.gradle files. Anyhow it doesn't
>>> > really add much value.
>>> >
>>> > The gradle runner _is_ able to use submodules and run individual tests
>>> > methods, and all that.
>>> >
>>> > Kenn
>>> >
>>> >
>>> > On Tue, Apr 10, 2018 at 9:31 AM Romain Manni-Bucau
>>> > > wrote:
>>> >
>>> > Runner a test doesnt have the right classpath (idea uses out/
>>> instead
>>> > of build/) then when you switch on gradle runner the launching uses
>>> > gradle which is not able to use submodules directly but reconsider
>>> the
>>> > whole project which is quite slow for normal dev iterations
>>> > compare to just run the test with the right classpath and a fast
>>> > compile step if needed. I lost literally 1h for something simple
>>> with
>>> > that tooling, this is way too much to be acceptable on my side
>>> since
>>> > I'm sadly not paid to work on beam (one day maybe ;)).
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >
>>> >
>>> > 2018-04-10 18:27 GMT+02:00 Reuven Lax >> > >:
>>> >  > Romain,
>>> >  >
>>> >  > Can you detail what's not working. I switched my IntelliJ over
>>> to
>>> > Gradle
>>> >  > about two weeks ago, and haven't had any trouble.
>>> >  >
>>> >  > Reuven
>>> >  >
>>> >  > On Tue, Apr 10, 2018 at 4:20 PM Romain Manni-Bucau
>>> > >
>>> >  > wrote:
>>> >  >>
>>> >  >> Ok, didn't find a way to make it working properly (only
>>> workaround
>>> >  >> with direct commands and no good idea integration for
>>> > debugging). I'm
>>> >  >> back with maven, if anyone knows how to properly solve it let's
>>> > do it.
>>> >  >> If not I think JB point is to consider more than any other
>>> criteria.
>>> >  >>
>>> >  >> Romain Manni-Bucau
>>> >  >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >  >>
>>> >  >>
>>> >  >> 2018-04-10 16:41 GMT+02:00 Romain Manni-Bucau
>>> > >:
>>> >  >> > side note: do NOT use auto-import until you are sure you can,
>>> > it locks
>>> >  >> > regularly on beam (pby too big for idea?) and makes idea
>>> ready
>>> > to be
>>> >  >> > killed :(
>>> >  >> >
>>> >  >> > Romain Manni-Bucau
>>> >  >> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >  >> >
>>> >  >> >
>>> >  >> > 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré
>>> > >:
>>> >  >> >> It's what I did, 

Re: Gradle Status [April 6]

[Reuven Lax]

Actually I always found the right-click to run tests to only sometimes work
in Maven, especially if there were changes to dependent AutoValue classes
where code had to be generated. Too often it would fail, and I would then
need to use Maven to rebuild the whole project. It would be cool if Gradle
could do this more reliably than Maven did.

Reuven

On Tue, Apr 10, 2018 at 8:46 PM Romain Manni-Bucau 
wrote:

> @jb: what did you change? I re-imported the project like 3 times earlier
> today and never got it working acceptably :(
>
> Personally if importing the project and right click on a test+debug works
> as good as maven in idea id be happy. I can manage other stuff in a console
> even if gradle reporting is not that efficient for me for now.
>
> Le 10 avr. 2018 21:37, "Reuven Lax"  a écrit :
>
>> There are a lot of ideas on how to increase usability, but I think
>> they'll get lost in the thread. I suggest we try to capture them in Jiras.
>>
>> I suggest we also find out what common use patterns are (people on this
>> thread are probably sufficient), as different people will have different
>> workflows. We can then make sure that all common workflows are documented.
>> As an example, one task I often do is to run just checkstyle over a module
>> or the entire project.
>>
>> Reuven
>>
>> On Tue, Apr 10, 2018 at 7:18 PM Jean-Baptiste Onofré 
>> wrote:
>>
>>> FYI, I did a new attempt and it works fine (pretty long). Previous try
>>> failed.
>>>
>>> Regards
>>> JB
>>>
>>> On 10/04/2018 19:52, Kenneth Knowles wrote:
>>> > I've been on Idea+Gradle for ~two months, around the time I added
>>> > https://github.com/apache/beam/pull/4583 and
>>> > https://github.com/apache/beam/pull/4626 to make the import require
>>> zero
>>> > user work. I have no fear of deleting my project any time and
>>> re-importing.
>>> >
>>> > I agree with not having auto-import on. It is just too slow. I can't
>>> > remember if it was importing too often due to build outputs or if it
>>> was
>>> > just that I was messing with the build.gradle files. Anyhow it doesn't
>>> > really add much value.
>>> >
>>> > The gradle runner _is_ able to use submodules and run individual tests
>>> > methods, and all that.
>>> >
>>> > Kenn
>>> >
>>> >
>>> > On Tue, Apr 10, 2018 at 9:31 AM Romain Manni-Bucau
>>> > > wrote:
>>> >
>>> > Runner a test doesnt have the right classpath (idea uses out/
>>> instead
>>> > of build/) then when you switch on gradle runner the launching uses
>>> > gradle which is not able to use submodules directly but reconsider
>>> the
>>> > whole project which is quite slow for normal dev iterations
>>> > compare to just run the test with the right classpath and a fast
>>> > compile step if needed. I lost literally 1h for something simple
>>> with
>>> > that tooling, this is way too much to be acceptable on my side
>>> since
>>> > I'm sadly not paid to work on beam (one day maybe ;)).
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >
>>> >
>>> > 2018-04-10 18:27 GMT+02:00 Reuven Lax >> > >:
>>> >  > Romain,
>>> >  >
>>> >  > Can you detail what's not working. I switched my IntelliJ over
>>> to
>>> > Gradle
>>> >  > about two weeks ago, and haven't had any trouble.
>>> >  >
>>> >  > Reuven
>>> >  >
>>> >  > On Tue, Apr 10, 2018 at 4:20 PM Romain Manni-Bucau
>>> > >
>>> >  > wrote:
>>> >  >>
>>> >  >> Ok, didn't find a way to make it working properly (only
>>> workaround
>>> >  >> with direct commands and no good idea integration for
>>> > debugging). I'm
>>> >  >> back with maven, if anyone knows how to properly solve it let's
>>> > do it.
>>> >  >> If not I think JB point is to consider more than any other
>>> criteria.
>>> >  >>
>>> >  >> Romain Manni-Bucau
>>> >  >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >  >>
>>> >  >>
>>> >  >> 2018-04-10 16:41 GMT+02:00 Romain Manni-Bucau
>>> > >:
>>> >  >> > side note: do NOT use auto-import until you are sure you can,
>>> > it locks
>>> >  >> > regularly on beam (pby too big for idea?) and makes idea
>>> ready
>>> > to be
>>> >  >> > killed :(
>>> >  >> >
>>> >  >> > Romain Manni-Bucau
>>> >  >> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >  >> >
>>> >  >> >
>>> >  >> > 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré
>>> > >:
>>> >  >> >> It's what I did, I'm trying a complete reload now (maybe
>>> this
>>> > step
>>> >  >> >> failed).
>>> >  >> >>
>>> >  >> >> On 10/04/2018 

Re: Gradle Status [April 6]

[Kenneth Knowles]

Reuven's point is good.

Once we hit the bare minimum of having things working, let's collect
usability improvements and engineering improvements on a separate JIRA from
the main migration.

I filed https://issues.apache.org/jira/browse/BEAM-4045 for these less
critical issues to separate them from blockers on
https://issues.apache.org/jira/browse/BEAM-3249.

Once we are through the migration, I expect the subtasks might just be
flattened to top-level tasks, but this is a nice view of them.

Kenn

On Tue, Apr 10, 2018 at 1:46 PM Romain Manni-Bucau 
wrote:

> @jb: what did you change? I re-imported the project like 3 times earlier
> today and never got it working acceptably :(
>
> Personally if importing the project and right click on a test+debug works
> as good as maven in idea id be happy. I can manage other stuff in a console
> even if gradle reporting is not that efficient for me for now.
>
> Le 10 avr. 2018 21:37, "Reuven Lax"  a écrit :
>
>> There are a lot of ideas on how to increase usability, but I think
>> they'll get lost in the thread. I suggest we try to capture them in Jiras.
>>
>> I suggest we also find out what common use patterns are (people on this
>> thread are probably sufficient), as different people will have different
>> workflows. We can then make sure that all common workflows are documented.
>> As an example, one task I often do is to run just checkstyle over a module
>> or the entire project.
>>
>> Reuven
>>
>> On Tue, Apr 10, 2018 at 7:18 PM Jean-Baptiste Onofré 
>> wrote:
>>
>>> FYI, I did a new attempt and it works fine (pretty long). Previous try
>>> failed.
>>>
>>> Regards
>>> JB
>>>
>>> On 10/04/2018 19:52, Kenneth Knowles wrote:
>>> > I've been on Idea+Gradle for ~two months, around the time I added
>>> > https://github.com/apache/beam/pull/4583 and
>>> > https://github.com/apache/beam/pull/4626 to make the import require
>>> zero
>>> > user work. I have no fear of deleting my project any time and
>>> re-importing.
>>> >
>>> > I agree with not having auto-import on. It is just too slow. I can't
>>> > remember if it was importing too often due to build outputs or if it
>>> was
>>> > just that I was messing with the build.gradle files. Anyhow it doesn't
>>> > really add much value.
>>> >
>>> > The gradle runner _is_ able to use submodules and run individual tests
>>> > methods, and all that.
>>> >
>>> > Kenn
>>> >
>>> >
>>> > On Tue, Apr 10, 2018 at 9:31 AM Romain Manni-Bucau
>>> > > wrote:
>>> >
>>> > Runner a test doesnt have the right classpath (idea uses out/
>>> instead
>>> > of build/) then when you switch on gradle runner the launching uses
>>> > gradle which is not able to use submodules directly but reconsider
>>> the
>>> > whole project which is quite slow for normal dev iterations
>>> > compare to just run the test with the right classpath and a fast
>>> > compile step if needed. I lost literally 1h for something simple
>>> with
>>> > that tooling, this is way too much to be acceptable on my side
>>> since
>>> > I'm sadly not paid to work on beam (one day maybe ;)).
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >
>>> >
>>> > 2018-04-10 18:27 GMT+02:00 Reuven Lax >> > >:
>>> >  > Romain,
>>> >  >
>>> >  > Can you detail what's not working. I switched my IntelliJ over
>>> to
>>> > Gradle
>>> >  > about two weeks ago, and haven't had any trouble.
>>> >  >
>>> >  > Reuven
>>> >  >
>>> >  > On Tue, Apr 10, 2018 at 4:20 PM Romain Manni-Bucau
>>> > >
>>> >  > wrote:
>>> >  >>
>>> >  >> Ok, didn't find a way to make it working properly (only
>>> workaround
>>> >  >> with direct commands and no good idea integration for
>>> > debugging). I'm
>>> >  >> back with maven, if anyone knows how to properly solve it let's
>>> > do it.
>>> >  >> If not I think JB point is to consider more than any other
>>> criteria.
>>> >  >>
>>> >  >> Romain Manni-Bucau
>>> >  >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >  >>
>>> >  >>
>>> >  >> 2018-04-10 16:41 GMT+02:00 Romain Manni-Bucau
>>> > >:
>>> >  >> > side note: do NOT use auto-import until you are sure you can,
>>> > it locks
>>> >  >> > regularly on beam (pby too big for idea?) and makes idea
>>> ready
>>> > to be
>>> >  >> > killed :(
>>> >  >> >
>>> >  >> > Romain Manni-Bucau
>>> >  >> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >  >> >
>>> >  >> >
>>> >  >> > 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré
>>> > >:
>>> >  >> >> It's what 

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

@jb: what did you change? I re-imported the project like 3 times earlier
today and never got it working acceptably :(

Personally if importing the project and right click on a test+debug works
as good as maven in idea id be happy. I can manage other stuff in a console
even if gradle reporting is not that efficient for me for now.

Le 10 avr. 2018 21:37, "Reuven Lax"  a écrit :

> There are a lot of ideas on how to increase usability, but I think they'll
> get lost in the thread. I suggest we try to capture them in Jiras.
>
> I suggest we also find out what common use patterns are (people on this
> thread are probably sufficient), as different people will have different
> workflows. We can then make sure that all common workflows are documented.
> As an example, one task I often do is to run just checkstyle over a module
> or the entire project.
>
> Reuven
>
> On Tue, Apr 10, 2018 at 7:18 PM Jean-Baptiste Onofré 
> wrote:
>
>> FYI, I did a new attempt and it works fine (pretty long). Previous try
>> failed.
>>
>> Regards
>> JB
>>
>> On 10/04/2018 19:52, Kenneth Knowles wrote:
>> > I've been on Idea+Gradle for ~two months, around the time I added
>> > https://github.com/apache/beam/pull/4583 and
>> > https://github.com/apache/beam/pull/4626 to make the import require
>> zero
>> > user work. I have no fear of deleting my project any time and
>> re-importing.
>> >
>> > I agree with not having auto-import on. It is just too slow. I can't
>> > remember if it was importing too often due to build outputs or if it
>> was
>> > just that I was messing with the build.gradle files. Anyhow it doesn't
>> > really add much value.
>> >
>> > The gradle runner _is_ able to use submodules and run individual tests
>> > methods, and all that.
>> >
>> > Kenn
>> >
>> >
>> > On Tue, Apr 10, 2018 at 9:31 AM Romain Manni-Bucau
>> > > wrote:
>> >
>> > Runner a test doesnt have the right classpath (idea uses out/
>> instead
>> > of build/) then when you switch on gradle runner the launching uses
>> > gradle which is not able to use submodules directly but reconsider
>> the
>> > whole project which is quite slow for normal dev iterations
>> > compare to just run the test with the right classpath and a fast
>> > compile step if needed. I lost literally 1h for something simple
>> with
>> > that tooling, this is way too much to be acceptable on my side since
>> > I'm sadly not paid to work on beam (one day maybe ;)).
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >
>> >
>> > 2018-04-10 18:27 GMT+02:00 Reuven Lax > > >:
>> >  > Romain,
>> >  >
>> >  > Can you detail what's not working. I switched my IntelliJ over to
>> > Gradle
>> >  > about two weeks ago, and haven't had any trouble.
>> >  >
>> >  > Reuven
>> >  >
>> >  > On Tue, Apr 10, 2018 at 4:20 PM Romain Manni-Bucau
>> > >
>> >  > wrote:
>> >  >>
>> >  >> Ok, didn't find a way to make it working properly (only
>> workaround
>> >  >> with direct commands and no good idea integration for
>> > debugging). I'm
>> >  >> back with maven, if anyone knows how to properly solve it let's
>> > do it.
>> >  >> If not I think JB point is to consider more than any other
>> criteria.
>> >  >>
>> >  >> Romain Manni-Bucau
>> >  >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >  >>
>> >  >>
>> >  >> 2018-04-10 16:41 GMT+02:00 Romain Manni-Bucau
>> > >:
>> >  >> > side note: do NOT use auto-import until you are sure you can,
>> > it locks
>> >  >> > regularly on beam (pby too big for idea?) and makes idea ready
>> > to be
>> >  >> > killed :(
>> >  >> >
>> >  >> > Romain Manni-Bucau
>> >  >> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >  >> >
>> >  >> >
>> >  >> > 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré
>> > >:
>> >  >> >> It's what I did, I'm trying a complete reload now (maybe this
>> > step
>> >  >> >> failed).
>> >  >> >>
>> >  >> >> On 10/04/2018 16:38, Lukasz Cwik wrote:
>> >  >> >>>
>> >  >> >>> beam-site PR/414 updates the instructions for using Intellij
>> > and how
>> >  >> >>> to
>> >  >> >>> import a module:
>> >  >> >>>
>> >  >> >>> 1. Create an empty IntelliJ project outside of the Beam
>> > source tree.
>> >  >> >>> 2. Under Project Structure > Project, select a Project SDK.
>> >  >> >>> 3. Under Project Structure > Modules, click the + sign to
>> > add a module
>> >  >> >>> and
>> >  >> >>> select "Import Module".
>> >  >> >>>  1. Select 

Re: Gradle Status [April 6]

[Reuven Lax]

There are a lot of ideas on how to increase usability, but I think they'll
get lost in the thread. I suggest we try to capture them in Jiras.

I suggest we also find out what common use patterns are (people on this
thread are probably sufficient), as different people will have different
workflows. We can then make sure that all common workflows are documented.
As an example, one task I often do is to run just checkstyle over a module
or the entire project.

Reuven

On Tue, Apr 10, 2018 at 7:18 PM Jean-Baptiste Onofré 
wrote:

> FYI, I did a new attempt and it works fine (pretty long). Previous try
> failed.
>
> Regards
> JB
>
> On 10/04/2018 19:52, Kenneth Knowles wrote:
> > I've been on Idea+Gradle for ~two months, around the time I added
> > https://github.com/apache/beam/pull/4583 and
> > https://github.com/apache/beam/pull/4626 to make the import require
> zero
> > user work. I have no fear of deleting my project any time and
> re-importing.
> >
> > I agree with not having auto-import on. It is just too slow. I can't
> > remember if it was importing too often due to build outputs or if it was
> > just that I was messing with the build.gradle files. Anyhow it doesn't
> > really add much value.
> >
> > The gradle runner _is_ able to use submodules and run individual tests
> > methods, and all that.
> >
> > Kenn
> >
> >
> > On Tue, Apr 10, 2018 at 9:31 AM Romain Manni-Bucau
> > > wrote:
> >
> > Runner a test doesnt have the right classpath (idea uses out/ instead
> > of build/) then when you switch on gradle runner the launching uses
> > gradle which is not able to use submodules directly but reconsider
> the
> > whole project which is quite slow for normal dev iterations
> > compare to just run the test with the right classpath and a fast
> > compile step if needed. I lost literally 1h for something simple with
> > that tooling, this is way too much to be acceptable on my side since
> > I'm sadly not paid to work on beam (one day maybe ;)).
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> >
> > 2018-04-10 18:27 GMT+02:00 Reuven Lax  > >:
> >  > Romain,
> >  >
> >  > Can you detail what's not working. I switched my IntelliJ over to
> > Gradle
> >  > about two weeks ago, and haven't had any trouble.
> >  >
> >  > Reuven
> >  >
> >  > On Tue, Apr 10, 2018 at 4:20 PM Romain Manni-Bucau
> > >
> >  > wrote:
> >  >>
> >  >> Ok, didn't find a way to make it working properly (only
> workaround
> >  >> with direct commands and no good idea integration for
> > debugging). I'm
> >  >> back with maven, if anyone knows how to properly solve it let's
> > do it.
> >  >> If not I think JB point is to consider more than any other
> criteria.
> >  >>
> >  >> Romain Manni-Bucau
> >  >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >  >>
> >  >>
> >  >> 2018-04-10 16:41 GMT+02:00 Romain Manni-Bucau
> > >:
> >  >> > side note: do NOT use auto-import until you are sure you can,
> > it locks
> >  >> > regularly on beam (pby too big for idea?) and makes idea ready
> > to be
> >  >> > killed :(
> >  >> >
> >  >> > Romain Manni-Bucau
> >  >> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >  >> >
> >  >> >
> >  >> > 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré
> > >:
> >  >> >> It's what I did, I'm trying a complete reload now (maybe this
> > step
> >  >> >> failed).
> >  >> >>
> >  >> >> On 10/04/2018 16:38, Lukasz Cwik wrote:
> >  >> >>>
> >  >> >>> beam-site PR/414 updates the instructions for using Intellij
> > and how
> >  >> >>> to
> >  >> >>> import a module:
> >  >> >>>
> >  >> >>> 1. Create an empty IntelliJ project outside of the Beam
> > source tree.
> >  >> >>> 2. Under Project Structure > Project, select a Project SDK.
> >  >> >>> 3. Under Project Structure > Modules, click the + sign to
> > add a module
> >  >> >>> and
> >  >> >>> select "Import Module".
> >  >> >>>  1. Select the directory containing the Beam source tree.
> >  >> >>>  2. Tick the "Import module from external model" button
> > and select
> >  >> >>> Gradle
> >  >> >>> from the list.
> >  >> >>>  3. Tick the following boxes.
> >  >> >>> * Use auto-import
> >  >> >>> * Create separate module per source set
> >  >> >>> * Store generated project files externally
> >  >> >>> * Use default gradle wrapper
> >  >> >>> 4. Delegate build actions to Gradle by 

Re: Gradle Status [April 6]

[Jean-Baptiste Onofré]

FYI, I did a new attempt and it works fine (pretty long). Previous try 
failed.


Regards
JB

On 10/04/2018 19:52, Kenneth Knowles wrote:
I've been on Idea+Gradle for ~two months, around the time I added 
https://github.com/apache/beam/pull/4583 and 
https://github.com/apache/beam/pull/4626 to make the import require zero 
user work. I have no fear of deleting my project any time and re-importing.


I agree with not having auto-import on. It is just too slow. I can't 
remember if it was importing too often due to build outputs or if it was 
just that I was messing with the build.gradle files. Anyhow it doesn't 
really add much value.


The gradle runner _is_ able to use submodules and run individual tests 
methods, and all that.


Kenn


On Tue, Apr 10, 2018 at 9:31 AM Romain Manni-Bucau 
> wrote:


Runner a test doesnt have the right classpath (idea uses out/ instead
of build/) then when you switch on gradle runner the launching uses
gradle which is not able to use submodules directly but reconsider the
whole project which is quite slow for normal dev iterations
compare to just run the test with the right classpath and a fast
compile step if needed. I lost literally 1h for something simple with
that tooling, this is way too much to be acceptable on my side since
I'm sadly not paid to work on beam (one day maybe ;)).

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-10 18:27 GMT+02:00 Reuven Lax >:
 > Romain,
 >
 > Can you detail what's not working. I switched my IntelliJ over to
Gradle
 > about two weeks ago, and haven't had any trouble.
 >
 > Reuven
 >
 > On Tue, Apr 10, 2018 at 4:20 PM Romain Manni-Bucau
>
 > wrote:
 >>
 >> Ok, didn't find a way to make it working properly (only workaround
 >> with direct commands and no good idea integration for
debugging). I'm
 >> back with maven, if anyone knows how to properly solve it let's
do it.
 >> If not I think JB point is to consider more than any other criteria.
 >>
 >> Romain Manni-Bucau
 >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
 >>
 >>
 >> 2018-04-10 16:41 GMT+02:00 Romain Manni-Bucau
>:
 >> > side note: do NOT use auto-import until you are sure you can,
it locks
 >> > regularly on beam (pby too big for idea?) and makes idea ready
to be
 >> > killed :(
 >> >
 >> > Romain Manni-Bucau
 >> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
 >> >
 >> >
 >> > 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré
>:
 >> >> It's what I did, I'm trying a complete reload now (maybe this
step
 >> >> failed).
 >> >>
 >> >> On 10/04/2018 16:38, Lukasz Cwik wrote:
 >> >>>
 >> >>> beam-site PR/414 updates the instructions for using Intellij
and how
 >> >>> to
 >> >>> import a module:
 >> >>>
 >> >>> 1. Create an empty IntelliJ project outside of the Beam
source tree.
 >> >>> 2. Under Project Structure > Project, select a Project SDK.
 >> >>> 3. Under Project Structure > Modules, click the + sign to
add a module
 >> >>> and
 >> >>>     select "Import Module".
 >> >>>      1. Select the directory containing the Beam source tree.
 >> >>>      2. Tick the "Import module from external model" button
and select
 >> >>> Gradle
 >> >>>         from the list.
 >> >>>      3. Tick the following boxes.
 >> >>>         * Use auto-import
 >> >>>         * Create separate module per source set
 >> >>>         * Store generated project files externally
 >> >>>         * Use default gradle wrapper
 >> >>> 4. Delegate build actions to Gradle by going to Settings >
Build,
 >> >>> Execution,
 >> >>>     Deployment > Build Tools > Gradle and checking "Delegate IDE
 >> >>> build/run
 >> >>>     actions to gradle".
 >> >>>
 >> >>> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré

 >> >>> >> wrote:
 >> >>>
 >> >>>     That's a very important issue for contribution. Up to
now, I used
 >> >>> Maven
 >> >>>     for setup IntelliJ (and it works just fine). If we
remove the
 >> >>> pom.xml,
 >> >>>     we have to support Eclipse and IntelliJ "smoothly".
 >> >>>
 >> >>>     Let me try in IntelliJ.
 >> >>>
 >> >>>     Regards
 >> >>>     JB
 >> >>>
 >> >>>     On 10/04/2018 15:21, Romain Manni-Bucau wrote:
 >> >>>      > You dont have issue due to the build setup with that
option. I

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

Le 10 avr. 2018 19:53, "Kenneth Knowles"  a écrit :

I've been on Idea+Gradle for ~two months, around the time I added
https://github.com/apache/beam/pull/4583 and https://github.com/apache/
beam/pull/4626 to make the import require zero user work. I have no fear of
deleting my project any time and re-importing.



The import works (is slow but works) but then i can run anything "normally".


I agree with not having auto-import on. It is just too slow. I can't
remember if it was importing too often due to build outputs or if it was
just that I was messing with the build.gradle files. Anyhow it doesn't
really add much value.

The gradle runner _is_ able to use submodules and run individual tests
methods, and all that.



If i run gradle from any io or any runner module it still loads the whole
project and checks it which is very slow compare to just run the
module...and i hope i dont need a 50chars long command specific each time
to do it, just "build" would be good.


Kenn


On Tue, Apr 10, 2018 at 9:31 AM Romain Manni-Bucau 
wrote:

> Runner a test doesnt have the right classpath (idea uses out/ instead
> of build/) then when you switch on gradle runner the launching uses
> gradle which is not able to use submodules directly but reconsider the
> whole project which is quite slow for normal dev iterations
> compare to just run the test with the right classpath and a fast
> compile step if needed. I lost literally 1h for something simple with
> that tooling, this is way too much to be acceptable on my side since
> I'm sadly not paid to work on beam (one day maybe ;)).
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
>
> 2018-04-10 18:27 GMT+02:00 Reuven Lax :
> > Romain,
> >
> > Can you detail what's not working. I switched my IntelliJ over to Gradle
> > about two weeks ago, and haven't had any trouble.
> >
> > Reuven
> >
> > On Tue, Apr 10, 2018 at 4:20 PM Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >>
> >> Ok, didn't find a way to make it working properly (only workaround
> >> with direct commands and no good idea integration for debugging). I'm
> >> back with maven, if anyone knows how to properly solve it let's do it.
> >> If not I think JB point is to consider more than any other criteria.
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >>
> >>
> >> 2018-04-10 16:41 GMT+02:00 Romain Manni-Bucau :
> >> > side note: do NOT use auto-import until you are sure you can, it locks
> >> > regularly on beam (pby too big for idea?) and makes idea ready to be
> >> > killed :(
> >> >
> >> > Romain Manni-Bucau
> >> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >> >
> >> >
> >> > 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré :
> >> >> It's what I did, I'm trying a complete reload now (maybe this step
> >> >> failed).
> >> >>
> >> >> On 10/04/2018 16:38, Lukasz Cwik wrote:
> >> >>>
> >> >>> beam-site PR/414 updates the instructions for using Intellij and how
> >> >>> to
> >> >>> import a module:
> >> >>>
> >> >>> 1. Create an empty IntelliJ project outside of the Beam source tree.
> >> >>> 2. Under Project Structure > Project, select a Project SDK.
> >> >>> 3. Under Project Structure > Modules, click the + sign to add a
> module
> >> >>> and
> >> >>> select "Import Module".
> >> >>>  1. Select the directory containing the Beam source tree.
> >> >>>  2. Tick the "Import module from external model" button and
> select
> >> >>> Gradle
> >> >>> from the list.
> >> >>>  3. Tick the following boxes.
> >> >>> * Use auto-import
> >> >>> * Create separate module per source set
> >> >>> * Store generated project files externally
> >> >>> * Use default gradle wrapper
> >> >>> 4. Delegate build actions to Gradle by going to Settings > Build,
> >> >>> Execution,
> >> >>> Deployment > Build Tools > Gradle and checking "Delegate IDE
> >> >>> build/run
> >> >>> actions to gradle".
> >> >>>
> >> >>> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré <
> j...@nanthrax.net
> >> >>> > wrote:
> >> >>>
> >> >>> That's a very important issue for contribution. Up to now, I
> used
> >> >>> Maven
> >> >>> for setup IntelliJ (and it works just fine). If we remove the
> >> >>> pom.xml,
> >> >>> we have to support Eclipse and IntelliJ "smoothly".
> >> >>>
> >> >>> Let me try in IntelliJ.
> >> >>>
> >> >>> Regards
> >> >>> JB
> >> >>>
> >> >>> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
> >> >>>  > You dont have issue due to the build setup with that option.
> I
> >> >>> get:
> >> >>>  >
> >> >>>  > avr. 10, 2018 3:20:10 PM
> >> >>>  > org.apache.beam.runners.direct.DirectTransformExecutor run
> >> >>>  > GRAVE: Error occurred within
> >> >>>  > 

Re: Gradle Status [April 6]

[Kenneth Knowles]

I've been on Idea+Gradle for ~two months, around the time I added
https://github.com/apache/beam/pull/4583 and
https://github.com/apache/beam/pull/4626 to make the import require zero
user work. I have no fear of deleting my project any time and re-importing.

I agree with not having auto-import on. It is just too slow. I can't
remember if it was importing too often due to build outputs or if it was
just that I was messing with the build.gradle files. Anyhow it doesn't
really add much value.

The gradle runner _is_ able to use submodules and run individual tests
methods, and all that.

Kenn


On Tue, Apr 10, 2018 at 9:31 AM Romain Manni-Bucau 
wrote:

> Runner a test doesnt have the right classpath (idea uses out/ instead
> of build/) then when you switch on gradle runner the launching uses
> gradle which is not able to use submodules directly but reconsider the
> whole project which is quite slow for normal dev iterations
> compare to just run the test with the right classpath and a fast
> compile step if needed. I lost literally 1h for something simple with
> that tooling, this is way too much to be acceptable on my side since
> I'm sadly not paid to work on beam (one day maybe ;)).
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
>
> 2018-04-10 18:27 GMT+02:00 Reuven Lax :
> > Romain,
> >
> > Can you detail what's not working. I switched my IntelliJ over to Gradle
> > about two weeks ago, and haven't had any trouble.
> >
> > Reuven
> >
> > On Tue, Apr 10, 2018 at 4:20 PM Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >>
> >> Ok, didn't find a way to make it working properly (only workaround
> >> with direct commands and no good idea integration for debugging). I'm
> >> back with maven, if anyone knows how to properly solve it let's do it.
> >> If not I think JB point is to consider more than any other criteria.
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >>
> >>
> >> 2018-04-10 16:41 GMT+02:00 Romain Manni-Bucau :
> >> > side note: do NOT use auto-import until you are sure you can, it locks
> >> > regularly on beam (pby too big for idea?) and makes idea ready to be
> >> > killed :(
> >> >
> >> > Romain Manni-Bucau
> >> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >> >
> >> >
> >> > 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré :
> >> >> It's what I did, I'm trying a complete reload now (maybe this step
> >> >> failed).
> >> >>
> >> >> On 10/04/2018 16:38, Lukasz Cwik wrote:
> >> >>>
> >> >>> beam-site PR/414 updates the instructions for using Intellij and how
> >> >>> to
> >> >>> import a module:
> >> >>>
> >> >>> 1. Create an empty IntelliJ project outside of the Beam source tree.
> >> >>> 2. Under Project Structure > Project, select a Project SDK.
> >> >>> 3. Under Project Structure > Modules, click the + sign to add a
> module
> >> >>> and
> >> >>> select "Import Module".
> >> >>>  1. Select the directory containing the Beam source tree.
> >> >>>  2. Tick the "Import module from external model" button and
> select
> >> >>> Gradle
> >> >>> from the list.
> >> >>>  3. Tick the following boxes.
> >> >>> * Use auto-import
> >> >>> * Create separate module per source set
> >> >>> * Store generated project files externally
> >> >>> * Use default gradle wrapper
> >> >>> 4. Delegate build actions to Gradle by going to Settings > Build,
> >> >>> Execution,
> >> >>> Deployment > Build Tools > Gradle and checking "Delegate IDE
> >> >>> build/run
> >> >>> actions to gradle".
> >> >>>
> >> >>> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré <
> j...@nanthrax.net
> >> >>> > wrote:
> >> >>>
> >> >>> That's a very important issue for contribution. Up to now, I
> used
> >> >>> Maven
> >> >>> for setup IntelliJ (and it works just fine). If we remove the
> >> >>> pom.xml,
> >> >>> we have to support Eclipse and IntelliJ "smoothly".
> >> >>>
> >> >>> Let me try in IntelliJ.
> >> >>>
> >> >>> Regards
> >> >>> JB
> >> >>>
> >> >>> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
> >> >>>  > You dont have issue due to the build setup with that option.
> I
> >> >>> get:
> >> >>>  >
> >> >>>  > avr. 10, 2018 3:20:10 PM
> >> >>>  > org.apache.beam.runners.direct.DirectTransformExecutor run
> >> >>>  > GRAVE: Error occurred within
> >> >>>  >
> org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
> >> >>>  > com.google.common.util.concurrent.ExecutionError:
> >> >>>  > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
> >> >>>  >
> >> >>>  > ?
> >> >>>  >
> >> >>>  > Romain Manni-Bucau
> >> >>>  > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >> >>>  >
> >> >>>  >
> >> >>>  > 2018-04-10 15:13 GMT+02:00 

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

Runner a test doesnt have the right classpath (idea uses out/ instead
of build/) then when you switch on gradle runner the launching uses
gradle which is not able to use submodules directly but reconsider the
whole project which is quite slow for normal dev iterations
compare to just run the test with the right classpath and a fast
compile step if needed. I lost literally 1h for something simple with
that tooling, this is way too much to be acceptable on my side since
I'm sadly not paid to work on beam (one day maybe ;)).

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-10 18:27 GMT+02:00 Reuven Lax :
> Romain,
>
> Can you detail what's not working. I switched my IntelliJ over to Gradle
> about two weeks ago, and haven't had any trouble.
>
> Reuven
>
> On Tue, Apr 10, 2018 at 4:20 PM Romain Manni-Bucau 
> wrote:
>>
>> Ok, didn't find a way to make it working properly (only workaround
>> with direct commands and no good idea integration for debugging). I'm
>> back with maven, if anyone knows how to properly solve it let's do it.
>> If not I think JB point is to consider more than any other criteria.
>>
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>
>>
>> 2018-04-10 16:41 GMT+02:00 Romain Manni-Bucau :
>> > side note: do NOT use auto-import until you are sure you can, it locks
>> > regularly on beam (pby too big for idea?) and makes idea ready to be
>> > killed :(
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >
>> >
>> > 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré :
>> >> It's what I did, I'm trying a complete reload now (maybe this step
>> >> failed).
>> >>
>> >> On 10/04/2018 16:38, Lukasz Cwik wrote:
>> >>>
>> >>> beam-site PR/414 updates the instructions for using Intellij and how
>> >>> to
>> >>> import a module:
>> >>>
>> >>> 1. Create an empty IntelliJ project outside of the Beam source tree.
>> >>> 2. Under Project Structure > Project, select a Project SDK.
>> >>> 3. Under Project Structure > Modules, click the + sign to add a module
>> >>> and
>> >>> select "Import Module".
>> >>>  1. Select the directory containing the Beam source tree.
>> >>>  2. Tick the "Import module from external model" button and select
>> >>> Gradle
>> >>> from the list.
>> >>>  3. Tick the following boxes.
>> >>> * Use auto-import
>> >>> * Create separate module per source set
>> >>> * Store generated project files externally
>> >>> * Use default gradle wrapper
>> >>> 4. Delegate build actions to Gradle by going to Settings > Build,
>> >>> Execution,
>> >>> Deployment > Build Tools > Gradle and checking "Delegate IDE
>> >>> build/run
>> >>> actions to gradle".
>> >>>
>> >>> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré > >>> > wrote:
>> >>>
>> >>> That's a very important issue for contribution. Up to now, I used
>> >>> Maven
>> >>> for setup IntelliJ (and it works just fine). If we remove the
>> >>> pom.xml,
>> >>> we have to support Eclipse and IntelliJ "smoothly".
>> >>>
>> >>> Let me try in IntelliJ.
>> >>>
>> >>> Regards
>> >>> JB
>> >>>
>> >>> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
>> >>>  > You dont have issue due to the build setup with that option. I
>> >>> get:
>> >>>  >
>> >>>  > avr. 10, 2018 3:20:10 PM
>> >>>  > org.apache.beam.runners.direct.DirectTransformExecutor run
>> >>>  > GRAVE: Error occurred within
>> >>>  > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
>> >>>  > com.google.common.util.concurrent.ExecutionError:
>> >>>  > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
>> >>>  >
>> >>>  > ?
>> >>>  >
>> >>>  > Romain Manni-Bucau
>> >>>  > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >>>  >
>> >>>  >
>> >>>  > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik > >>> >:
>> >>>  >> I have found that the simplest setup is to delegate the
>> >>> build/test actions
>> >>>  >> to Gradle. This allows you to run unit tests very easily and
>> >>> since its in
>> >>>  >> the same manner that Gradle would have, you know that if its
>> >>> passing it will
>> >>>  >> pass on the command line and on Jenkins. Here is one site that
>> >>> discusses how
>> >>>  >> to set this up:
>> >>>  >>
>> >>>
>> >>>
>> >>> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
>> >>>  >>
>> >>>  >>
>> >>>  >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau
>> >>> >
>> >>>  >> wrote:
>> >>>  >>>
>> >>>  >>> What's the plan to make idea supporting gradle on beam
>> >>> project?

Re: Gradle Status [April 6]

[Reuven Lax]

Romain,

Can you detail what's not working. I switched my IntelliJ over to Gradle
about two weeks ago, and haven't had any trouble.

Reuven

On Tue, Apr 10, 2018 at 4:20 PM Romain Manni-Bucau 
wrote:

> Ok, didn't find a way to make it working properly (only workaround
> with direct commands and no good idea integration for debugging). I'm
> back with maven, if anyone knows how to properly solve it let's do it.
> If not I think JB point is to consider more than any other criteria.
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
>
> 2018-04-10 16:41 GMT+02:00 Romain Manni-Bucau :
> > side note: do NOT use auto-import until you are sure you can, it locks
> > regularly on beam (pby too big for idea?) and makes idea ready to be
> > killed :(
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> >
> > 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré :
> >> It's what I did, I'm trying a complete reload now (maybe this step
> failed).
> >>
> >> On 10/04/2018 16:38, Lukasz Cwik wrote:
> >>>
> >>> beam-site PR/414 updates the instructions for using Intellij and how to
> >>> import a module:
> >>>
> >>> 1. Create an empty IntelliJ project outside of the Beam source tree.
> >>> 2. Under Project Structure > Project, select a Project SDK.
> >>> 3. Under Project Structure > Modules, click the + sign to add a module
> and
> >>> select "Import Module".
> >>>  1. Select the directory containing the Beam source tree.
> >>>  2. Tick the "Import module from external model" button and select
> >>> Gradle
> >>> from the list.
> >>>  3. Tick the following boxes.
> >>> * Use auto-import
> >>> * Create separate module per source set
> >>> * Store generated project files externally
> >>> * Use default gradle wrapper
> >>> 4. Delegate build actions to Gradle by going to Settings > Build,
> >>> Execution,
> >>> Deployment > Build Tools > Gradle and checking "Delegate IDE
> build/run
> >>> actions to gradle".
> >>>
> >>> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré  >>> > wrote:
> >>>
> >>> That's a very important issue for contribution. Up to now, I used
> >>> Maven
> >>> for setup IntelliJ (and it works just fine). If we remove the
> pom.xml,
> >>> we have to support Eclipse and IntelliJ "smoothly".
> >>>
> >>> Let me try in IntelliJ.
> >>>
> >>> Regards
> >>> JB
> >>>
> >>> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
> >>>  > You dont have issue due to the build setup with that option. I
> get:
> >>>  >
> >>>  > avr. 10, 2018 3:20:10 PM
> >>>  > org.apache.beam.runners.direct.DirectTransformExecutor run
> >>>  > GRAVE: Error occurred within
> >>>  > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
> >>>  > com.google.common.util.concurrent.ExecutionError:
> >>>  > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
> >>>  >
> >>>  > ?
> >>>  >
> >>>  > Romain Manni-Bucau
> >>>  > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >>>  >
> >>>  >
> >>>  > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik  >>> >:
> >>>  >> I have found that the simplest setup is to delegate the
> >>> build/test actions
> >>>  >> to Gradle. This allows you to run unit tests very easily and
> >>> since its in
> >>>  >> the same manner that Gradle would have, you know that if its
> >>> passing it will
> >>>  >> pass on the command line and on Jenkins. Here is one site that
> >>> discusses how
> >>>  >> to set this up:
> >>>  >>
> >>>
> >>>
> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
> >>>  >>
> >>>  >>
> >>>  >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau
> >>> >
> >>>  >> wrote:
> >>>  >>>
> >>>  >>> What's the plan to make idea supporting gradle on beam
> project?
> >>> Do we
> >>>  >>> import the workaround mentionned in
> >>>  >>> https://youtrack.jetbrains.com/issue/IDEA-175172?
> >>>  >>> For the ones who didn't see this issue in action: idea will
> >>> compile in
> >>>  >>> out/ instead of build/ and you will just miss all the
> resources
> >>> you
> >>>  >>> need like some SPI registration which are used by all our
> >>> registrar =>
> >>>  >>> no way to run tests in idea without hacking the configuration
> >>> quite
> >>>  >>> deeply :(
> >>>  >>>
> >>>  >>> Romain Manni-Bucau
> >>>  >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >>>  >>>
> >>>  >>>
> >>>  >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot
> >>> >:
> >>>
> >>>

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

Ok, didn't find a way to make it working properly (only workaround
with direct commands and no good idea integration for debugging). I'm
back with maven, if anyone knows how to properly solve it let's do it.
If not I think JB point is to consider more than any other criteria.

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-10 16:41 GMT+02:00 Romain Manni-Bucau :
> side note: do NOT use auto-import until you are sure you can, it locks
> regularly on beam (pby too big for idea?) and makes idea ready to be
> killed :(
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
>
> 2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré :
>> It's what I did, I'm trying a complete reload now (maybe this step failed).
>>
>> On 10/04/2018 16:38, Lukasz Cwik wrote:
>>>
>>> beam-site PR/414 updates the instructions for using Intellij and how to
>>> import a module:
>>>
>>> 1. Create an empty IntelliJ project outside of the Beam source tree.
>>> 2. Under Project Structure > Project, select a Project SDK.
>>> 3. Under Project Structure > Modules, click the + sign to add a module and
>>> select "Import Module".
>>>  1. Select the directory containing the Beam source tree.
>>>  2. Tick the "Import module from external model" button and select
>>> Gradle
>>> from the list.
>>>  3. Tick the following boxes.
>>> * Use auto-import
>>> * Create separate module per source set
>>> * Store generated project files externally
>>> * Use default gradle wrapper
>>> 4. Delegate build actions to Gradle by going to Settings > Build,
>>> Execution,
>>> Deployment > Build Tools > Gradle and checking "Delegate IDE build/run
>>> actions to gradle".
>>>
>>> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré >> > wrote:
>>>
>>> That's a very important issue for contribution. Up to now, I used
>>> Maven
>>> for setup IntelliJ (and it works just fine). If we remove the pom.xml,
>>> we have to support Eclipse and IntelliJ "smoothly".
>>>
>>> Let me try in IntelliJ.
>>>
>>> Regards
>>> JB
>>>
>>> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
>>>  > You dont have issue due to the build setup with that option. I get:
>>>  >
>>>  > avr. 10, 2018 3:20:10 PM
>>>  > org.apache.beam.runners.direct.DirectTransformExecutor run
>>>  > GRAVE: Error occurred within
>>>  > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
>>>  > com.google.common.util.concurrent.ExecutionError:
>>>  > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
>>>  >
>>>  > ?
>>>  >
>>>  > Romain Manni-Bucau
>>>  > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>  >
>>>  >
>>>  > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik >> >:
>>>  >> I have found that the simplest setup is to delegate the
>>> build/test actions
>>>  >> to Gradle. This allows you to run unit tests very easily and
>>> since its in
>>>  >> the same manner that Gradle would have, you know that if its
>>> passing it will
>>>  >> pass on the command line and on Jenkins. Here is one site that
>>> discusses how
>>>  >> to set this up:
>>>  >>
>>>
>>> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
>>>  >>
>>>  >>
>>>  >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau
>>> >
>>>  >> wrote:
>>>  >>>
>>>  >>> What's the plan to make idea supporting gradle on beam project?
>>> Do we
>>>  >>> import the workaround mentionned in
>>>  >>> https://youtrack.jetbrains.com/issue/IDEA-175172?
>>>  >>> For the ones who didn't see this issue in action: idea will
>>> compile in
>>>  >>> out/ instead of build/ and you will just miss all the resources
>>> you
>>>  >>> need like some SPI registration which are used by all our
>>> registrar =>
>>>  >>> no way to run tests in idea without hacking the configuration
>>> quite
>>>  >>> deeply :(
>>>  >>>
>>>  >>> Romain Manni-Bucau
>>>  >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>  >>>
>>>  >>>
>>>  >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot
>>> >:
>>>
>>>   As a gradle beginner, I could not agree more !
>>>   +1
>>>   Etienne
>>>   Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a
>>> écrit :
>>>  
>>>   Hi all,
>>>  
>>>   I did multiple gradle build since last week and I would like
>>> to share
>>>   one of my concern: it's about the communities.
>>>  
>>>   If I think our users won't see any change for them 

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

side note: do NOT use auto-import until you are sure you can, it locks
regularly on beam (pby too big for idea?) and makes idea ready to be
killed :(

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-10 16:40 GMT+02:00 Jean-Baptiste Onofré :
> It's what I did, I'm trying a complete reload now (maybe this step failed).
>
> On 10/04/2018 16:38, Lukasz Cwik wrote:
>>
>> beam-site PR/414 updates the instructions for using Intellij and how to
>> import a module:
>>
>> 1. Create an empty IntelliJ project outside of the Beam source tree.
>> 2. Under Project Structure > Project, select a Project SDK.
>> 3. Under Project Structure > Modules, click the + sign to add a module and
>> select "Import Module".
>>  1. Select the directory containing the Beam source tree.
>>  2. Tick the "Import module from external model" button and select
>> Gradle
>> from the list.
>>  3. Tick the following boxes.
>> * Use auto-import
>> * Create separate module per source set
>> * Store generated project files externally
>> * Use default gradle wrapper
>> 4. Delegate build actions to Gradle by going to Settings > Build,
>> Execution,
>> Deployment > Build Tools > Gradle and checking "Delegate IDE build/run
>> actions to gradle".
>>
>> On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré > > wrote:
>>
>> That's a very important issue for contribution. Up to now, I used
>> Maven
>> for setup IntelliJ (and it works just fine). If we remove the pom.xml,
>> we have to support Eclipse and IntelliJ "smoothly".
>>
>> Let me try in IntelliJ.
>>
>> Regards
>> JB
>>
>> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
>>  > You dont have issue due to the build setup with that option. I get:
>>  >
>>  > avr. 10, 2018 3:20:10 PM
>>  > org.apache.beam.runners.direct.DirectTransformExecutor run
>>  > GRAVE: Error occurred within
>>  > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
>>  > com.google.common.util.concurrent.ExecutionError:
>>  > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
>>  >
>>  > ?
>>  >
>>  > Romain Manni-Bucau
>>  > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>  >
>>  >
>>  > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik > >:
>>  >> I have found that the simplest setup is to delegate the
>> build/test actions
>>  >> to Gradle. This allows you to run unit tests very easily and
>> since its in
>>  >> the same manner that Gradle would have, you know that if its
>> passing it will
>>  >> pass on the command line and on Jenkins. Here is one site that
>> discusses how
>>  >> to set this up:
>>  >>
>>
>> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
>>  >>
>>  >>
>>  >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau
>> >
>>  >> wrote:
>>  >>>
>>  >>> What's the plan to make idea supporting gradle on beam project?
>> Do we
>>  >>> import the workaround mentionned in
>>  >>> https://youtrack.jetbrains.com/issue/IDEA-175172?
>>  >>> For the ones who didn't see this issue in action: idea will
>> compile in
>>  >>> out/ instead of build/ and you will just miss all the resources
>> you
>>  >>> need like some SPI registration which are used by all our
>> registrar =>
>>  >>> no way to run tests in idea without hacking the configuration
>> quite
>>  >>> deeply :(
>>  >>>
>>  >>> Romain Manni-Bucau
>>  >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>  >>>
>>  >>>
>>  >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot
>> >:
>>
>>   As a gradle beginner, I could not agree more !
>>   +1
>>   Etienne
>>   Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a
>> écrit :
>>  
>>   Hi all,
>>  
>>   I did multiple gradle build since last week and I would like
>> to share
>>   one of my concern: it's about the communities.
>>  
>>   If I think our users won't see any change for them due to
>> Gradle build
>>   (I think that most of our users will still use Maven with
>> artifacts
>>   provided by Gradle), I'm more concerned by the dev community
>> and the
>>   contribution.
>>  
>>   Maven is well known and straight forward for a large part of
>> potential
>>   contributors. I think we have to keep in mind that we still
>> have to grow
>>   up our contributors community.
>>  
>>   Today, maybe I'm wrong, but I have the feeling that 

Re: Gradle Status [April 6]

[Jean-Baptiste Onofré]

It's what I did, I'm trying a complete reload now (maybe this step failed).

On 10/04/2018 16:38, Lukasz Cwik wrote:
beam-site PR/414 updates the instructions for using Intellij and how to 
import a module:


1. Create an empty IntelliJ project outside of the Beam source tree.
2. Under Project Structure > Project, select a Project SDK.
3. Under Project Structure > Modules, click the + sign to add a module and
    select "Import Module".
     1. Select the directory containing the Beam source tree.
     2. Tick the "Import module from external model" button and select 
Gradle

        from the list.
     3. Tick the following boxes.
        * Use auto-import
        * Create separate module per source set
        * Store generated project files externally
        * Use default gradle wrapper
4. Delegate build actions to Gradle by going to Settings > Build, Execution,
    Deployment > Build Tools > Gradle and checking "Delegate IDE build/run
    actions to gradle".

On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré > wrote:


That's a very important issue for contribution. Up to now, I used Maven
for setup IntelliJ (and it works just fine). If we remove the pom.xml,
we have to support Eclipse and IntelliJ "smoothly".

Let me try in IntelliJ.

Regards
JB

On 10/04/2018 15:21, Romain Manni-Bucau wrote:
 > You dont have issue due to the build setup with that option. I get:
 >
 > avr. 10, 2018 3:20:10 PM
 > org.apache.beam.runners.direct.DirectTransformExecutor run
 > GRAVE: Error occurred within
 > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
 > com.google.common.util.concurrent.ExecutionError:
 > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
 >
 > ?
 >
 > Romain Manni-Bucau
 > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
 >
 >
 > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik >:
 >> I have found that the simplest setup is to delegate the
build/test actions
 >> to Gradle. This allows you to run unit tests very easily and
since its in
 >> the same manner that Gradle would have, you know that if its
passing it will
 >> pass on the command line and on Jenkins. Here is one site that
discusses how
 >> to set this up:
 >>

http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
 >>
 >>
 >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau
>
 >> wrote:
 >>>
 >>> What's the plan to make idea supporting gradle on beam project?
Do we
 >>> import the workaround mentionned in
 >>> https://youtrack.jetbrains.com/issue/IDEA-175172?
 >>> For the ones who didn't see this issue in action: idea will
compile in
 >>> out/ instead of build/ and you will just miss all the resources you
 >>> need like some SPI registration which are used by all our
registrar =>
 >>> no way to run tests in idea without hacking the configuration quite
 >>> deeply :(
 >>>
 >>> Romain Manni-Bucau
 >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
 >>>
 >>>
 >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot
>:
  As a gradle beginner, I could not agree more !
  +1
  Etienne
  Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a
écrit :
 
  Hi all,
 
  I did multiple gradle build since last week and I would like
to share
  one of my concern: it's about the communities.
 
  If I think our users won't see any change for them due to
Gradle build
  (I think that most of our users will still use Maven with
artifacts
  provided by Gradle), I'm more concerned by the dev community
and the
  contribution.
 
  Maven is well known and straight forward for a large part of
potential
  contributors. I think we have to keep in mind that we still
have to grow
  up our contributors community.
 
  Today, maybe I'm wrong, but I have the feeling that gradle
build is not
  straight forward (build.gradle includes build_rules.gradle,
gathering
  all taks all together).
 
  I would like to add a task in the gradle "migration" process:
simplify
  the gradle structure and files, and document this.
 
  I know we already have a Jira about the documentation part,
but I would
  like to "polish" and use a clean structure for the Gradle
resources. As
  already quickly discussed, I think that having one gradle file
per tasks
  in the .gradle directory would be helpful.
 
  The 

Re: Gradle Status [April 6]

[Lukasz Cwik]

beam-site PR/414 updates the instructions for using Intellij and how to
import a module:

1. Create an empty IntelliJ project outside of the Beam source tree.
2. Under Project Structure > Project, select a Project SDK.
3. Under Project Structure > Modules, click the + sign to add a module and
   select "Import Module".
1. Select the directory containing the Beam source tree.
2. Tick the "Import module from external model" button and select Gradle
   from the list.
3. Tick the following boxes.
   * Use auto-import
   * Create separate module per source set
   * Store generated project files externally
   * Use default gradle wrapper
4. Delegate build actions to Gradle by going to Settings > Build, Execution,
   Deployment > Build Tools > Gradle and checking "Delegate IDE build/run
   actions to gradle".

On Tue, Apr 10, 2018 at 10:34 AM Jean-Baptiste Onofré 
wrote:

> That's a very important issue for contribution. Up to now, I used Maven
> for setup IntelliJ (and it works just fine). If we remove the pom.xml,
> we have to support Eclipse and IntelliJ "smoothly".
>
> Let me try in IntelliJ.
>
> Regards
> JB
>
> On 10/04/2018 15:21, Romain Manni-Bucau wrote:
> > You dont have issue due to the build setup with that option. I get:
> >
> > avr. 10, 2018 3:20:10 PM
> > org.apache.beam.runners.direct.DirectTransformExecutor run
> > GRAVE: Error occurred within
> > org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
> > com.google.common.util.concurrent.ExecutionError:
> > java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
> >
> > ?
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> >
> > 2018-04-10 15:13 GMT+02:00 Lukasz Cwik :
> >> I have found that the simplest setup is to delegate the build/test
> actions
> >> to Gradle. This allows you to run unit tests very easily and since its
> in
> >> the same manner that Gradle would have, you know that if its passing it
> will
> >> pass on the command line and on Jenkins. Here is one site that
> discusses how
> >> to set this up:
> >>
> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
> >>
> >>
> >> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau <
> rmannibu...@gmail.com>
> >> wrote:
> >>>
> >>> What's the plan to make idea supporting gradle on beam project? Do we
> >>> import the workaround mentionned in
> >>> https://youtrack.jetbrains.com/issue/IDEA-175172?
> >>> For the ones who didn't see this issue in action: idea will compile in
> >>> out/ instead of build/ and you will just miss all the resources you
> >>> need like some SPI registration which are used by all our registrar =>
> >>> no way to run tests in idea without hacking the configuration quite
> >>> deeply :(
> >>>
> >>> Romain Manni-Bucau
> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >>>
> >>>
> >>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot :
>  As a gradle beginner, I could not agree more !
>  +1
>  Etienne
>  Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :
> 
>  Hi all,
> 
>  I did multiple gradle build since last week and I would like to share
>  one of my concern: it's about the communities.
> 
>  If I think our users won't see any change for them due to Gradle build
>  (I think that most of our users will still use Maven with artifacts
>  provided by Gradle), I'm more concerned by the dev community and the
>  contribution.
> 
>  Maven is well known and straight forward for a large part of potential
>  contributors. I think we have to keep in mind that we still have to
> grow
>  up our contributors community.
> 
>  Today, maybe I'm wrong, but I have the feeling that gradle build is
> not
>  straight forward (build.gradle includes build_rules.gradle, gathering
>  all taks all together).
> 
>  I would like to add a task in the gradle "migration" process: simplify
>  the gradle structure and files, and document this.
> 
>  I know we already have a Jira about the documentation part, but I
> would
>  like to "polish" and use a clean structure for the Gradle resources.
> As
>  already quickly discussed, I think that having one gradle file per
> tasks
>  in the .gradle directory would be helpful.
> 
>  The goal is really to simplify the contribution.
> 
>  Do you agree if I add a Jira about "Gradle polish" ?
>  Thoughts ?
> 
>  Regards
>  JB
> 
>  On 07/04/2018 04:52, Scott Wegner wrote:
> 
>  Here's an end-of-day update on migration work:
> 
>  * Snapshot unsigned dailies and signed release builds are working
> (!!).
>  PR/5048 [1] merges changes from Luke's branch
>  * python precommit failing... will investigate python precommit
>  Monday
>  * All Precommits are gradle only
>  

Re: Gradle Status [April 6]

[Jean-Baptiste Onofré]

That's a very important issue for contribution. Up to now, I used Maven 
for setup IntelliJ (and it works just fine). If we remove the pom.xml, 
we have to support Eclipse and IntelliJ "smoothly".


Let me try in IntelliJ.

Regards
JB

On 10/04/2018 15:21, Romain Manni-Bucau wrote:

You dont have issue due to the build setup with that option. I get:

avr. 10, 2018 3:20:10 PM
org.apache.beam.runners.direct.DirectTransformExecutor run
GRAVE: Error occurred within
org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
com.google.common.util.concurrent.ExecutionError:
java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy

?

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-10 15:13 GMT+02:00 Lukasz Cwik :

I have found that the simplest setup is to delegate the build/test actions
to Gradle. This allows you to run unit tests very easily and since its in
the same manner that Gradle would have, you know that if its passing it will
pass on the command line and on Jenkins. Here is one site that discusses how
to set this up:
http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html


On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau 
wrote:


What's the plan to make idea supporting gradle on beam project? Do we
import the workaround mentionned in
https://youtrack.jetbrains.com/issue/IDEA-175172?
For the ones who didn't see this issue in action: idea will compile in
out/ instead of build/ and you will just miss all the resources you
need like some SPI registration which are used by all our registrar =>
no way to run tests in idea without hacking the configuration quite
deeply :(

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-10 10:08 GMT+02:00 Etienne Chauchot :

As a gradle beginner, I could not agree more !
+1
Etienne
Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :

Hi all,

I did multiple gradle build since last week and I would like to share
one of my concern: it's about the communities.

If I think our users won't see any change for them due to Gradle build
(I think that most of our users will still use Maven with artifacts
provided by Gradle), I'm more concerned by the dev community and the
contribution.

Maven is well known and straight forward for a large part of potential
contributors. I think we have to keep in mind that we still have to grow
up our contributors community.

Today, maybe I'm wrong, but I have the feeling that gradle build is not
straight forward (build.gradle includes build_rules.gradle, gathering
all taks all together).

I would like to add a task in the gradle "migration" process: simplify
the gradle structure and files, and document this.

I know we already have a Jira about the documentation part, but I would
like to "polish" and use a clean structure for the Gradle resources. As
already quickly discussed, I think that having one gradle file per tasks
in the .gradle directory would be helpful.

The goal is really to simplify the contribution.

Do you agree if I add a Jira about "Gradle polish" ?
Thoughts ?

Regards
JB

On 07/04/2018 04:52, Scott Wegner wrote:

Here's an end-of-day update on migration work:

* Snapshot unsigned dailies and signed release builds are working (!!).
PR/5048 [1] merges changes from Luke's branch
* python precommit failing... will investigate python precommit
Monday
* All Precommits are gradle only
* All Postcommits except performance tests and Java_JDK_Versions_Test
use gradle (after PR/5047 [2] merged)
* Nightly snapshot release using gradle is ready; needs PR/5048 to be
merged before switching
* ValidatesRunner_Spark failing consistently; investigating

Thanks for another productive day of hacking. I'll pick up again on
Monday.

[1] https://github.com/apache/beam/pull/5048
[2] https://github.com/apache/beam/pull/5047


On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
> wrote:

 Why building a zip per runner which its stack and just pointing out
 on that zip and let beam lazy load the runner:

 --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or
 the fromSystemProperties() if it gets merged a day ;))

 Le 6 avr. 2018 20:21, "Kenneth Knowles" > a écrit :

 I'm working on finding a solution for launching the Nexmark
 suite with each runner. This doesn't have to be done via Gradle,
 but we anyhow need built artifacts that don't require user
 classpath intervention.

 It looks to me like the examples are also missing this - they
 have separate configuration e.g. sparkRunnerPreCommit but that
 is overspecified compared to a free-form launching of a main()
 program with a runner profile.

 On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik 

Re: Gradle Status [April 6]

[Lukasz Cwik]

Romain, I haven't seen that error. At the very top of your test execution
log it gives you the tasks that it is running, for example:
6:41:33 AM: Executing tasks ':beam-sdks-java-core:cleanTest
:beam-sdks-java-core:test --tests
"org.apache.beam.sdk.coders.AvroCoderTest.testAvroCoderEncoding"'...

What task did it fail for you for?

On Tue, Apr 10, 2018 at 9:21 AM Romain Manni-Bucau 
wrote:

> You dont have issue due to the build setup with that option. I get:
>
> avr. 10, 2018 3:20:10 PM
> org.apache.beam.runners.direct.DirectTransformExecutor run
> GRAVE: Error occurred within
> org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
> com.google.common.util.concurrent.ExecutionError:
> java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy
>
> ?
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
>
> 2018-04-10 15:13 GMT+02:00 Lukasz Cwik :
> > I have found that the simplest setup is to delegate the build/test
> actions
> > to Gradle. This allows you to run unit tests very easily and since its in
> > the same manner that Gradle would have, you know that if its passing it
> will
> > pass on the command line and on Jenkins. Here is one site that discusses
> how
> > to set this up:
> >
> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
> >
> >
> > On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau <
> rmannibu...@gmail.com>
> > wrote:
> >>
> >> What's the plan to make idea supporting gradle on beam project? Do we
> >> import the workaround mentionned in
> >> https://youtrack.jetbrains.com/issue/IDEA-175172?
> >> For the ones who didn't see this issue in action: idea will compile in
> >> out/ instead of build/ and you will just miss all the resources you
> >> need like some SPI registration which are used by all our registrar =>
> >> no way to run tests in idea without hacking the configuration quite
> >> deeply :(
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >>
> >>
> >> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot :
> >> > As a gradle beginner, I could not agree more !
> >> > +1
> >> > Etienne
> >> > Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :
> >> >
> >> > Hi all,
> >> >
> >> > I did multiple gradle build since last week and I would like to share
> >> > one of my concern: it's about the communities.
> >> >
> >> > If I think our users won't see any change for them due to Gradle build
> >> > (I think that most of our users will still use Maven with artifacts
> >> > provided by Gradle), I'm more concerned by the dev community and the
> >> > contribution.
> >> >
> >> > Maven is well known and straight forward for a large part of potential
> >> > contributors. I think we have to keep in mind that we still have to
> grow
> >> > up our contributors community.
> >> >
> >> > Today, maybe I'm wrong, but I have the feeling that gradle build is
> not
> >> > straight forward (build.gradle includes build_rules.gradle, gathering
> >> > all taks all together).
> >> >
> >> > I would like to add a task in the gradle "migration" process: simplify
> >> > the gradle structure and files, and document this.
> >> >
> >> > I know we already have a Jira about the documentation part, but I
> would
> >> > like to "polish" and use a clean structure for the Gradle resources.
> As
> >> > already quickly discussed, I think that having one gradle file per
> tasks
> >> > in the .gradle directory would be helpful.
> >> >
> >> > The goal is really to simplify the contribution.
> >> >
> >> > Do you agree if I add a Jira about "Gradle polish" ?
> >> > Thoughts ?
> >> >
> >> > Regards
> >> > JB
> >> >
> >> > On 07/04/2018 04:52, Scott Wegner wrote:
> >> >
> >> > Here's an end-of-day update on migration work:
> >> >
> >> > * Snapshot unsigned dailies and signed release builds are working
> (!!).
> >> > PR/5048 [1] merges changes from Luke's branch
> >> >* python precommit failing... will investigate python precommit
> >> > Monday
> >> > * All Precommits are gradle only
> >> > * All Postcommits except performance tests and Java_JDK_Versions_Test
> >> > use gradle (after PR/5047 [2] merged)
> >> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be
> >> > merged before switching
> >> > * ValidatesRunner_Spark failing consistently; investigating
> >> >
> >> > Thanks for another productive day of hacking. I'll pick up again on
> >> > Monday.
> >> >
> >> > [1] https://github.com/apache/beam/pull/5048
> >> > [2] https://github.com/apache/beam/pull/5047
> >> >
> >> >
> >> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
> >> > > wrote:
> >> >
> >> > Why building a zip per runner which its stack and just pointing
> out
> >> > on that zip and let beam lazy load the runner:
> >> >
> >> > --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=...
> 

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

You dont have issue due to the build setup with that option. I get:

avr. 10, 2018 3:20:10 PM
org.apache.beam.runners.direct.DirectTransformExecutor run
GRAVE: Error occurred within
org.apache.beam.runners.direct.DirectTransformExecutor@66761b7a
com.google.common.util.concurrent.ExecutionError:
java.lang.NoClassDefFoundError: net/bytebuddy/NamingStrategy

?

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-10 15:13 GMT+02:00 Lukasz Cwik :
> I have found that the simplest setup is to delegate the build/test actions
> to Gradle. This allows you to run unit tests very easily and since its in
> the same manner that Gradle would have, you know that if its passing it will
> pass on the command line and on Jenkins. Here is one site that discusses how
> to set this up:
> http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html
>
>
> On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau 
> wrote:
>>
>> What's the plan to make idea supporting gradle on beam project? Do we
>> import the workaround mentionned in
>> https://youtrack.jetbrains.com/issue/IDEA-175172?
>> For the ones who didn't see this issue in action: idea will compile in
>> out/ instead of build/ and you will just miss all the resources you
>> need like some SPI registration which are used by all our registrar =>
>> no way to run tests in idea without hacking the configuration quite
>> deeply :(
>>
>> Romain Manni-Bucau
>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>
>>
>> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot :
>> > As a gradle beginner, I could not agree more !
>> > +1
>> > Etienne
>> > Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :
>> >
>> > Hi all,
>> >
>> > I did multiple gradle build since last week and I would like to share
>> > one of my concern: it's about the communities.
>> >
>> > If I think our users won't see any change for them due to Gradle build
>> > (I think that most of our users will still use Maven with artifacts
>> > provided by Gradle), I'm more concerned by the dev community and the
>> > contribution.
>> >
>> > Maven is well known and straight forward for a large part of potential
>> > contributors. I think we have to keep in mind that we still have to grow
>> > up our contributors community.
>> >
>> > Today, maybe I'm wrong, but I have the feeling that gradle build is not
>> > straight forward (build.gradle includes build_rules.gradle, gathering
>> > all taks all together).
>> >
>> > I would like to add a task in the gradle "migration" process: simplify
>> > the gradle structure and files, and document this.
>> >
>> > I know we already have a Jira about the documentation part, but I would
>> > like to "polish" and use a clean structure for the Gradle resources. As
>> > already quickly discussed, I think that having one gradle file per tasks
>> > in the .gradle directory would be helpful.
>> >
>> > The goal is really to simplify the contribution.
>> >
>> > Do you agree if I add a Jira about "Gradle polish" ?
>> > Thoughts ?
>> >
>> > Regards
>> > JB
>> >
>> > On 07/04/2018 04:52, Scott Wegner wrote:
>> >
>> > Here's an end-of-day update on migration work:
>> >
>> > * Snapshot unsigned dailies and signed release builds are working (!!).
>> > PR/5048 [1] merges changes from Luke's branch
>> >* python precommit failing... will investigate python precommit
>> > Monday
>> > * All Precommits are gradle only
>> > * All Postcommits except performance tests and Java_JDK_Versions_Test
>> > use gradle (after PR/5047 [2] merged)
>> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be
>> > merged before switching
>> > * ValidatesRunner_Spark failing consistently; investigating
>> >
>> > Thanks for another productive day of hacking. I'll pick up again on
>> > Monday.
>> >
>> > [1] https://github.com/apache/beam/pull/5048
>> > [2] https://github.com/apache/beam/pull/5047
>> >
>> >
>> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
>> > > wrote:
>> >
>> > Why building a zip per runner which its stack and just pointing out
>> > on that zip and let beam lazy load the runner:
>> >
>> > --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or
>> > the fromSystemProperties() if it gets merged a day ;))
>> >
>> > Le 6 avr. 2018 20:21, "Kenneth Knowles" > > > a écrit :
>> >
>> > I'm working on finding a solution for launching the Nexmark
>> > suite with each runner. This doesn't have to be done via Gradle,
>> > but we anyhow need built artifacts that don't require user
>> > classpath intervention.
>> >
>> > It looks to me like the examples are also missing this - they
>> > have separate configuration e.g. sparkRunnerPreCommit but that
>> > is overspecified compared to a 

Re: Gradle Status [April 6]

[Lukasz Cwik]

I have found that the simplest setup is to delegate the build/test actions
to Gradle. This allows you to run unit tests very easily and since its in
the same manner that Gradle would have, you know that if its passing it
will pass on the command line and on Jenkins. Here is one site that
discusses how to set this up:
http://mrhaki.blogspot.com/2016/11/gradle-goodness-delegate-build-and-run.html


On Tue, Apr 10, 2018 at 8:45 AM Romain Manni-Bucau 
wrote:

> What's the plan to make idea supporting gradle on beam project? Do we
> import the workaround mentionned in
> https://youtrack.jetbrains.com/issue/IDEA-175172?
> For the ones who didn't see this issue in action: idea will compile in
> out/ instead of build/ and you will just miss all the resources you
> need like some SPI registration which are used by all our registrar =>
> no way to run tests in idea without hacking the configuration quite
> deeply :(
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
>
> 2018-04-10 10:08 GMT+02:00 Etienne Chauchot :
> > As a gradle beginner, I could not agree more !
> > +1
> > Etienne
> > Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :
> >
> > Hi all,
> >
> > I did multiple gradle build since last week and I would like to share
> > one of my concern: it's about the communities.
> >
> > If I think our users won't see any change for them due to Gradle build
> > (I think that most of our users will still use Maven with artifacts
> > provided by Gradle), I'm more concerned by the dev community and the
> > contribution.
> >
> > Maven is well known and straight forward for a large part of potential
> > contributors. I think we have to keep in mind that we still have to grow
> > up our contributors community.
> >
> > Today, maybe I'm wrong, but I have the feeling that gradle build is not
> > straight forward (build.gradle includes build_rules.gradle, gathering
> > all taks all together).
> >
> > I would like to add a task in the gradle "migration" process: simplify
> > the gradle structure and files, and document this.
> >
> > I know we already have a Jira about the documentation part, but I would
> > like to "polish" and use a clean structure for the Gradle resources. As
> > already quickly discussed, I think that having one gradle file per tasks
> > in the .gradle directory would be helpful.
> >
> > The goal is really to simplify the contribution.
> >
> > Do you agree if I add a Jira about "Gradle polish" ?
> > Thoughts ?
> >
> > Regards
> > JB
> >
> > On 07/04/2018 04:52, Scott Wegner wrote:
> >
> > Here's an end-of-day update on migration work:
> >
> > * Snapshot unsigned dailies and signed release builds are working (!!).
> > PR/5048 [1] merges changes from Luke's branch
> >* python precommit failing... will investigate python precommit Monday
> > * All Precommits are gradle only
> > * All Postcommits except performance tests and Java_JDK_Versions_Test
> > use gradle (after PR/5047 [2] merged)
> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be
> > merged before switching
> > * ValidatesRunner_Spark failing consistently; investigating
> >
> > Thanks for another productive day of hacking. I'll pick up again on
> Monday.
> >
> > [1] https://github.com/apache/beam/pull/5048
> > [2] https://github.com/apache/beam/pull/5047
> >
> >
> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
> > > wrote:
> >
> > Why building a zip per runner which its stack and just pointing out
> > on that zip and let beam lazy load the runner:
> >
> > --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or
> > the fromSystemProperties() if it gets merged a day ;))
> >
> > Le 6 avr. 2018 20:21, "Kenneth Knowles"  > > a écrit :
> >
> > I'm working on finding a solution for launching the Nexmark
> > suite with each runner. This doesn't have to be done via Gradle,
> > but we anyhow need built artifacts that don't require user
> > classpath intervention.
> >
> > It looks to me like the examples are also missing this - they
> > have separate configuration e.g. sparkRunnerPreCommit but that
> > is overspecified compared to a free-form launching of a main()
> > program with a runner profile.
> >
> > On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  > > wrote:
> >
> > Romain, are you talking about the profiles that exist as
> > part of the archetype examples?
> >
> > If so, then those still exist and haven't been changed. If
> > not, can you provide a link to the profile in a pom file to
> > be clearer?
> >
> > On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau
> > 

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

What's the plan to make idea supporting gradle on beam project? Do we
import the workaround mentionned in
https://youtrack.jetbrains.com/issue/IDEA-175172?
For the ones who didn't see this issue in action: idea will compile in
out/ instead of build/ and you will just miss all the resources you
need like some SPI registration which are used by all our registrar =>
no way to run tests in idea without hacking the configuration quite
deeply :(

Romain Manni-Bucau
@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book


2018-04-10 10:08 GMT+02:00 Etienne Chauchot :
> As a gradle beginner, I could not agree more !
> +1
> Etienne
> Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :
>
> Hi all,
>
> I did multiple gradle build since last week and I would like to share
> one of my concern: it's about the communities.
>
> If I think our users won't see any change for them due to Gradle build
> (I think that most of our users will still use Maven with artifacts
> provided by Gradle), I'm more concerned by the dev community and the
> contribution.
>
> Maven is well known and straight forward for a large part of potential
> contributors. I think we have to keep in mind that we still have to grow
> up our contributors community.
>
> Today, maybe I'm wrong, but I have the feeling that gradle build is not
> straight forward (build.gradle includes build_rules.gradle, gathering
> all taks all together).
>
> I would like to add a task in the gradle "migration" process: simplify
> the gradle structure and files, and document this.
>
> I know we already have a Jira about the documentation part, but I would
> like to "polish" and use a clean structure for the Gradle resources. As
> already quickly discussed, I think that having one gradle file per tasks
> in the .gradle directory would be helpful.
>
> The goal is really to simplify the contribution.
>
> Do you agree if I add a Jira about "Gradle polish" ?
> Thoughts ?
>
> Regards
> JB
>
> On 07/04/2018 04:52, Scott Wegner wrote:
>
> Here's an end-of-day update on migration work:
>
> * Snapshot unsigned dailies and signed release builds are working (!!).
> PR/5048 [1] merges changes from Luke's branch
>* python precommit failing... will investigate python precommit Monday
> * All Precommits are gradle only
> * All Postcommits except performance tests and Java_JDK_Versions_Test
> use gradle (after PR/5047 [2] merged)
> * Nightly snapshot release using gradle is ready; needs PR/5048 to be
> merged before switching
> * ValidatesRunner_Spark failing consistently; investigating
>
> Thanks for another productive day of hacking. I'll pick up again on Monday.
>
> [1] https://github.com/apache/beam/pull/5048
> [2] https://github.com/apache/beam/pull/5047
>
>
> On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
> > wrote:
>
> Why building a zip per runner which its stack and just pointing out
> on that zip and let beam lazy load the runner:
>
> --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or
> the fromSystemProperties() if it gets merged a day ;))
>
> Le 6 avr. 2018 20:21, "Kenneth Knowles"  > a écrit :
>
> I'm working on finding a solution for launching the Nexmark
> suite with each runner. This doesn't have to be done via Gradle,
> but we anyhow need built artifacts that don't require user
> classpath intervention.
>
> It looks to me like the examples are also missing this - they
> have separate configuration e.g. sparkRunnerPreCommit but that
> is overspecified compared to a free-form launching of a main()
> program with a runner profile.
>
> On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  > wrote:
>
> Romain, are you talking about the profiles that exist as
> part of the archetype examples?
>
> If so, then those still exist and haven't been changed. If
> not, can you provide a link to the profile in a pom file to
> be clearer?
>
> On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau
> > wrote:
>
> Hi Scott,
>
> is it right that 2 doesn't handle the hierachy anymore
> and that it doesn't handle profiles for runners as it is
> currently with maven?
>
>
> Romain Manni-Bucau
> @rmannibucau  | Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
>
> 

Re: Gradle Status [April 6]

[Etienne Chauchot]

As a gradle beginner, I could not agree more ! 
+1
Etienne
Le lundi 09 avril 2018 à 18:47 +0200, Jean-Baptiste Onofré a écrit :
> Hi all,
> 
> I did multiple gradle build since last week and I would like to share 
> one of my concern: it's about the communities.
> 
> If I think our users won't see any change for them due to Gradle build 
> (I think that most of our users will still use Maven with artifacts 
> provided by Gradle), I'm more concerned by the dev community and the 
> contribution.
> 
> Maven is well known and straight forward for a large part of potential 
> contributors. I think we have to keep in mind that we still have to grow 
> up our contributors community.
> 
> Today, maybe I'm wrong, but I have the feeling that gradle build is not 
> straight forward (build.gradle includes build_rules.gradle, gathering 
> all taks all together).
> 
> I would like to add a task in the gradle "migration" process: simplify 
> the gradle structure and files, and document this.
> 
> I know we already have a Jira about the documentation part, but I would 
> like to "polish" and use a clean structure for the Gradle resources. As 
> already quickly discussed, I think that having one gradle file per tasks 
> in the .gradle directory would be helpful.
> 
> The goal is really to simplify the contribution.
> 
> Do you agree if I add a Jira about "Gradle polish" ?
> Thoughts ?
> 
> Regards
> JB
> 
> On 07/04/2018 04:52, Scott Wegner wrote:
> > 
> > Here's an end-of-day update on migration work:
> > 
> > * Snapshot unsigned dailies and signed release builds are working (!!). 
> > PR/5048 [1] merges changes from Luke's branch
> >    * python precommit failing... will investigate python precommit Monday
> > * All Precommits are gradle only
> > * All Postcommits except performance tests and Java_JDK_Versions_Test  
> > use gradle (after PR/5047 [2] merged)
> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be 
> > merged before switching
> > * ValidatesRunner_Spark failing consistently; investigating
> > 
> > Thanks for another productive day of hacking. I'll pick up again on Monday.
> > 
> > [1] https://github.com/apache/beam/pull/5048
> > [2] https://github.com/apache/beam/pull/5047
> > 
> > 
> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau 
> > > wrote:
> > 
> > Why building a zip per runner which its stack and just pointing out
> > on that zip and let beam lazy load the runner:
> > 
> > --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or
> > the fromSystemProperties() if it gets merged a day ;))
> > 
> > Le 6 avr. 2018 20:21, "Kenneth Knowles"  > k...@google.com>> a écrit :
> > 
> > I'm working on finding a solution for launching the Nexmark
> > suite with each runner. This doesn't have to be done via Gradle,
> > but we anyhow need built artifacts that don't require user
> > classpath intervention.
> > 
> > It looks to me like the examples are also missing this - they
> > have separate configuration e.g. sparkRunnerPreCommit but that
> > is overspecified compared to a free-form launching of a main()
> > program with a runner profile.
> > 
> > On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  > lc...@google.com>> wrote:
> > 
> > Romain, are you talking about the profiles that exist as
> > part of the archetype examples?
> > 
> > If so, then those still exist and haven't been changed. If
> > not, can you provide a link to the profile in a pom file to
> > be clearer?
> > 
> > On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau
> > > wrote:
> > 
> > Hi Scott,
> > 
> > is it right that 2 doesn't handle the hierachy anymore
> > and that it doesn't handle profiles for runners as it is
> > currently with maven?
> > 
> > 
> > Romain Manni-Bucau
> > @rmannibucau  | Blog
> >  | Old Blog
> >  | Github
> >  | LinkedIn
> >  | Book
> > 
> > 
> > 
> > 2018-04-06 18:32 GMT+02:00 Scott Wegner
> > >:
> > 
> > I wanted to start a thread to summarize the current
> > state of Gradle migration. We've made lots of good
> > progress so far this week. Here's the status from
> > what I can tell-- please add or correct anything I

Re: Gradle Status [April 6]

[Lukasz Cwik]

Simplifying examples/java precommit is encompassed in BEAM-4033. Both you
and Kenn pointed to the same thing. To my knowledge it is the only place
where we have such a loop that generates tasks.

On Mon, Apr 9, 2018 at 3:37 PM Romain Manni-Bucau 
wrote:

>
>
> Le 9 avr. 2018 21:04, "Lukasz Cwik"  a écrit :
>
> Romain, can you clarify by the weird task naming? (Give some examples
> using our current project and Gradle and what you would have expected.)
>
>
> Sure
>
>
> https://github.com/apache/beam/blob/fb6ba3bfd43605a0f4944828b5f19e75840fa7aa/examples/java/build.gradle#L78
> for instance
>
> It is way more natural to have one generic command you can select a runner
> (including a csv value) than N commands for this kind of things.
>
> Gradle allows to define custom tasks and "scopes". This must not be used
> at Xtrem until justifiable. We now have a lot of configurations not making
> much sense too until you scripted yourself the script or know beam well.
> This doesnt enable contribs :(.
>
> Hope it is clearer
>
>
> On Mon, Apr 9, 2018 at 2:49 PM Romain Manni-Bucau 
> wrote:
>
>> +1 gradle not being mainstream - and even more in beam scope than other
>> java scope - it must stay simple. I know that personally it takes me x2 or
>> 3 to say "ok ill fork" a project using gradle, in particular with custom
>> scripts and not only parent/module scripts.
>>
>> If needed we can write custom plugins to simply it and hide the
>> complexity and weird task naming can move to "standard" names with
>> options/parameters maybe?
>>
>> Le 9 avr. 2018 20:27, "Kenneth Knowles"  a écrit :
>>
>>> My phrasing sounded a bit too much like adding a blocking condition. I
>>> absolutely don't want to do that. Our maven build was far from clear, and
>>> had lots of tech debt and spooky action at a distance,
>>>
>>> I just want to emphatically agree with JB's sentiment that we have an
>>> opportunity to improve it before it ossifies in the current state. Maybe we
>>> won't really have time, and that's OK too and we can just improve as we go.
>>>
>>> Luke - I know we have different standards of clarity but I doubt we'll
>>> have specific disagreements on cleanups to the build as they happen.* I was
>>> mostly listing elementary programming concepts that apply to our build and
>>> tend to get forgotten in configs & build files. Being "natural gradle" is
>>> also important. Let's see how it goes.
>>>
>>> Kenn
>>>
>>> *FWIW I did know that we have some string-munged identifiers already,
>>> but mostly I am speaking from experience beyond & before Beam too. I'll
>>> likely tidy the examples build.gradle as I finish up nexmark in a similar
>>> way.
>>>
>>> On Mon, Apr 9, 2018 at 10:50 AM Lukasz Cwik  wrote:
>>>
 JB, learning the build system in a project is hopefully avoided by most
 users if the contribution guide can clearly explain what users need to do.
 But for everyone else who wants to change a dependency version or add a
 dependency it should be as simple as copy/paste (which I believe it already
 is). Note that people who want to do anything more complicated like add new
 jenkins job configurations for new integration test targets needs to learn
 how the build system works, how maven profiles work, how to plumb the
 required set of attributes from Jenkins into the test target via the build
 system.

 Kenn, I have to disagree with a lot in 1 and 2. For users who are
 unfamiliar with Maven, we didn't setup the Maven build in such a way that
 anyone unfamiliar with Maven knew what was going on or try to use concepts
 unnatural to Maven in an attempt to make it seem easier. I believe we
 should stick with Gradle/Groovy conventions. Users who are not familiar
 with how Gradle/Groovy works will either ask the community or look at
 StackOverflow and doing things like passing the project around into
 functions is extremely uncommon compared to using the current context and
 applying closures to it. For users who are familiar with Gradle, the builds
 should be natural Gradle.

 *- If you are going to refer to an identifier, it should have an
 explicit point of origin (not strings smashed together in a loop)*
 I assume your referring to how the examples java precommit is done. It
 is the only case of it to my knowledge and extra hands to migrate it to be
 like how the validates runner tests run would be useful. Filed
 https://issues.apache.org/jira/browse/BEAM-4033


 On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles  wrote:

> Huge +1 to the idea of investing in simplification and polish of the
> Gradle files before considering the migration complete.
>
> 1. build.gradle files should be as close to straight-line
> configuration as possible:
>  - You should be able to understand a 

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

Le 9 avr. 2018 21:04, "Lukasz Cwik"  a écrit :

Romain, can you clarify by the weird task naming? (Give some examples using
our current project and Gradle and what you would have expected.)


Sure

https://github.com/apache/beam/blob/fb6ba3bfd43605a0f4944828b5f19e75840fa7aa/examples/java/build.gradle#L78
for instance

It is way more natural to have one generic command you can select a runner
(including a csv value) than N commands for this kind of things.

Gradle allows to define custom tasks and "scopes". This must not be used at
Xtrem until justifiable. We now have a lot of configurations not making
much sense too until you scripted yourself the script or know beam well.
This doesnt enable contribs :(.

Hope it is clearer


On Mon, Apr 9, 2018 at 2:49 PM Romain Manni-Bucau 
wrote:

> +1 gradle not being mainstream - and even more in beam scope than other
> java scope - it must stay simple. I know that personally it takes me x2 or
> 3 to say "ok ill fork" a project using gradle, in particular with custom
> scripts and not only parent/module scripts.
>
> If needed we can write custom plugins to simply it and hide the complexity
> and weird task naming can move to "standard" names with options/parameters
> maybe?
>
> Le 9 avr. 2018 20:27, "Kenneth Knowles"  a écrit :
>
>> My phrasing sounded a bit too much like adding a blocking condition. I
>> absolutely don't want to do that. Our maven build was far from clear, and
>> had lots of tech debt and spooky action at a distance,
>>
>> I just want to emphatically agree with JB's sentiment that we have an
>> opportunity to improve it before it ossifies in the current state. Maybe we
>> won't really have time, and that's OK too and we can just improve as we go.
>>
>> Luke - I know we have different standards of clarity but I doubt we'll
>> have specific disagreements on cleanups to the build as they happen.* I was
>> mostly listing elementary programming concepts that apply to our build and
>> tend to get forgotten in configs & build files. Being "natural gradle" is
>> also important. Let's see how it goes.
>>
>> Kenn
>>
>> *FWIW I did know that we have some string-munged identifiers already, but
>> mostly I am speaking from experience beyond & before Beam too. I'll likely
>> tidy the examples build.gradle as I finish up nexmark in a similar way.
>>
>> On Mon, Apr 9, 2018 at 10:50 AM Lukasz Cwik  wrote:
>>
>>> JB, learning the build system in a project is hopefully avoided by most
>>> users if the contribution guide can clearly explain what users need to do.
>>> But for everyone else who wants to change a dependency version or add a
>>> dependency it should be as simple as copy/paste (which I believe it already
>>> is). Note that people who want to do anything more complicated like add new
>>> jenkins job configurations for new integration test targets needs to learn
>>> how the build system works, how maven profiles work, how to plumb the
>>> required set of attributes from Jenkins into the test target via the build
>>> system.
>>>
>>> Kenn, I have to disagree with a lot in 1 and 2. For users who are
>>> unfamiliar with Maven, we didn't setup the Maven build in such a way that
>>> anyone unfamiliar with Maven knew what was going on or try to use concepts
>>> unnatural to Maven in an attempt to make it seem easier. I believe we
>>> should stick with Gradle/Groovy conventions. Users who are not familiar
>>> with how Gradle/Groovy works will either ask the community or look at
>>> StackOverflow and doing things like passing the project around into
>>> functions is extremely uncommon compared to using the current context and
>>> applying closures to it. For users who are familiar with Gradle, the builds
>>> should be natural Gradle.
>>>
>>> *- If you are going to refer to an identifier, it should have an
>>> explicit point of origin (not strings smashed together in a loop)*
>>> I assume your referring to how the examples java precommit is done. It
>>> is the only case of it to my knowledge and extra hands to migrate it to be
>>> like how the validates runner tests run would be useful. Filed
>>> https://issues.apache.org/jira/browse/BEAM-4033
>>>
>>>
>>> On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles  wrote:
>>>
 Huge +1 to the idea of investing in simplification and polish of the
 Gradle files before considering the migration complete.

 1. build.gradle files should be as close to straight-line configuration
 as possible:
  - You should be able to understand a module's build easily, locally,
 without knowing Groovy
  - As little Groovy programming as possible, focused on providing
 well-defined utilities

 2. Explicit > implicit, especially for config files
  - Ditto about being able to understand a module's build locally
  - Minimize globally inherited config, in favor of explicitly calling
 utilities
  - Passing 

Re: Gradle Status [April 6]

[Jean-Baptiste Onofré]

Hi Luke,

let me take a concrete example.

I'm developing a Beam extension. Most of the time, I will ask myself:

1. How do I do the build.gradle and include my extension in Beam project
2. How do my extension is compile (the equivalent of 
maven-compiler-plugin) and how to deal with dependency and scope

3. How my extension is packaged
4. How my test are executed

Of course, we can document this when we are on the happy path.
Now, imagine, dependency is not correctly resolved, rat is failing, 
findbugs is complaining, etc. I have to check the "core" Beam gradle rules.
Today, I think it's not easy at all to find the corresponding tasks and 
configuration.


I saw several projects storing the tasks in .gradle (like Apache Geode 
for instance).


My point is that I think it's a right timing to polish/cleanup our 
build. Else, I'm afraid we will never do that in the future.


Regards
JB

On 09/04/2018 19:50, Lukasz Cwik wrote:
JB, learning the build system in a project is hopefully avoided by most 
users if the contribution guide can clearly explain what users need to 
do. But for everyone else who wants to change a dependency version or 
add a dependency it should be as simple as copy/paste (which I believe 
it already is). Note that people who want to do anything more 
complicated like add new jenkins job configurations for new integration 
test targets needs to learn how the build system works, how maven 
profiles work, how to plumb the required set of attributes from Jenkins 
into the test target via the build system.


Kenn, I have to disagree with a lot in 1 and 2. For users who are 
unfamiliar with Maven, we didn't setup the Maven build in such a way 
that anyone unfamiliar with Maven knew what was going on or try to use 
concepts unnatural to Maven in an attempt to make it seem easier. I 
believe we should stick with Gradle/Groovy conventions. Users who are 
not familiar with how Gradle/Groovy works will either ask the community 
or look at StackOverflow and doing things like passing the project 
around into functions is extremely uncommon compared to using the 
current context and applying closures to it. For users who are familiar 
with Gradle, the builds should be natural Gradle.


*- If you are going to refer to an identifier, it should have an 
explicit point of origin (not strings smashed together in a loop)*
I assume your referring to how the examples java precommit is done. It 
is the only case of it to my knowledge and extra hands to migrate it to 
be like how the validates runner tests run would be useful. Filed 
https://issues.apache.org/jira/browse/BEAM-4033



On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles > wrote:


Huge +1 to the idea of investing in simplification and polish of the
Gradle files before considering the migration complete.

1. build.gradle files should be as close to straight-line
configuration as possible:
  - You should be able to understand a module's build easily,
locally, without knowing Groovy
  - As little Groovy programming as possible, focused on providing
well-defined utilities

2. Explicit > implicit, especially for config files
  - Ditto about being able to understand a module's build locally
  - Minimize globally inherited config, in favor of explicitly
calling utilities
  - Passing current project to a utility is better than trying to
make something the "closure"
  - If you are going to refer to an identifier, it should have an
explicit point of origin (not strings smashed together in a loop)

When in doubt, a little redundancy to improve readability is worth
it in this context.

Kenn

On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré > wrote:

Hi all,

I did multiple gradle build since last week and I would like to
share
one of my concern: it's about the communities.

If I think our users won't see any change for them due to Gradle
build
(I think that most of our users will still use Maven with artifacts
provided by Gradle), I'm more concerned by the dev community and the
contribution.

Maven is well known and straight forward for a large part of
potential
contributors. I think we have to keep in mind that we still have
to grow
up our contributors community.

Today, maybe I'm wrong, but I have the feeling that gradle build
is not
straight forward (build.gradle includes build_rules.gradle,
gathering
all taks all together).

I would like to add a task in the gradle "migration" process:
simplify
the gradle structure and files, and document this.

I know we already have a Jira about the documentation part, but
I would
like to "polish" and use a clean structure for the Gradle
resources. 

Re: Gradle Status [April 6]

[Lukasz Cwik]

Romain, can you clarify by the weird task naming? (Give some examples using
our current project and Gradle and what you would have expected.)

On Mon, Apr 9, 2018 at 2:49 PM Romain Manni-Bucau 
wrote:

> +1 gradle not being mainstream - and even more in beam scope than other
> java scope - it must stay simple. I know that personally it takes me x2 or
> 3 to say "ok ill fork" a project using gradle, in particular with custom
> scripts and not only parent/module scripts.
>
> If needed we can write custom plugins to simply it and hide the complexity
> and weird task naming can move to "standard" names with options/parameters
> maybe?
>
> Le 9 avr. 2018 20:27, "Kenneth Knowles"  a écrit :
>
>> My phrasing sounded a bit too much like adding a blocking condition. I
>> absolutely don't want to do that. Our maven build was far from clear, and
>> had lots of tech debt and spooky action at a distance,
>>
>> I just want to emphatically agree with JB's sentiment that we have an
>> opportunity to improve it before it ossifies in the current state. Maybe we
>> won't really have time, and that's OK too and we can just improve as we go.
>>
>> Luke - I know we have different standards of clarity but I doubt we'll
>> have specific disagreements on cleanups to the build as they happen.* I was
>> mostly listing elementary programming concepts that apply to our build and
>> tend to get forgotten in configs & build files. Being "natural gradle" is
>> also important. Let's see how it goes.
>>
>> Kenn
>>
>> *FWIW I did know that we have some string-munged identifiers already, but
>> mostly I am speaking from experience beyond & before Beam too. I'll likely
>> tidy the examples build.gradle as I finish up nexmark in a similar way.
>>
>> On Mon, Apr 9, 2018 at 10:50 AM Lukasz Cwik  wrote:
>>
>>> JB, learning the build system in a project is hopefully avoided by most
>>> users if the contribution guide can clearly explain what users need to do.
>>> But for everyone else who wants to change a dependency version or add a
>>> dependency it should be as simple as copy/paste (which I believe it already
>>> is). Note that people who want to do anything more complicated like add new
>>> jenkins job configurations for new integration test targets needs to learn
>>> how the build system works, how maven profiles work, how to plumb the
>>> required set of attributes from Jenkins into the test target via the build
>>> system.
>>>
>>> Kenn, I have to disagree with a lot in 1 and 2. For users who are
>>> unfamiliar with Maven, we didn't setup the Maven build in such a way that
>>> anyone unfamiliar with Maven knew what was going on or try to use concepts
>>> unnatural to Maven in an attempt to make it seem easier. I believe we
>>> should stick with Gradle/Groovy conventions. Users who are not familiar
>>> with how Gradle/Groovy works will either ask the community or look at
>>> StackOverflow and doing things like passing the project around into
>>> functions is extremely uncommon compared to using the current context and
>>> applying closures to it. For users who are familiar with Gradle, the builds
>>> should be natural Gradle.
>>>
>>> *- If you are going to refer to an identifier, it should have an
>>> explicit point of origin (not strings smashed together in a loop)*
>>> I assume your referring to how the examples java precommit is done. It
>>> is the only case of it to my knowledge and extra hands to migrate it to be
>>> like how the validates runner tests run would be useful. Filed
>>> https://issues.apache.org/jira/browse/BEAM-4033
>>>
>>>
>>> On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles  wrote:
>>>
 Huge +1 to the idea of investing in simplification and polish of the
 Gradle files before considering the migration complete.

 1. build.gradle files should be as close to straight-line configuration
 as possible:
  - You should be able to understand a module's build easily, locally,
 without knowing Groovy
  - As little Groovy programming as possible, focused on providing
 well-defined utilities

 2. Explicit > implicit, especially for config files
  - Ditto about being able to understand a module's build locally
  - Minimize globally inherited config, in favor of explicitly calling
 utilities
  - Passing current project to a utility is better than trying to make
 something the "closure"
  - If you are going to refer to an identifier, it should have an
 explicit point of origin (not strings smashed together in a loop)

 When in doubt, a little redundancy to improve readability is worth it
 in this context.

 Kenn

 On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré 
 wrote:

> Hi all,
>
> I did multiple gradle build since last week and I would like to share
> one of my concern: it's about the communities.
>
> If I 

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

+1 gradle not being mainstream - and even more in beam scope than other
java scope - it must stay simple. I know that personally it takes me x2 or
3 to say "ok ill fork" a project using gradle, in particular with custom
scripts and not only parent/module scripts.

If needed we can write custom plugins to simply it and hide the complexity
and weird task naming can move to "standard" names with options/parameters
maybe?

Le 9 avr. 2018 20:27, "Kenneth Knowles"  a écrit :

> My phrasing sounded a bit too much like adding a blocking condition. I
> absolutely don't want to do that. Our maven build was far from clear, and
> had lots of tech debt and spooky action at a distance,
>
> I just want to emphatically agree with JB's sentiment that we have an
> opportunity to improve it before it ossifies in the current state. Maybe we
> won't really have time, and that's OK too and we can just improve as we go.
>
> Luke - I know we have different standards of clarity but I doubt we'll
> have specific disagreements on cleanups to the build as they happen.* I was
> mostly listing elementary programming concepts that apply to our build and
> tend to get forgotten in configs & build files. Being "natural gradle" is
> also important. Let's see how it goes.
>
> Kenn
>
> *FWIW I did know that we have some string-munged identifiers already, but
> mostly I am speaking from experience beyond & before Beam too. I'll likely
> tidy the examples build.gradle as I finish up nexmark in a similar way.
>
> On Mon, Apr 9, 2018 at 10:50 AM Lukasz Cwik  wrote:
>
>> JB, learning the build system in a project is hopefully avoided by most
>> users if the contribution guide can clearly explain what users need to do.
>> But for everyone else who wants to change a dependency version or add a
>> dependency it should be as simple as copy/paste (which I believe it already
>> is). Note that people who want to do anything more complicated like add new
>> jenkins job configurations for new integration test targets needs to learn
>> how the build system works, how maven profiles work, how to plumb the
>> required set of attributes from Jenkins into the test target via the build
>> system.
>>
>> Kenn, I have to disagree with a lot in 1 and 2. For users who are
>> unfamiliar with Maven, we didn't setup the Maven build in such a way that
>> anyone unfamiliar with Maven knew what was going on or try to use concepts
>> unnatural to Maven in an attempt to make it seem easier. I believe we
>> should stick with Gradle/Groovy conventions. Users who are not familiar
>> with how Gradle/Groovy works will either ask the community or look at
>> StackOverflow and doing things like passing the project around into
>> functions is extremely uncommon compared to using the current context and
>> applying closures to it. For users who are familiar with Gradle, the builds
>> should be natural Gradle.
>>
>> *- If you are going to refer to an identifier, it should have an explicit
>> point of origin (not strings smashed together in a loop)*
>> I assume your referring to how the examples java precommit is done. It is
>> the only case of it to my knowledge and extra hands to migrate it to be
>> like how the validates runner tests run would be useful. Filed
>> https://issues.apache.org/jira/browse/BEAM-4033
>>
>>
>> On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles  wrote:
>>
>>> Huge +1 to the idea of investing in simplification and polish of the
>>> Gradle files before considering the migration complete.
>>>
>>> 1. build.gradle files should be as close to straight-line configuration
>>> as possible:
>>>  - You should be able to understand a module's build easily, locally,
>>> without knowing Groovy
>>>  - As little Groovy programming as possible, focused on providing
>>> well-defined utilities
>>>
>>> 2. Explicit > implicit, especially for config files
>>>  - Ditto about being able to understand a module's build locally
>>>  - Minimize globally inherited config, in favor of explicitly calling
>>> utilities
>>>  - Passing current project to a utility is better than trying to make
>>> something the "closure"
>>>  - If you are going to refer to an identifier, it should have an
>>> explicit point of origin (not strings smashed together in a loop)
>>>
>>> When in doubt, a little redundancy to improve readability is worth it in
>>> this context.
>>>
>>> Kenn
>>>
>>> On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré 
>>> wrote:
>>>
 Hi all,

 I did multiple gradle build since last week and I would like to share
 one of my concern: it's about the communities.

 If I think our users won't see any change for them due to Gradle build
 (I think that most of our users will still use Maven with artifacts
 provided by Gradle), I'm more concerned by the dev community and the
 contribution.

 Maven is well known and straight forward for a large part of potential
 contributors. 

Re: Gradle Status [April 6]

[Kenneth Knowles]

My phrasing sounded a bit too much like adding a blocking condition. I
absolutely don't want to do that. Our maven build was far from clear, and
had lots of tech debt and spooky action at a distance,

I just want to emphatically agree with JB's sentiment that we have an
opportunity to improve it before it ossifies in the current state. Maybe we
won't really have time, and that's OK too and we can just improve as we go.

Luke - I know we have different standards of clarity but I doubt we'll have
specific disagreements on cleanups to the build as they happen.* I was
mostly listing elementary programming concepts that apply to our build and
tend to get forgotten in configs & build files. Being "natural gradle" is
also important. Let's see how it goes.

Kenn

*FWIW I did know that we have some string-munged identifiers already, but
mostly I am speaking from experience beyond & before Beam too. I'll likely
tidy the examples build.gradle as I finish up nexmark in a similar way.

On Mon, Apr 9, 2018 at 10:50 AM Lukasz Cwik  wrote:

> JB, learning the build system in a project is hopefully avoided by most
> users if the contribution guide can clearly explain what users need to do.
> But for everyone else who wants to change a dependency version or add a
> dependency it should be as simple as copy/paste (which I believe it already
> is). Note that people who want to do anything more complicated like add new
> jenkins job configurations for new integration test targets needs to learn
> how the build system works, how maven profiles work, how to plumb the
> required set of attributes from Jenkins into the test target via the build
> system.
>
> Kenn, I have to disagree with a lot in 1 and 2. For users who are
> unfamiliar with Maven, we didn't setup the Maven build in such a way that
> anyone unfamiliar with Maven knew what was going on or try to use concepts
> unnatural to Maven in an attempt to make it seem easier. I believe we
> should stick with Gradle/Groovy conventions. Users who are not familiar
> with how Gradle/Groovy works will either ask the community or look at
> StackOverflow and doing things like passing the project around into
> functions is extremely uncommon compared to using the current context and
> applying closures to it. For users who are familiar with Gradle, the builds
> should be natural Gradle.
>
> *- If you are going to refer to an identifier, it should have an explicit
> point of origin (not strings smashed together in a loop)*
> I assume your referring to how the examples java precommit is done. It is
> the only case of it to my knowledge and extra hands to migrate it to be
> like how the validates runner tests run would be useful. Filed
> https://issues.apache.org/jira/browse/BEAM-4033
>
>
> On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles  wrote:
>
>> Huge +1 to the idea of investing in simplification and polish of the
>> Gradle files before considering the migration complete.
>>
>> 1. build.gradle files should be as close to straight-line configuration
>> as possible:
>>  - You should be able to understand a module's build easily, locally,
>> without knowing Groovy
>>  - As little Groovy programming as possible, focused on providing
>> well-defined utilities
>>
>> 2. Explicit > implicit, especially for config files
>>  - Ditto about being able to understand a module's build locally
>>  - Minimize globally inherited config, in favor of explicitly calling
>> utilities
>>  - Passing current project to a utility is better than trying to make
>> something the "closure"
>>  - If you are going to refer to an identifier, it should have an explicit
>> point of origin (not strings smashed together in a loop)
>>
>> When in doubt, a little redundancy to improve readability is worth it in
>> this context.
>>
>> Kenn
>>
>> On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré 
>> wrote:
>>
>>> Hi all,
>>>
>>> I did multiple gradle build since last week and I would like to share
>>> one of my concern: it's about the communities.
>>>
>>> If I think our users won't see any change for them due to Gradle build
>>> (I think that most of our users will still use Maven with artifacts
>>> provided by Gradle), I'm more concerned by the dev community and the
>>> contribution.
>>>
>>> Maven is well known and straight forward for a large part of potential
>>> contributors. I think we have to keep in mind that we still have to grow
>>> up our contributors community.
>>>
>>> Today, maybe I'm wrong, but I have the feeling that gradle build is not
>>> straight forward (build.gradle includes build_rules.gradle, gathering
>>> all taks all together).
>>>
>>> I would like to add a task in the gradle "migration" process: simplify
>>> the gradle structure and files, and document this.
>>>
>>> I know we already have a Jira about the documentation part, but I would
>>> like to "polish" and use a clean structure for the Gradle resources. As
>>> already quickly discussed, I 

Re: Gradle Status [April 6]

[Lukasz Cwik]

Fixed up prior e-mail.

On Mon, Apr 9, 2018 at 1:50 PM Lukasz Cwik  wrote:

> JB, learning the build system in a project is hopefully avoided by most
> users if the contribution guide can clearly explain what users need to do.
> But for everyone else who wants to change a dependency version or add a
> dependency it should be as simple as copy/paste (which I believe it already
> is). Note that people who want to do anything more complicated like add new
> jenkins job configurations for new integration test targets needs to learn
> how the build system works, how to plumb the required set of attributes
> from Jenkins into the test target via the build system.
>
> Kenn, I have to disagree with a lot in 1 and 2. For users who are
> unfamiliar with Maven, we didn't setup the Maven build in such a way that
> anyone unfamiliar with Maven knew what was going on or try to use concepts
> unnatural to Maven in an attempt to make it seem easier. I believe we
> should stick with Gradle/Groovy conventions. Users who are not familiar
> with how Gradle/Groovy works will either ask the community or look at
> StackOverflow and doing things like passing the project around into
> functions is extremely uncommon compared to using the current context and
> applying closures to it. For users who are familiar with Gradle, the builds
> should be natural Gradle.
>
> *- If you are going to refer to an identifier, it should have an explicit
> point of origin (not strings smashed together in a loop)*
> I assume your referring to how the examples java precommit is done. It is
> the only case of it to my knowledge and extra hands to migrate it to be
> like how the validates runner tests run would be useful. Filed
> https://issues.apache.org/jira/browse/BEAM-4033
>
>
> On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles  wrote:
>
>> Huge +1 to the idea of investing in simplification and polish of the
>> Gradle files before considering the migration complete.
>>
>> 1. build.gradle files should be as close to straight-line configuration
>> as possible:
>>  - You should be able to understand a module's build easily, locally,
>> without knowing Groovy
>>  - As little Groovy programming as possible, focused on providing
>> well-defined utilities
>>
>> 2. Explicit > implicit, especially for config files
>>  - Ditto about being able to understand a module's build locally
>>  - Minimize globally inherited config, in favor of explicitly calling
>> utilities
>>  - Passing current project to a utility is better than trying to make
>> something the "closure"
>>  - If you are going to refer to an identifier, it should have an explicit
>> point of origin (not strings smashed together in a loop)
>>
>> When in doubt, a little redundancy to improve readability is worth it in
>> this context.
>>
>> Kenn
>>
>> On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré 
>> wrote:
>>
>>> Hi all,
>>>
>>> I did multiple gradle build since last week and I would like to share
>>> one of my concern: it's about the communities.
>>>
>>> If I think our users won't see any change for them due to Gradle build
>>> (I think that most of our users will still use Maven with artifacts
>>> provided by Gradle), I'm more concerned by the dev community and the
>>> contribution.
>>>
>>> Maven is well known and straight forward for a large part of potential
>>> contributors. I think we have to keep in mind that we still have to grow
>>> up our contributors community.
>>>
>>> Today, maybe I'm wrong, but I have the feeling that gradle build is not
>>> straight forward (build.gradle includes build_rules.gradle, gathering
>>> all taks all together).
>>>
>>> I would like to add a task in the gradle "migration" process: simplify
>>> the gradle structure and files, and document this.
>>>
>>> I know we already have a Jira about the documentation part, but I would
>>> like to "polish" and use a clean structure for the Gradle resources. As
>>> already quickly discussed, I think that having one gradle file per tasks
>>> in the .gradle directory would be helpful.
>>>
>>> The goal is really to simplify the contribution.
>>>
>>> Do you agree if I add a Jira about "Gradle polish" ?
>>> Thoughts ?
>>>
>>> Regards
>>> JB
>>>
>>> On 07/04/2018 04:52, Scott Wegner wrote:
>>> > Here's an end-of-day update on migration work:
>>> >
>>> > * Snapshot unsigned dailies and signed release builds are working (!!).
>>> > PR/5048 [1] merges changes from Luke's branch
>>> >* python precommit failing... will investigate python precommit
>>> Monday
>>> > * All Precommits are gradle only
>>> > * All Postcommits except performance tests and Java_JDK_Versions_Test
>>> > use gradle (after PR/5047 [2] merged)
>>> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be
>>> > merged before switching
>>> > * ValidatesRunner_Spark failing consistently; investigating
>>> >
>>> > Thanks for another productive day of hacking. I'll pick up again on
>>> Monday.

Re: Gradle Status [April 6]

[Lukasz Cwik]

JB, learning the build system in a project is hopefully avoided by most
users if the contribution guide can clearly explain what users need to do.
But for everyone else who wants to change a dependency version or add a
dependency it should be as simple as copy/paste (which I believe it already
is). Note that people who want to do anything more complicated like add new
jenkins job configurations for new integration test targets needs to learn
how the build system works, how maven profiles work, how to plumb the
required set of attributes from Jenkins into the test target via the build
system.

Kenn, I have to disagree with a lot in 1 and 2. For users who are
unfamiliar with Maven, we didn't setup the Maven build in such a way that
anyone unfamiliar with Maven knew what was going on or try to use concepts
unnatural to Maven in an attempt to make it seem easier. I believe we
should stick with Gradle/Groovy conventions. Users who are not familiar
with how Gradle/Groovy works will either ask the community or look at
StackOverflow and doing things like passing the project around into
functions is extremely uncommon compared to using the current context and
applying closures to it. For users who are familiar with Gradle, the builds
should be natural Gradle.

*- If you are going to refer to an identifier, it should have an explicit
point of origin (not strings smashed together in a loop)*
I assume your referring to how the examples java precommit is done. It is
the only case of it to my knowledge and extra hands to migrate it to be
like how the validates runner tests run would be useful. Filed
https://issues.apache.org/jira/browse/BEAM-4033


On Mon, Apr 9, 2018 at 1:24 PM Kenneth Knowles  wrote:

> Huge +1 to the idea of investing in simplification and polish of the
> Gradle files before considering the migration complete.
>
> 1. build.gradle files should be as close to straight-line configuration as
> possible:
>  - You should be able to understand a module's build easily, locally,
> without knowing Groovy
>  - As little Groovy programming as possible, focused on providing
> well-defined utilities
>
> 2. Explicit > implicit, especially for config files
>  - Ditto about being able to understand a module's build locally
>  - Minimize globally inherited config, in favor of explicitly calling
> utilities
>  - Passing current project to a utility is better than trying to make
> something the "closure"
>  - If you are going to refer to an identifier, it should have an explicit
> point of origin (not strings smashed together in a loop)
>
> When in doubt, a little redundancy to improve readability is worth it in
> this context.
>
> Kenn
>
> On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré 
> wrote:
>
>> Hi all,
>>
>> I did multiple gradle build since last week and I would like to share
>> one of my concern: it's about the communities.
>>
>> If I think our users won't see any change for them due to Gradle build
>> (I think that most of our users will still use Maven with artifacts
>> provided by Gradle), I'm more concerned by the dev community and the
>> contribution.
>>
>> Maven is well known and straight forward for a large part of potential
>> contributors. I think we have to keep in mind that we still have to grow
>> up our contributors community.
>>
>> Today, maybe I'm wrong, but I have the feeling that gradle build is not
>> straight forward (build.gradle includes build_rules.gradle, gathering
>> all taks all together).
>>
>> I would like to add a task in the gradle "migration" process: simplify
>> the gradle structure and files, and document this.
>>
>> I know we already have a Jira about the documentation part, but I would
>> like to "polish" and use a clean structure for the Gradle resources. As
>> already quickly discussed, I think that having one gradle file per tasks
>> in the .gradle directory would be helpful.
>>
>> The goal is really to simplify the contribution.
>>
>> Do you agree if I add a Jira about "Gradle polish" ?
>> Thoughts ?
>>
>> Regards
>> JB
>>
>> On 07/04/2018 04:52, Scott Wegner wrote:
>> > Here's an end-of-day update on migration work:
>> >
>> > * Snapshot unsigned dailies and signed release builds are working (!!).
>> > PR/5048 [1] merges changes from Luke's branch
>> >* python precommit failing... will investigate python precommit
>> Monday
>> > * All Precommits are gradle only
>> > * All Postcommits except performance tests and Java_JDK_Versions_Test
>> > use gradle (after PR/5047 [2] merged)
>> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be
>> > merged before switching
>> > * ValidatesRunner_Spark failing consistently; investigating
>> >
>> > Thanks for another productive day of hacking. I'll pick up again on
>> Monday.
>> >
>> > [1] https://github.com/apache/beam/pull/5048
>> > [2] https://github.com/apache/beam/pull/5047
>> >
>> >
>> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
>> > 

Re: Gradle Status [April 6]

[Kenneth Knowles]

Huge +1 to the idea of investing in simplification and polish of the Gradle
files before considering the migration complete.

1. build.gradle files should be as close to straight-line configuration as
possible:
 - You should be able to understand a module's build easily, locally,
without knowing Groovy
 - As little Groovy programming as possible, focused on providing
well-defined utilities

2. Explicit > implicit, especially for config files
 - Ditto about being able to understand a module's build locally
 - Minimize globally inherited config, in favor of explicitly calling
utilities
 - Passing current project to a utility is better than trying to make
something the "closure"
 - If you are going to refer to an identifier, it should have an explicit
point of origin (not strings smashed together in a loop)

When in doubt, a little redundancy to improve readability is worth it in
this context.

Kenn

On Mon, Apr 9, 2018 at 9:47 AM Jean-Baptiste Onofré  wrote:

> Hi all,
>
> I did multiple gradle build since last week and I would like to share
> one of my concern: it's about the communities.
>
> If I think our users won't see any change for them due to Gradle build
> (I think that most of our users will still use Maven with artifacts
> provided by Gradle), I'm more concerned by the dev community and the
> contribution.
>
> Maven is well known and straight forward for a large part of potential
> contributors. I think we have to keep in mind that we still have to grow
> up our contributors community.
>
> Today, maybe I'm wrong, but I have the feeling that gradle build is not
> straight forward (build.gradle includes build_rules.gradle, gathering
> all taks all together).
>
> I would like to add a task in the gradle "migration" process: simplify
> the gradle structure and files, and document this.
>
> I know we already have a Jira about the documentation part, but I would
> like to "polish" and use a clean structure for the Gradle resources. As
> already quickly discussed, I think that having one gradle file per tasks
> in the .gradle directory would be helpful.
>
> The goal is really to simplify the contribution.
>
> Do you agree if I add a Jira about "Gradle polish" ?
> Thoughts ?
>
> Regards
> JB
>
> On 07/04/2018 04:52, Scott Wegner wrote:
> > Here's an end-of-day update on migration work:
> >
> > * Snapshot unsigned dailies and signed release builds are working (!!).
> > PR/5048 [1] merges changes from Luke's branch
> >* python precommit failing... will investigate python precommit Monday
> > * All Precommits are gradle only
> > * All Postcommits except performance tests and Java_JDK_Versions_Test
> > use gradle (after PR/5047 [2] merged)
> > * Nightly snapshot release using gradle is ready; needs PR/5048 to be
> > merged before switching
> > * ValidatesRunner_Spark failing consistently; investigating
> >
> > Thanks for another productive day of hacking. I'll pick up again on
> Monday.
> >
> > [1] https://github.com/apache/beam/pull/5048
> > [2] https://github.com/apache/beam/pull/5047
> >
> >
> > On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau
> > > wrote:
> >
> > Why building a zip per runner which its stack and just pointing out
> > on that zip and let beam lazy load the runner:
> >
> > --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or
> > the fromSystemProperties() if it gets merged a day ;))
> >
> > Le 6 avr. 2018 20:21, "Kenneth Knowles"  > > a écrit :
> >
> > I'm working on finding a solution for launching the Nexmark
> > suite with each runner. This doesn't have to be done via Gradle,
> > but we anyhow need built artifacts that don't require user
> > classpath intervention.
> >
> > It looks to me like the examples are also missing this - they
> > have separate configuration e.g. sparkRunnerPreCommit but that
> > is overspecified compared to a free-form launching of a main()
> > program with a runner profile.
> >
> > On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  > > wrote:
> >
> > Romain, are you talking about the profiles that exist as
> > part of the archetype examples?
> >
> > If so, then those still exist and haven't been changed. If
> > not, can you provide a link to the profile in a pom file to
> > be clearer?
> >
> > On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau
> > >
> wrote:
> >
> > Hi Scott,
> >
> > is it right that 2 doesn't handle the hierachy anymore
> > and that it doesn't handle profiles for runners as it is
> > currently with maven?
> >
> >
> > Romain Manni-Bucau
> > 

Re: Gradle Status [April 6]

[Jean-Baptiste Onofré]

Hi all,

I did multiple gradle build since last week and I would like to share 
one of my concern: it's about the communities.


If I think our users won't see any change for them due to Gradle build 
(I think that most of our users will still use Maven with artifacts 
provided by Gradle), I'm more concerned by the dev community and the 
contribution.


Maven is well known and straight forward for a large part of potential 
contributors. I think we have to keep in mind that we still have to grow 
up our contributors community.


Today, maybe I'm wrong, but I have the feeling that gradle build is not 
straight forward (build.gradle includes build_rules.gradle, gathering 
all taks all together).


I would like to add a task in the gradle "migration" process: simplify 
the gradle structure and files, and document this.


I know we already have a Jira about the documentation part, but I would 
like to "polish" and use a clean structure for the Gradle resources. As 
already quickly discussed, I think that having one gradle file per tasks 
in the .gradle directory would be helpful.


The goal is really to simplify the contribution.

Do you agree if I add a Jira about "Gradle polish" ?
Thoughts ?

Regards
JB

On 07/04/2018 04:52, Scott Wegner wrote:

Here's an end-of-day update on migration work:

* Snapshot unsigned dailies and signed release builds are working (!!). 
PR/5048 [1] merges changes from Luke's branch

   * python precommit failing... will investigate python precommit Monday
* All Precommits are gradle only
* All Postcommits except performance tests and Java_JDK_Versions_Test  
use gradle (after PR/5047 [2] merged)
* Nightly snapshot release using gradle is ready; needs PR/5048 to be 
merged before switching

* ValidatesRunner_Spark failing consistently; investigating

Thanks for another productive day of hacking. I'll pick up again on Monday.

[1] https://github.com/apache/beam/pull/5048
[2] https://github.com/apache/beam/pull/5047


On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau 
> wrote:


Why building a zip per runner which its stack and just pointing out
on that zip and let beam lazy load the runner:

--runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or
the fromSystemProperties() if it gets merged a day ;))

Le 6 avr. 2018 20:21, "Kenneth Knowles" > a écrit :

I'm working on finding a solution for launching the Nexmark
suite with each runner. This doesn't have to be done via Gradle,
but we anyhow need built artifacts that don't require user
classpath intervention.

It looks to me like the examples are also missing this - they
have separate configuration e.g. sparkRunnerPreCommit but that
is overspecified compared to a free-form launching of a main()
program with a runner profile.

On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik > wrote:

Romain, are you talking about the profiles that exist as
part of the archetype examples?

If so, then those still exist and haven't been changed. If
not, can you provide a link to the profile in a pom file to
be clearer?

On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau
> wrote:

Hi Scott,

is it right that 2 doesn't handle the hierachy anymore
and that it doesn't handle profiles for runners as it is
currently with maven?


Romain Manni-Bucau
@rmannibucau  | Blog
 | Old Blog
 | Github
 | LinkedIn
 | Book



2018-04-06 18:32 GMT+02:00 Scott Wegner
>:

I wanted to start a thread to summarize the current
state of Gradle migration. We've made lots of good
progress so far this week. Here's the status from
what I can tell-- please add or correct anything I
missed:

* Release artifacts can be built and published for
Snapshot and officlal releases [1]
* Gradle-generated releases have been validated with
the the Apache Beam archetype generation quickstart;
still needs additional validation.
* Generated release pom files have correct project
metadata 

Re: Gradle Status [April 6]

[Dan Halperin]

On Sat, Apr 7, 2018 at 12:43 Reuven Lax  wrote:

> So if I understand correctly, we've migrated all precommit, most
> postcommits, and we have a working release process using Gradle. There are
> a few bugs left, but at this pace it sounds like we're close to fully
> migrated.
>
> I know that multiple people put it long hours last getting this done last
> week (just look at the Slack messages!). This is awesome progress, and a
> hearty thank you to everyone who put in their time.
>
> Reuven
>
> On Fri, Apr 6, 2018 at 7:52 PM Scott Wegner  wrote:
>
>> Here's an end-of-day update on migration work:
>>
>> * Snapshot unsigned dailies and signed release builds are working (!!).
>> PR/5048 [1] merges changes from Luke's branch
>>   * python precommit failing... will investigate python precommit Monday
>> * All Precommits are gradle only
>> * All Postcommits except performance tests and Java_JDK_Versions_Test
>> use gradle (after PR/5047 [2] merged)
>> * Nightly snapshot release using gradle is ready; needs PR/5048 to be
>> merged before switching
>> * ValidatesRunner_Spark failing consistently; investigating
>>
>> Thanks for another productive day of hacking. I'll pick up again on
>> Monday.
>>
>> [1] https://github.com/apache/beam/pull/5048
>> [2] https://github.com/apache/beam/pull/5047
>>
>>
This is really phenomenal work, and a huge boost to the community. Thanks,
everyone who participated!
Dan


>
>> On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau 
>> wrote:
>>
>>> Why building a zip per runner which its stack and just pointing out on
>>> that zip and let beam lazy load the runner:
>>>
>>> --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or the
>>> fromSystemProperties() if it gets merged a day ;))
>>>
>>> Le 6 avr. 2018 20:21, "Kenneth Knowles"  a écrit :
>>>
 I'm working on finding a solution for launching the Nexmark suite with
 each runner. This doesn't have to be done via Gradle, but we anyhow need
 built artifacts that don't require user classpath intervention.

 It looks to me like the examples are also missing this - they have
 separate configuration e.g. sparkRunnerPreCommit but that is overspecified
 compared to a free-form launching of a main() program with a runner 
 profile.

 On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  wrote:

> Romain, are you talking about the profiles that exist as part of the
> archetype examples?
>
> If so, then those still exist and haven't been changed. If not, can
> you provide a link to the profile in a pom file to be clearer?
>
> On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Hi Scott,
>>
>> is it right that 2 doesn't handle the hierachy anymore and that it
>> doesn't handle profiles for runners as it is currently with maven?
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>> 2018-04-06 18:32 GMT+02:00 Scott Wegner :
>>
>>> I wanted to start a thread to summarize the current state of Gradle
>>> migration. We've made lots of good progress so far this week. Here's the
>>> status from what I can tell-- please add or correct anything I missed:
>>>
>>> * Release artifacts can be built and published for Snapshot and
>>> officlal releases [1]
>>> * Gradle-generated releases have been validated with the the Apache
>>> Beam archetype generation quickstart; still needs additional validation.
>>> * Generated release pom files have correct project metadata [2]
>>> * The python pre-commits are now working in Gradle [3]
>>> * Ismaël has started a collaborative doc of Gradle tips [4] as we
>>> all learn the new system-- please add your own. This will eventually 
>>> feed
>>> into official documentation on the website.
>>> * Łukasz Gajowy is working on migrating performance testing
>>> framework [5]
>>> * Daniel is working on updating documentation to refer to Gradle
>>> instead of maven
>>>
>>> If I missed anything, please add it to this thread.
>>>
>>> The general roadmap we're working towards is:
>>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed
>>> releases)
>>> (b) Postcommits migrated to Gradle
>>> (c) Migrate documentation from maven to Gradle
>>> (d) Migrate perfkit suites to use Gradle
>>>
>>> For those of you that are hacking: thanks for your help so far!
>>> Progress is being roughly tracked on the Kanban 

Re: Gradle Status [April 6]

[Reuven Lax]

So if I understand correctly, we've migrated all precommit, most
postcommits, and we have a working release process using Gradle. There are
a few bugs left, but at this pace it sounds like we're close to fully
migrated.

I know that multiple people put it long hours last getting this done last
week (just look at the Slack messages!). This is awesome progress, and a
hearty thank you to everyone who put in their time.

Reuven

On Fri, Apr 6, 2018 at 7:52 PM Scott Wegner  wrote:

> Here's an end-of-day update on migration work:
>
> * Snapshot unsigned dailies and signed release builds are working (!!).
> PR/5048 [1] merges changes from Luke's branch
>   * python precommit failing... will investigate python precommit Monday
> * All Precommits are gradle only
> * All Postcommits except performance tests and Java_JDK_Versions_Test  use
> gradle (after PR/5047 [2] merged)
> * Nightly snapshot release using gradle is ready; needs PR/5048 to be
> merged before switching
> * ValidatesRunner_Spark failing consistently; investigating
>
> Thanks for another productive day of hacking. I'll pick up again on Monday.
>
> [1] https://github.com/apache/beam/pull/5048
> [2] https://github.com/apache/beam/pull/5047
>
>
> On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau 
> wrote:
>
>> Why building a zip per runner which its stack and just pointing out on
>> that zip and let beam lazy load the runner:
>>
>> --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or the
>> fromSystemProperties() if it gets merged a day ;))
>>
>> Le 6 avr. 2018 20:21, "Kenneth Knowles"  a écrit :
>>
>>> I'm working on finding a solution for launching the Nexmark suite with
>>> each runner. This doesn't have to be done via Gradle, but we anyhow need
>>> built artifacts that don't require user classpath intervention.
>>>
>>> It looks to me like the examples are also missing this - they have
>>> separate configuration e.g. sparkRunnerPreCommit but that is overspecified
>>> compared to a free-form launching of a main() program with a runner profile.
>>>
>>> On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  wrote:
>>>
 Romain, are you talking about the profiles that exist as part of the
 archetype examples?

 If so, then those still exist and haven't been changed. If not, can you
 provide a link to the profile in a pom file to be clearer?

 On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Hi Scott,
>
> is it right that 2 doesn't handle the hierachy anymore and that it
> doesn't handle profiles for runners as it is currently with maven?
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-04-06 18:32 GMT+02:00 Scott Wegner :
>
>> I wanted to start a thread to summarize the current state of Gradle
>> migration. We've made lots of good progress so far this week. Here's the
>> status from what I can tell-- please add or correct anything I missed:
>>
>> * Release artifacts can be built and published for Snapshot and
>> officlal releases [1]
>> * Gradle-generated releases have been validated with the the Apache
>> Beam archetype generation quickstart; still needs additional validation.
>> * Generated release pom files have correct project metadata [2]
>> * The python pre-commits are now working in Gradle [3]
>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>> learn the new system-- please add your own. This will eventually feed 
>> into
>> official documentation on the website.
>> * Łukasz Gajowy is working on migrating performance testing framework
>> [5]
>> * Daniel is working on updating documentation to refer to Gradle
>> instead of maven
>>
>> If I missed anything, please add it to this thread.
>>
>> The general roadmap we're working towards is:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed
>> releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> For those of you that are hacking: thanks for your help so far!
>> Progress is being roughly tracked on the Kanban [6]; please make sure the
>> issues assigned to you are up-to-date. Many of the changes are staged on
>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>
>>
>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>> [2] 

Re: Gradle Status [April 6]

[Scott Wegner]

Here's an end-of-day update on migration work:

* Snapshot unsigned dailies and signed release builds are working (!!).
PR/5048 [1] merges changes from Luke's branch
  * python precommit failing... will investigate python precommit Monday
* All Precommits are gradle only
* All Postcommits except performance tests and Java_JDK_Versions_Test  use
gradle (after PR/5047 [2] merged)
* Nightly snapshot release using gradle is ready; needs PR/5048 to be
merged before switching
* ValidatesRunner_Spark failing consistently; investigating

Thanks for another productive day of hacking. I'll pick up again on Monday.

[1] https://github.com/apache/beam/pull/5048
[2] https://github.com/apache/beam/pull/5047


On Fri, Apr 6, 2018 at 11:24 AM Romain Manni-Bucau 
wrote:

> Why building a zip per runner which its stack and just pointing out on
> that zip and let beam lazy load the runner:
>
> --runner=LazyRunner --lazyRunnerDir=... --lazyRunnerOptions=... (or the
> fromSystemProperties() if it gets merged a day ;))
>
> Le 6 avr. 2018 20:21, "Kenneth Knowles"  a écrit :
>
>> I'm working on finding a solution for launching the Nexmark suite with
>> each runner. This doesn't have to be done via Gradle, but we anyhow need
>> built artifacts that don't require user classpath intervention.
>>
>> It looks to me like the examples are also missing this - they have
>> separate configuration e.g. sparkRunnerPreCommit but that is overspecified
>> compared to a free-form launching of a main() program with a runner profile.
>>
>> On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  wrote:
>>
>>> Romain, are you talking about the profiles that exist as part of the
>>> archetype examples?
>>>
>>> If so, then those still exist and haven't been changed. If not, can you
>>> provide a link to the profile in a pom file to be clearer?
>>>
>>> On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Hi Scott,

 is it right that 2 doesn't handle the hierachy anymore and that it
 doesn't handle profiles for runners as it is currently with maven?


 Romain Manni-Bucau
 @rmannibucau  |  Blog
  | Old Blog
  | Github
  | LinkedIn
  | Book
 

 2018-04-06 18:32 GMT+02:00 Scott Wegner :

> I wanted to start a thread to summarize the current state of Gradle
> migration. We've made lots of good progress so far this week. Here's the
> status from what I can tell-- please add or correct anything I missed:
>
> * Release artifacts can be built and published for Snapshot and
> officlal releases [1]
> * Gradle-generated releases have been validated with the the Apache
> Beam archetype generation quickstart; still needs additional validation.
> * Generated release pom files have correct project metadata [2]
> * The python pre-commits are now working in Gradle [3]
> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
> learn the new system-- please add your own. This will eventually feed into
> official documentation on the website.
> * Łukasz Gajowy is working on migrating performance testing framework
> [5]
> * Daniel is working on updating documentation to refer to Gradle
> instead of maven
>
> If I missed anything, please add it to this thread.
>
> The general roadmap we're working towards is:
> (a) Publish release artifacts with Gradle (SNAPSHOT and signed
> releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to Gradle
> (d) Migrate perfkit suites to use Gradle
>
> For those of you that are hacking: thanks for your help so far!
> Progress is being roughly tracked on the Kanban [6]; please make sure the
> issues assigned to you are up-to-date. Many of the changes are staged on
> lukecwik's local branch [7]; we'll work on merging them back soon.
>
>
> [1] https://github.com/lukecwik/incubator-beam/pull/7
> [2] https://github.com/lukecwik/incubator-beam/pull/3
> [3] https://github.com/apache/beam/pull/5032
> [4]
> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
> [5] https://github.com/apache/beam/pull/5003
> [6]
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
> --
>
>
> Got feedback? http://go/swegner-feedback
>

 --


Got feedback? http://go/swegner-feedback

Re: Gradle Status [April 6]

[Romain Manni-Bucau]

Le 6 avr. 2018 20:09, "Lukasz Cwik"  a écrit :

Romain, are you talking about the profiles that exist as part of the
archetype examples?


Was more thinking to this kind of profiles
https://github.com/apache/beam/blob/master/sdks/java/nexmark/pom.xml (which
should hit all IO at some point to ensure their portability)

Idea is to be able to extract the deps for a runner from a particular pom
since it sometimes requires some dependencies work for conflicts.





If so, then those still exist and haven't been changed. If not, can you
provide a link to the profile in a pom file to be clearer?

On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau 
wrote:

> Hi Scott,
>
> is it right that 2 doesn't handle the hierachy anymore and that it doesn't
> handle profiles for runners as it is currently with maven?
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-04-06 18:32 GMT+02:00 Scott Wegner :
>
>> I wanted to start a thread to summarize the current state of Gradle
>> migration. We've made lots of good progress so far this week. Here's the
>> status from what I can tell-- please add or correct anything I missed:
>>
>> * Release artifacts can be built and published for Snapshot and officlal
>> releases [1]
>> * Gradle-generated releases have been validated with the the Apache Beam
>> archetype generation quickstart; still needs additional validation.
>> * Generated release pom files have correct project metadata [2]
>> * The python pre-commits are now working in Gradle [3]
>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>> learn the new system-- please add your own. This will eventually feed into
>> official documentation on the website.
>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>> * Daniel is working on updating documentation to refer to Gradle instead
>> of maven
>>
>> If I missed anything, please add it to this thread.
>>
>> The general roadmap we're working towards is:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> For those of you that are hacking: thanks for your help so far! Progress
>> is being roughly tracked on the Kanban [6]; please make sure the issues
>> assigned to you are up-to-date. Many of the changes are staged on
>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>
>>
>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>> [3] https://github.com/apache/beam/pull/5032
>> [4] https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDf
>> RDVkxzeDlbdVSQ/edit
>> [5] https://github.com/apache/beam/pull/5003
>> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
>> --
>>
>>
>> Got feedback? http://go/swegner-feedback
>>
>
>

Re: Gradle Status [April 6]

[Kenneth Knowles]

I'm working on finding a solution for launching the Nexmark suite with each
runner. This doesn't have to be done via Gradle, but we anyhow need built
artifacts that don't require user classpath intervention.

It looks to me like the examples are also missing this - they have separate
configuration e.g. sparkRunnerPreCommit but that is overspecified compared
to a free-form launching of a main() program with a runner profile.

On Fri, Apr 6, 2018 at 11:09 AM Lukasz Cwik  wrote:

> Romain, are you talking about the profiles that exist as part of the
> archetype examples?
>
> If so, then those still exist and haven't been changed. If not, can you
> provide a link to the profile in a pom file to be clearer?
>
> On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau 
> wrote:
>
>> Hi Scott,
>>
>> is it right that 2 doesn't handle the hierachy anymore and that it
>> doesn't handle profiles for runners as it is currently with maven?
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>> 2018-04-06 18:32 GMT+02:00 Scott Wegner :
>>
>>> I wanted to start a thread to summarize the current state of Gradle
>>> migration. We've made lots of good progress so far this week. Here's the
>>> status from what I can tell-- please add or correct anything I missed:
>>>
>>> * Release artifacts can be built and published for Snapshot and officlal
>>> releases [1]
>>> * Gradle-generated releases have been validated with the the Apache Beam
>>> archetype generation quickstart; still needs additional validation.
>>> * Generated release pom files have correct project metadata [2]
>>> * The python pre-commits are now working in Gradle [3]
>>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>>> learn the new system-- please add your own. This will eventually feed into
>>> official documentation on the website.
>>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>>> * Daniel is working on updating documentation to refer to Gradle instead
>>> of maven
>>>
>>> If I missed anything, please add it to this thread.
>>>
>>> The general roadmap we're working towards is:
>>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>>> (b) Postcommits migrated to Gradle
>>> (c) Migrate documentation from maven to Gradle
>>> (d) Migrate perfkit suites to use Gradle
>>>
>>> For those of you that are hacking: thanks for your help so far! Progress
>>> is being roughly tracked on the Kanban [6]; please make sure the issues
>>> assigned to you are up-to-date. Many of the changes are staged on
>>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>>
>>>
>>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>>> [3] https://github.com/apache/beam/pull/5032
>>> [4]
>>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
>>> [5] https://github.com/apache/beam/pull/5003
>>> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>>> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
>>> --
>>>
>>>
>>> Got feedback? http://go/swegner-feedback
>>>
>>
>>

Re: Gradle Status [April 6]

[Lukasz Cwik]

Romain, are you talking about the profiles that exist as part of the
archetype examples?

If so, then those still exist and haven't been changed. If not, can you
provide a link to the profile in a pom file to be clearer?

On Fri, Apr 6, 2018 at 12:40 PM Romain Manni-Bucau 
wrote:

> Hi Scott,
>
> is it right that 2 doesn't handle the hierachy anymore and that it doesn't
> handle profiles for runners as it is currently with maven?
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-04-06 18:32 GMT+02:00 Scott Wegner :
>
>> I wanted to start a thread to summarize the current state of Gradle
>> migration. We've made lots of good progress so far this week. Here's the
>> status from what I can tell-- please add or correct anything I missed:
>>
>> * Release artifacts can be built and published for Snapshot and officlal
>> releases [1]
>> * Gradle-generated releases have been validated with the the Apache Beam
>> archetype generation quickstart; still needs additional validation.
>> * Generated release pom files have correct project metadata [2]
>> * The python pre-commits are now working in Gradle [3]
>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>> learn the new system-- please add your own. This will eventually feed into
>> official documentation on the website.
>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>> * Daniel is working on updating documentation to refer to Gradle instead
>> of maven
>>
>> If I missed anything, please add it to this thread.
>>
>> The general roadmap we're working towards is:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> For those of you that are hacking: thanks for your help so far! Progress
>> is being roughly tracked on the Kanban [6]; please make sure the issues
>> assigned to you are up-to-date. Many of the changes are staged on
>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>
>>
>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>> [3] https://github.com/apache/beam/pull/5032
>> [4]
>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
>> [5] https://github.com/apache/beam/pull/5003
>> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
>> --
>>
>>
>> Got feedback? http://go/swegner-feedback
>>
>
>

Re: Gradle status

[Scott Wegner]

Thanks for organizing, Reuven. I too would like to see us move back to a
single build system to reduce complexity. Count me in for the fixit.

On Thu, Mar 22, 2018 at 11:27 PM Reuven Lax  wrote:

> I'll send an email tomorrow with a few proposed dates and set up a
> burndown list of tasks.
>
> Reuven
>
> On Thu, Mar 22, 2018 at 11:24 PM Romain Manni-Bucau 
> wrote:
>
>> Can work Reuven, where is the "todo" list? Thought we were done (= the
>> replacement was not blocking the dev) multiple times due to other threads
>> but reading this week mails it sounds like we are not yet here.
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>> 2018-03-22 22:51 GMT+01:00 Reuven Lax :
>>
>>> Let's back up for a second.
>>>
>>> Earlier in the thread we agreed to organize a community "fixit" day to
>>> try and migrate remaining Maven items to Gradle. I had thought that Romain
>>> had volunteered to run this, but reading back in the thread it appears that
>>> I misunderstood this. I would suggest that we organize this first, and make
>>> the concerted push to migrate the remaining items. After this is done, we
>>> can evaluate the state we're left in and hold a process vote if necessary.
>>>
>>> I can volunteer to help coordinate this fixit.
>>>
>>> Reuven
>>>
>>> On Thu, Mar 22, 2018 at 1:35 PM Dan Halperin 
>>> wrote:
>>>
 On Thu, Mar 22, 2018 at 11:19 AM, Chamikara Jayalath <
 chamik...@google.com> wrote:

> I don't think incremental progress is a bad thing as long as we are
> making progress towards the goal. Do we need better metrics (a weekly 
> email
> ?) about the progress towards moving everything to Gradle ? I agree with
> others who pointed out that there are many unresolved JIRAs and simply
> deleting Maven artifacts could break many things (for example, performance
> tests).
>

 The problem does not seem to be incremental progress on its face, or a
 lack of metrics.

 The problem is that there are two build systems with separate features
 and issues, doubling (or worse) Jenkins cycles, mental effort, maintenance
 burden, etc. It hurts the community and casual contributors.

 As Luke suggested,
 > A process vote can be happen if the in-between state is too painful
 to maintain.

 Given that the in-between state has lasted so long, and there is it may
 be time.

 Dan



>
> Thanks,
> Cham
>
>
> On Thu, Mar 22, 2018 at 10:56 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>>
>>
>> Le 22 mars 2018 18:49, "Dan Halperin"  a écrit :
>>
>> It seems that a few groups are talking past each other.
>>
>> * A sizable contingent is interested in a move to Gradle -- it shows
>> promise, but the work is incomplete.
>> * Another contingent noticing the large burden of maintaining
>> multiple build systems. FWICT, both test suites have been broken quite a
>> lot recently, mainly the gradle ones, which is a cost to the community.
>> This is creating a barrier to entry for new contributors – especially 
>> those
>> who don't work in the same room or do their primary development in a
>> different repository.
>>
>> I don't see this situation being resolved to anyone's satisfaction
>> until there's only one build system left. The onus is clearly on the 
>> Gradle
>> promoters to finish the work.
>>
>> Luke made a suggestion 2.5 months ago that we should have a process
>> vote if this situation is untenable. It seems like we're there.
>>
>>
>> Yes but beam voted to move to gradle so we should but we shouldnt
>> maintain 2 build systems for more than 2 months (weway overpassed that) 
>> and
>> therefore the vote should be cancelled or validated by an action.
>>
>> I understand you want gradle but you dont want to pay the cost to
>> move to gradle, it doesnt work for the project do please another option
>> (rollbacking gradle or removing maven, all other options are negative for
>> the project and a pain for committers and contributors whatever you 
>> think).
>>
>>
>>
>> Thanks,
>> Dan
>>
>> On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Ok so to be clear for any contributor (which is the goal of this
>>> thread): maven is still the main build system and no need to maintain
>>> gradle in PR then until beam switches.
>>>
>>> 

Re: Gradle status

[Reuven Lax]

I'll send an email tomorrow with a few proposed dates and set up a burndown
list of tasks.

Reuven

On Thu, Mar 22, 2018 at 11:24 PM Romain Manni-Bucau 
wrote:

> Can work Reuven, where is the "todo" list? Thought we were done (= the
> replacement was not blocking the dev) multiple times due to other threads
> but reading this week mails it sounds like we are not yet here.
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-03-22 22:51 GMT+01:00 Reuven Lax :
>
>> Let's back up for a second.
>>
>> Earlier in the thread we agreed to organize a community "fixit" day to
>> try and migrate remaining Maven items to Gradle. I had thought that Romain
>> had volunteered to run this, but reading back in the thread it appears that
>> I misunderstood this. I would suggest that we organize this first, and make
>> the concerted push to migrate the remaining items. After this is done, we
>> can evaluate the state we're left in and hold a process vote if necessary.
>>
>> I can volunteer to help coordinate this fixit.
>>
>> Reuven
>>
>> On Thu, Mar 22, 2018 at 1:35 PM Dan Halperin  wrote:
>>
>>> On Thu, Mar 22, 2018 at 11:19 AM, Chamikara Jayalath <
>>> chamik...@google.com> wrote:
>>>
 I don't think incremental progress is a bad thing as long as we are
 making progress towards the goal. Do we need better metrics (a weekly email
 ?) about the progress towards moving everything to Gradle ? I agree with
 others who pointed out that there are many unresolved JIRAs and simply
 deleting Maven artifacts could break many things (for example, performance
 tests).

>>>
>>> The problem does not seem to be incremental progress on its face, or a
>>> lack of metrics.
>>>
>>> The problem is that there are two build systems with separate features
>>> and issues, doubling (or worse) Jenkins cycles, mental effort, maintenance
>>> burden, etc. It hurts the community and casual contributors.
>>>
>>> As Luke suggested,
>>> > A process vote can be happen if the in-between state is too painful to
>>> maintain.
>>>
>>> Given that the in-between state has lasted so long, and there is it may
>>> be time.
>>>
>>> Dan
>>>
>>>
>>>

 Thanks,
 Cham


 On Thu, Mar 22, 2018 at 10:56 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

>
>
> Le 22 mars 2018 18:49, "Dan Halperin"  a écrit :
>
> It seems that a few groups are talking past each other.
>
> * A sizable contingent is interested in a move to Gradle -- it shows
> promise, but the work is incomplete.
> * Another contingent noticing the large burden of maintaining multiple
> build systems. FWICT, both test suites have been broken quite a lot
> recently, mainly the gradle ones, which is a cost to the community. This 
> is
> creating a barrier to entry for new contributors – especially those who
> don't work in the same room or do their primary development in a different
> repository.
>
> I don't see this situation being resolved to anyone's satisfaction
> until there's only one build system left. The onus is clearly on the 
> Gradle
> promoters to finish the work.
>
> Luke made a suggestion 2.5 months ago that we should have a process
> vote if this situation is untenable. It seems like we're there.
>
>
> Yes but beam voted to move to gradle so we should but we shouldnt
> maintain 2 build systems for more than 2 months (weway overpassed that) 
> and
> therefore the vote should be cancelled or validated by an action.
>
> I understand you want gradle but you dont want to pay the cost to move
> to gradle, it doesnt work for the project do please another option
> (rollbacking gradle or removing maven, all other options are negative for
> the project and a pain for committers and contributors whatever you 
> think).
>
>
>
> Thanks,
> Dan
>
> On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Ok so to be clear for any contributor (which is the goal of this
>> thread): maven is still the main build system and no need to maintain
>> gradle in PR then until beam switches.
>>
>> Im more than fine with that.
>>
>> Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :
>>
>>> I think the investment in gradle is worthwhile, and incrementally we
>>> will continue to make progress. From what I've seem, gradle is a good 
>>> fit
>>> for this project and a path to a faster, more reliable 

Re: Gradle status

[Romain Manni-Bucau]

Can work Reuven, where is the "todo" list? Thought we were done (= the
replacement was not blocking the dev) multiple times due to other threads
but reading this week mails it sounds like we are not yet here.


Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book


2018-03-22 22:51 GMT+01:00 Reuven Lax :

> Let's back up for a second.
>
> Earlier in the thread we agreed to organize a community "fixit" day to try
> and migrate remaining Maven items to Gradle. I had thought that Romain had
> volunteered to run this, but reading back in the thread it appears that I
> misunderstood this. I would suggest that we organize this first, and make
> the concerted push to migrate the remaining items. After this is done, we
> can evaluate the state we're left in and hold a process vote if necessary.
>
> I can volunteer to help coordinate this fixit.
>
> Reuven
>
> On Thu, Mar 22, 2018 at 1:35 PM Dan Halperin  wrote:
>
>> On Thu, Mar 22, 2018 at 11:19 AM, Chamikara Jayalath <
>> chamik...@google.com> wrote:
>>
>>> I don't think incremental progress is a bad thing as long as we are
>>> making progress towards the goal. Do we need better metrics (a weekly email
>>> ?) about the progress towards moving everything to Gradle ? I agree with
>>> others who pointed out that there are many unresolved JIRAs and simply
>>> deleting Maven artifacts could break many things (for example, performance
>>> tests).
>>>
>>
>> The problem does not seem to be incremental progress on its face, or a
>> lack of metrics.
>>
>> The problem is that there are two build systems with separate features
>> and issues, doubling (or worse) Jenkins cycles, mental effort, maintenance
>> burden, etc. It hurts the community and casual contributors.
>>
>> As Luke suggested,
>> > A process vote can be happen if the in-between state is too painful to
>> maintain.
>>
>> Given that the in-between state has lasted so long, and there is it may
>> be time.
>>
>> Dan
>>
>>
>>
>>>
>>> Thanks,
>>> Cham
>>>
>>>
>>> On Thu, Mar 22, 2018 at 10:56 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>


 Le 22 mars 2018 18:49, "Dan Halperin"  a écrit :

 It seems that a few groups are talking past each other.

 * A sizable contingent is interested in a move to Gradle -- it shows
 promise, but the work is incomplete.
 * Another contingent noticing the large burden of maintaining multiple
 build systems. FWICT, both test suites have been broken quite a lot
 recently, mainly the gradle ones, which is a cost to the community. This is
 creating a barrier to entry for new contributors – especially those who
 don't work in the same room or do their primary development in a different
 repository.

 I don't see this situation being resolved to anyone's satisfaction
 until there's only one build system left. The onus is clearly on the Gradle
 promoters to finish the work.

 Luke made a suggestion 2.5 months ago that we should have a process
 vote if this situation is untenable. It seems like we're there.


 Yes but beam voted to move to gradle so we should but we shouldnt
 maintain 2 build systems for more than 2 months (weway overpassed that) and
 therefore the vote should be cancelled or validated by an action.

 I understand you want gradle but you dont want to pay the cost to move
 to gradle, it doesnt work for the project do please another option
 (rollbacking gradle or removing maven, all other options are negative for
 the project and a pain for committers and contributors whatever you think).



 Thanks,
 Dan

 On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Ok so to be clear for any contributor (which is the goal of this
> thread): maven is still the main build system and no need to maintain
> gradle in PR then until beam switches.
>
> Im more than fine with that.
>
> Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :
>
>> I think the investment in gradle is worthwhile, and incrementally we
>> will continue to make progress. From what I've seem, gradle is a good fit
>> for this project and a path to a faster, more reliable build system.
>>
>> pull/4812  creates the
>> release artifacts, although it is not hooked up yet with authentication.
>>
>> I expect to help make incremental progress over the next month
>> converting some of the integration tests, but welcome incremental
>> improvements from 

Re: Gradle status

[Reuven Lax]

Let's back up for a second.

Earlier in the thread we agreed to organize a community "fixit" day to try
and migrate remaining Maven items to Gradle. I had thought that Romain had
volunteered to run this, but reading back in the thread it appears that I
misunderstood this. I would suggest that we organize this first, and make
the concerted push to migrate the remaining items. After this is done, we
can evaluate the state we're left in and hold a process vote if necessary.

I can volunteer to help coordinate this fixit.

Reuven

On Thu, Mar 22, 2018 at 1:35 PM Dan Halperin  wrote:

> On Thu, Mar 22, 2018 at 11:19 AM, Chamikara Jayalath  > wrote:
>
>> I don't think incremental progress is a bad thing as long as we are
>> making progress towards the goal. Do we need better metrics (a weekly email
>> ?) about the progress towards moving everything to Gradle ? I agree with
>> others who pointed out that there are many unresolved JIRAs and simply
>> deleting Maven artifacts could break many things (for example, performance
>> tests).
>>
>
> The problem does not seem to be incremental progress on its face, or a
> lack of metrics.
>
> The problem is that there are two build systems with separate features and
> issues, doubling (or worse) Jenkins cycles, mental effort, maintenance
> burden, etc. It hurts the community and casual contributors.
>
> As Luke suggested,
> > A process vote can be happen if the in-between state is too painful to
> maintain.
>
> Given that the in-between state has lasted so long, and there is it may be
> time.
>
> Dan
>
>
>
>>
>> Thanks,
>> Cham
>>
>>
>> On Thu, Mar 22, 2018 at 10:56 AM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>>
>>>
>>> Le 22 mars 2018 18:49, "Dan Halperin"  a écrit :
>>>
>>> It seems that a few groups are talking past each other.
>>>
>>> * A sizable contingent is interested in a move to Gradle -- it shows
>>> promise, but the work is incomplete.
>>> * Another contingent noticing the large burden of maintaining multiple
>>> build systems. FWICT, both test suites have been broken quite a lot
>>> recently, mainly the gradle ones, which is a cost to the community. This is
>>> creating a barrier to entry for new contributors – especially those who
>>> don't work in the same room or do their primary development in a different
>>> repository.
>>>
>>> I don't see this situation being resolved to anyone's satisfaction until
>>> there's only one build system left. The onus is clearly on the Gradle
>>> promoters to finish the work.
>>>
>>> Luke made a suggestion 2.5 months ago that we should have a process vote
>>> if this situation is untenable. It seems like we're there.
>>>
>>>
>>> Yes but beam voted to move to gradle so we should but we shouldnt
>>> maintain 2 build systems for more than 2 months (weway overpassed that) and
>>> therefore the vote should be cancelled or validated by an action.
>>>
>>> I understand you want gradle but you dont want to pay the cost to move
>>> to gradle, it doesnt work for the project do please another option
>>> (rollbacking gradle or removing maven, all other options are negative for
>>> the project and a pain for committers and contributors whatever you think).
>>>
>>>
>>>
>>> Thanks,
>>> Dan
>>>
>>> On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Ok so to be clear for any contributor (which is the goal of this
 thread): maven is still the main build system and no need to maintain
 gradle in PR then until beam switches.

 Im more than fine with that.

 Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :

> I think the investment in gradle is worthwhile, and incrementally we
> will continue to make progress. From what I've seem, gradle is a good fit
> for this project and a path to a faster, more reliable build system.
>
> pull/4812  creates the
> release artifacts, although it is not hooked up yet with authentication.
>
> I expect to help make incremental progress over the next month
> converting some of the integration tests, but welcome incremental
> improvements from others.
>
>
>
> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>>
>>
>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>>
>>> what do we do? "Gradle migration will happen incrementally."
>>>
>>> "last months prooved beam cant maintain 2 systems, easier with that
>>> state is then to drop gradle since it is a 0 investment compared to the
>>> opposite"
>>> Its unfortunate that you feel this way but many people do not share
>>> your opinion.
>>>
>>
>> And a lot do so when a project is 50-50 it is time to act.
>>
>> Incrementally kind of means never (makes 4 months and nothing really
>> 

Re: Gradle status

[Dan Halperin]

On Thu, Mar 22, 2018 at 11:19 AM, Chamikara Jayalath 
wrote:

> I don't think incremental progress is a bad thing as long as we are making
> progress towards the goal. Do we need better metrics (a weekly email ?)
> about the progress towards moving everything to Gradle ? I agree with
> others who pointed out that there are many unresolved JIRAs and simply
> deleting Maven artifacts could break many things (for example, performance
> tests).
>

The problem does not seem to be incremental progress on its face, or a lack
of metrics.

The problem is that there are two build systems with separate features and
issues, doubling (or worse) Jenkins cycles, mental effort, maintenance
burden, etc. It hurts the community and casual contributors.

As Luke suggested,
> A process vote can be happen if the in-between state is too painful to
maintain.

Given that the in-between state has lasted so long, and there is it may be
time.

Dan



>
> Thanks,
> Cham
>
>
> On Thu, Mar 22, 2018 at 10:56 AM Romain Manni-Bucau 
> wrote:
>
>>
>>
>> Le 22 mars 2018 18:49, "Dan Halperin"  a écrit :
>>
>> It seems that a few groups are talking past each other.
>>
>> * A sizable contingent is interested in a move to Gradle -- it shows
>> promise, but the work is incomplete.
>> * Another contingent noticing the large burden of maintaining multiple
>> build systems. FWICT, both test suites have been broken quite a lot
>> recently, mainly the gradle ones, which is a cost to the community. This is
>> creating a barrier to entry for new contributors – especially those who
>> don't work in the same room or do their primary development in a different
>> repository.
>>
>> I don't see this situation being resolved to anyone's satisfaction until
>> there's only one build system left. The onus is clearly on the Gradle
>> promoters to finish the work.
>>
>> Luke made a suggestion 2.5 months ago that we should have a process vote
>> if this situation is untenable. It seems like we're there.
>>
>>
>> Yes but beam voted to move to gradle so we should but we shouldnt
>> maintain 2 build systems for more than 2 months (weway overpassed that) and
>> therefore the vote should be cancelled or validated by an action.
>>
>> I understand you want gradle but you dont want to pay the cost to move to
>> gradle, it doesnt work for the project do please another option
>> (rollbacking gradle or removing maven, all other options are negative for
>> the project and a pain for committers and contributors whatever you think).
>>
>>
>>
>> Thanks,
>> Dan
>>
>> On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Ok so to be clear for any contributor (which is the goal of this
>>> thread): maven is still the main build system and no need to maintain
>>> gradle in PR then until beam switches.
>>>
>>> Im more than fine with that.
>>>
>>> Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :
>>>
 I think the investment in gradle is worthwhile, and incrementally we
 will continue to make progress. From what I've seem, gradle is a good fit
 for this project and a path to a faster, more reliable build system.

 pull/4812  creates the
 release artifacts, although it is not hooked up yet with authentication.

 I expect to help make incremental progress over the next month
 converting some of the integration tests, but welcome incremental
 improvements from others.



 On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

>
>
> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>
>> what do we do? "Gradle migration will happen incrementally."
>>
>> "last months prooved beam cant maintain 2 systems, easier with that
>> state is then to drop gradle since it is a 0 investment compared to the
>> opposite"
>> Its unfortunate that you feel this way but many people do not share
>> your opinion.
>>
>
> And a lot do so when a project is 50-50 it is time to act.
>
> Incrementally kind of means never (makes 4 months and nothing really
> changed in PRs and habits, gradle maintener(s) are still alone)
>
>
>>
>>
>> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> @Valentyn: concretely any user can PR and be part of that process so
>>> anyone can do it wrong (me first)
>>> @Luskasz, Hennking: fine but what do we do? last months prooved beam
>>> cant maintain 2 systems, easier with that state is then to drop gradle
>>> since it is a 0 investment compared to the opposite
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau  |  Blog
>>>  | Old Blog
>>>  | 

Re: Gradle status

[Chamikara Jayalath]

I don't think incremental progress is a bad thing as long as we are making
progress towards the goal. Do we need better metrics (a weekly email ?)
about the progress towards moving everything to Gradle ? I agree with
others who pointed out that there are many unresolved JIRAs and simply
deleting Maven artifacts could break many things (for example, performance
tests).

Thanks,
Cham

On Thu, Mar 22, 2018 at 10:56 AM Romain Manni-Bucau 
wrote:

>
>
> Le 22 mars 2018 18:49, "Dan Halperin"  a écrit :
>
> It seems that a few groups are talking past each other.
>
> * A sizable contingent is interested in a move to Gradle -- it shows
> promise, but the work is incomplete.
> * Another contingent noticing the large burden of maintaining multiple
> build systems. FWICT, both test suites have been broken quite a lot
> recently, mainly the gradle ones, which is a cost to the community. This is
> creating a barrier to entry for new contributors – especially those who
> don't work in the same room or do their primary development in a different
> repository.
>
> I don't see this situation being resolved to anyone's satisfaction until
> there's only one build system left. The onus is clearly on the Gradle
> promoters to finish the work.
>
> Luke made a suggestion 2.5 months ago that we should have a process vote
> if this situation is untenable. It seems like we're there.
>
>
> Yes but beam voted to move to gradle so we should but we shouldnt maintain
> 2 build systems for more than 2 months (weway overpassed that) and
> therefore the vote should be cancelled or validated by an action.
>
> I understand you want gradle but you dont want to pay the cost to move to
> gradle, it doesnt work for the project do please another option
> (rollbacking gradle or removing maven, all other options are negative for
> the project and a pain for committers and contributors whatever you think).
>
>
>
> Thanks,
> Dan
>
> On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Ok so to be clear for any contributor (which is the goal of this thread):
>> maven is still the main build system and no need to maintain gradle in PR
>> then until beam switches.
>>
>> Im more than fine with that.
>>
>> Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :
>>
>>> I think the investment in gradle is worthwhile, and incrementally we
>>> will continue to make progress. From what I've seem, gradle is a good fit
>>> for this project and a path to a faster, more reliable build system.
>>>
>>> pull/4812  creates the
>>> release artifacts, although it is not hooked up yet with authentication.
>>>
>>> I expect to help make incremental progress over the next month
>>> converting some of the integration tests, but welcome incremental
>>> improvements from others.
>>>
>>>
>>>
>>> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>


 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :

> what do we do? "Gradle migration will happen incrementally."
>
> "last months prooved beam cant maintain 2 systems, easier with that
> state is then to drop gradle since it is a 0 investment compared to the
> opposite"
> Its unfortunate that you feel this way but many people do not share
> your opinion.
>

 And a lot do so when a project is 50-50 it is time to act.

 Incrementally kind of means never (makes 4 months and nothing really
 changed in PRs and habits, gradle maintener(s) are still alone)


>
>
> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> @Valentyn: concretely any user can PR and be part of that process so
>> anyone can do it wrong (me first)
>> @Luskasz, Hennking: fine but what do we do? last months prooved beam
>> cant maintain 2 systems, easier with that state is then to drop gradle
>> since it is a 0 investment compared to the opposite
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>>
>>> Romain, from the previous discussions several people agreed that
>>> running a fixit that migrated Maven to Gradle over a 1 or 2 day period 
>>> was
>>> worthwhile but there was nobody in the community with the time 
>>> commitment
>>> to organize and run it so the status quo plan remained where the Gradle
>>> migration will happen incrementally.
>>>
>>>
>>> On Thu, Mar 22, 2018 

Re: Gradle status

[Romain Manni-Bucau]

Le 22 mars 2018 18:49, "Dan Halperin"  a écrit :

It seems that a few groups are talking past each other.

* A sizable contingent is interested in a move to Gradle -- it shows
promise, but the work is incomplete.
* Another contingent noticing the large burden of maintaining multiple
build systems. FWICT, both test suites have been broken quite a lot
recently, mainly the gradle ones, which is a cost to the community. This is
creating a barrier to entry for new contributors – especially those who
don't work in the same room or do their primary development in a different
repository.

I don't see this situation being resolved to anyone's satisfaction until
there's only one build system left. The onus is clearly on the Gradle
promoters to finish the work.

Luke made a suggestion 2.5 months ago that we should have a process vote if
this situation is untenable. It seems like we're there.


Yes but beam voted to move to gradle so we should but we shouldnt maintain
2 build systems for more than 2 months (weway overpassed that) and
therefore the vote should be cancelled or validated by an action.

I understand you want gradle but you dont want to pay the cost to move to
gradle, it doesnt work for the project do please another option
(rollbacking gradle or removing maven, all other options are negative for
the project and a pain for committers and contributors whatever you think).



Thanks,
Dan

On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau 
wrote:

> Ok so to be clear for any contributor (which is the goal of this thread):
> maven is still the main build system and no need to maintain gradle in PR
> then until beam switches.
>
> Im more than fine with that.
>
> Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :
>
>> I think the investment in gradle is worthwhile, and incrementally we will
>> continue to make progress. From what I've seem, gradle is a good fit for
>> this project and a path to a faster, more reliable build system.
>>
>> pull/4812  creates the release
>> artifacts, although it is not hooked up yet with authentication.
>>
>> I expect to help make incremental progress over the next month converting
>> some of the integration tests, but welcome incremental improvements from
>> others.
>>
>>
>>
>> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau 
>> wrote:
>>
>>>
>>>
>>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>>>
 what do we do? "Gradle migration will happen incrementally."

 "last months prooved beam cant maintain 2 systems, easier with that
 state is then to drop gradle since it is a 0 investment compared to the
 opposite"
 Its unfortunate that you feel this way but many people do not share
 your opinion.

>>>
>>> And a lot do so when a project is 50-50 it is time to act.
>>>
>>> Incrementally kind of means never (makes 4 months and nothing really
>>> changed in PRs and habits, gradle maintener(s) are still alone)
>>>
>>>


 On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> @Valentyn: concretely any user can PR and be part of that process so
> anyone can do it wrong (me first)
> @Luskasz, Hennking: fine but what do we do? last months prooved beam
> cant maintain 2 systems, easier with that state is then to drop gradle
> since it is a 0 investment compared to the opposite
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>
>> Romain, from the previous discussions several people agreed that
>> running a fixit that migrated Maven to Gradle over a 1 or 2 day period 
>> was
>> worthwhile but there was nobody in the community with the time commitment
>> to organize and run it so the status quo plan remained where the Gradle
>> migration will happen incrementally.
>>
>>
>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde 
>> wrote:
>>
>>> My understanding was the same as Ismaël's. I don't think breaking
>>> the build with a large known gaps (but not fully known cost) is 
>>> practical.
>>> Also, most items in the jira are not even assigned yet.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Not really Ismaël, this thread was about to do it at once and have
 1 day to fix it all.

 As mentionned at the very beginning nobody maintains the 2 system
 so 

Re: Gradle status

[Dan Halperin]

It seems that a few groups are talking past each other.

* A sizable contingent is interested in a move to Gradle -- it shows
promise, but the work is incomplete.
* Another contingent noticing the large burden of maintaining multiple
build systems. FWICT, both test suites have been broken quite a lot
recently, mainly the gradle ones, which is a cost to the community. This is
creating a barrier to entry for new contributors – especially those who
don't work in the same room or do their primary development in a different
repository.

I don't see this situation being resolved to anyone's satisfaction until
there's only one build system left. The onus is clearly on the Gradle
promoters to finish the work.

Luke made a suggestion 2.5 months ago that we should have a process vote if
this situation is untenable. It seems like we're there.

Thanks,
Dan

On Thu, Mar 22, 2018 at 10:30 AM, Romain Manni-Bucau 
wrote:

> Ok so to be clear for any contributor (which is the goal of this thread):
> maven is still the main build system and no need to maintain gradle in PR
> then until beam switches.
>
> Im more than fine with that.
>
> Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :
>
>> I think the investment in gradle is worthwhile, and incrementally we will
>> continue to make progress. From what I've seem, gradle is a good fit for
>> this project and a path to a faster, more reliable build system.
>>
>> pull/4812  creates the release
>> artifacts, although it is not hooked up yet with authentication.
>>
>> I expect to help make incremental progress over the next month converting
>> some of the integration tests, but welcome incremental improvements from
>> others.
>>
>>
>>
>> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau 
>> wrote:
>>
>>>
>>>
>>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>>>
 what do we do? "Gradle migration will happen incrementally."

 "last months prooved beam cant maintain 2 systems, easier with that
 state is then to drop gradle since it is a 0 investment compared to the
 opposite"
 Its unfortunate that you feel this way but many people do not share
 your opinion.

>>>
>>> And a lot do so when a project is 50-50 it is time to act.
>>>
>>> Incrementally kind of means never (makes 4 months and nothing really
>>> changed in PRs and habits, gradle maintener(s) are still alone)
>>>
>>>


 On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> @Valentyn: concretely any user can PR and be part of that process so
> anyone can do it wrong (me first)
> @Luskasz, Hennking: fine but what do we do? last months prooved beam
> cant maintain 2 systems, easier with that state is then to drop gradle
> since it is a 0 investment compared to the opposite
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>
>> Romain, from the previous discussions several people agreed that
>> running a fixit that migrated Maven to Gradle over a 1 or 2 day period 
>> was
>> worthwhile but there was nobody in the community with the time commitment
>> to organize and run it so the status quo plan remained where the Gradle
>> migration will happen incrementally.
>>
>>
>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde 
>> wrote:
>>
>>> My understanding was the same as Ismaël's. I don't think breaking
>>> the build with a large known gaps (but not fully known cost) is 
>>> practical.
>>> Also, most items in the jira are not even assigned yet.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Not really Ismaël, this thread was about to do it at once and have
 1 day to fix it all.

 As mentionned at the very beginning nobody maintains the 2 system
 so it must stop after months so either we drop maven or gradle *at 
 once*
 or we keep a state where each dev does what he wants and the build
 system just doesn't work.

 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :

> I don't think that removing all maven descriptors was the expected
> path, no ? Or even a good idea at this moment.
>
> I understood that what we were going to do was to replace
> incrementally the CI until we cover the whole maven functionality

Re: Gradle status

[Lukasz Cwik]

Romain, that is incorrect. Contributors are to maintain both systems until
Gradle replaces that functionality in Maven and then it only needs to be
maintained in Gradle.


On Thu, Mar 22, 2018 at 10:30 AM Romain Manni-Bucau 
wrote:

> Ok so to be clear for any contributor (which is the goal of this thread):
> maven is still the main build system and no need to maintain gradle in PR
> then until beam switches.
>
> Im more than fine with that.
>
> Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :
>
>> I think the investment in gradle is worthwhile, and incrementally we will
>> continue to make progress. From what I've seem, gradle is a good fit for
>> this project and a path to a faster, more reliable build system.
>>
>> pull/4812  creates the release
>> artifacts, although it is not hooked up yet with authentication.
>>
>> I expect to help make incremental progress over the next month converting
>> some of the integration tests, but welcome incremental improvements from
>> others.
>>
>>
>>
>> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau 
>> wrote:
>>
>>>
>>>
>>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>>>
 what do we do? "Gradle migration will happen incrementally."

 "last months prooved beam cant maintain 2 systems, easier with that
 state is then to drop gradle since it is a 0 investment compared to the
 opposite"
 Its unfortunate that you feel this way but many people do not share
 your opinion.

>>>
>>> And a lot do so when a project is 50-50 it is time to act.
>>>
>>> Incrementally kind of means never (makes 4 months and nothing really
>>> changed in PRs and habits, gradle maintener(s) are still alone)
>>>
>>>


 On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> @Valentyn: concretely any user can PR and be part of that process so
> anyone can do it wrong (me first)
> @Luskasz, Hennking: fine but what do we do? last months prooved beam
> cant maintain 2 systems, easier with that state is then to drop gradle
> since it is a 0 investment compared to the opposite
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>
>> Romain, from the previous discussions several people agreed that
>> running a fixit that migrated Maven to Gradle over a 1 or 2 day period 
>> was
>> worthwhile but there was nobody in the community with the time commitment
>> to organize and run it so the status quo plan remained where the Gradle
>> migration will happen incrementally.
>>
>>
>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde 
>> wrote:
>>
>>> My understanding was the same as Ismaël's. I don't think breaking
>>> the build with a large known gaps (but not fully known cost) is 
>>> practical.
>>> Also, most items in the jira are not even assigned yet.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Not really Ismaël, this thread was about to do it at once and have
 1 day to fix it all.

 As mentionned at the very beginning nobody maintains the 2 system
 so it must stop after months so either we drop maven or gradle *at 
 once*
 or we keep a state where each dev does what he wants and the build
 system just doesn't work.

 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :

> I don't think that removing all maven descriptors was the expected
> path, no ? Or even a good idea at this moment.
>
> I understood that what we were going to do was to replace
> incrementally the CI until we cover the whole maven functionality
> and
> then remove it, from looking at the JIRA ticket
> https://issues.apache.org/jira/browse/BEAM-3249 we are still far
> from
> covering the complete maven functionality in particular for the
> release part that could be the biggest pain point.
>
>
> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>  wrote:
> > hey guys,
> >
> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or
> on monday?
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >

Re: Gradle status

[Romain Manni-Bucau]

Ok so to be clear for any contributor (which is the goal of this thread):
maven is still the main build system and no need to maintain gradle in PR
then until beam switches.

Im more than fine with that.

Le 22 mars 2018 18:22, "Alan Myrvold"  a écrit :

> I think the investment in gradle is worthwhile, and incrementally we will
> continue to make progress. From what I've seem, gradle is a good fit for
> this project and a path to a faster, more reliable build system.
>
> pull/4812  creates the release
> artifacts, although it is not hooked up yet with authentication.
>
> I expect to help make incremental progress over the next month converting
> some of the integration tests, but welcome incremental improvements from
> others.
>
>
>
> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau 
> wrote:
>
>>
>>
>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>>
>>> what do we do? "Gradle migration will happen incrementally."
>>>
>>> "last months prooved beam cant maintain 2 systems, easier with that
>>> state is then to drop gradle since it is a 0 investment compared to the
>>> opposite"
>>> Its unfortunate that you feel this way but many people do not share your
>>> opinion.
>>>
>>
>> And a lot do so when a project is 50-50 it is time to act.
>>
>> Incrementally kind of means never (makes 4 months and nothing really
>> changed in PRs and habits, gradle maintener(s) are still alone)
>>
>>
>>>
>>>
>>> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 @Valentyn: concretely any user can PR and be part of that process so
 anyone can do it wrong (me first)
 @Luskasz, Hennking: fine but what do we do? last months prooved beam
 cant maintain 2 systems, easier with that state is then to drop gradle
 since it is a 0 investment compared to the opposite


 Romain Manni-Bucau
 @rmannibucau  |  Blog
  | Old Blog
  | Github
  | LinkedIn
  | Book
 

 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :

> Romain, from the previous discussions several people agreed that
> running a fixit that migrated Maven to Gradle over a 1 or 2 day period was
> worthwhile but there was nobody in the community with the time commitment
> to organize and run it so the status quo plan remained where the Gradle
> migration will happen incrementally.
>
>
> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde 
> wrote:
>
>> My understanding was the same as Ismaël's. I don't think breaking the
>> build with a large known gaps (but not fully known cost) is practical.
>> Also, most items in the jira are not even assigned yet.
>>
>>
>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Not really Ismaël, this thread was about to do it at once and have 1
>>> day to fix it all.
>>>
>>> As mentionned at the very beginning nobody maintains the 2 system so
>>> it must stop after months so either we drop maven or gradle *at once*
>>> or we keep a state where each dev does what he wants and the build
>>> system just doesn't work.
>>>
>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :
>>>
 I don't think that removing all maven descriptors was the expected
 path, no ? Or even a good idea at this moment.

 I understood that what we were going to do was to replace
 incrementally the CI until we cover the whole maven functionality
 and
 then remove it, from looking at the JIRA ticket
 https://issues.apache.org/jira/browse/BEAM-3249 we are still far
 from
 covering the complete maven functionality in particular for the
 release part that could be the biggest pain point.


 On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
  wrote:
 > hey guys,
 >
 > 2.4 is out, do we plan to drop all maven descriptors tomorrow or
 on monday?
 >
 >
 > Romain Manni-Bucau
 > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
 >
 > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
 >>
 >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik 
 wrote:
 >>>
 >>> Based upon your description it seems as though you would rather
 have a
 >>> way to run existing postcommits without it impacting
 dashboard/health
 >>> 

Re: Gradle status

[Alan Myrvold]

I think the investment in gradle is worthwhile, and incrementally we will
continue to make progress. From what I've seem, gradle is a good fit for
this project and a path to a faster, more reliable build system.

pull/4812  creates the release
artifacts, although it is not hooked up yet with authentication.

I expect to help make incremental progress over the next month converting
some of the integration tests, but welcome incremental improvements from
others.



On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau 
wrote:

>
>
> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>
>> what do we do? "Gradle migration will happen incrementally."
>>
>> "last months prooved beam cant maintain 2 systems, easier with that
>> state is then to drop gradle since it is a 0 investment compared to the
>> opposite"
>> Its unfortunate that you feel this way but many people do not share your
>> opinion.
>>
>
> And a lot do so when a project is 50-50 it is time to act.
>
> Incrementally kind of means never (makes 4 months and nothing really
> changed in PRs and habits, gradle maintener(s) are still alone)
>
>
>>
>>
>> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau 
>> wrote:
>>
>>> @Valentyn: concretely any user can PR and be part of that process so
>>> anyone can do it wrong (me first)
>>> @Luskasz, Hennking: fine but what do we do? last months prooved beam
>>> cant maintain 2 systems, easier with that state is then to drop gradle
>>> since it is a 0 investment compared to the opposite
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau  |  Blog
>>>  | Old Blog
>>>  | Github
>>>  | LinkedIn
>>>  | Book
>>> 
>>>
>>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>>>
 Romain, from the previous discussions several people agreed that
 running a fixit that migrated Maven to Gradle over a 1 or 2 day period was
 worthwhile but there was nobody in the community with the time commitment
 to organize and run it so the status quo plan remained where the Gradle
 migration will happen incrementally.


 On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde 
 wrote:

> My understanding was the same as Ismaël's. I don't think breaking the
> build with a large known gaps (but not fully known cost) is practical.
> Also, most items in the jira are not even assigned yet.
>
>
> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Not really Ismaël, this thread was about to do it at once and have 1
>> day to fix it all.
>>
>> As mentionned at the very beginning nobody maintains the 2 system so
>> it must stop after months so either we drop maven or gradle *at once*
>> or we keep a state where each dev does what he wants and the build
>> system just doesn't work.
>>
>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :
>>
>>> I don't think that removing all maven descriptors was the expected
>>> path, no ? Or even a good idea at this moment.
>>>
>>> I understood that what we were going to do was to replace
>>> incrementally the CI until we cover the whole maven functionality and
>>> then remove it, from looking at the JIRA ticket
>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far
>>> from
>>> covering the complete maven functionality in particular for the
>>> release part that could be the biggest pain point.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>>  wrote:
>>> > hey guys,
>>> >
>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or
>>> on monday?
>>> >
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >
>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
>>> >>
>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik 
>>> wrote:
>>> >>>
>>> >>> Based upon your description it seems as though you would rather
>>> have a
>>> >>> way to run existing postcommits without it impacting
>>> dashboard/health
>>> >>> stats/notifications/ (We have just run the PostCommits on
>>> PRs for
>>> >>> additional validation (like upgrading the Dataflow container
>>> image)).
>>> >>
>>> >>
>>> >> Yes, that is exactly what I have described.
>>> >>
>>> >>> I don't think that keeping the current Java PreCommit as a proxy
>>> for the
>>> >>> the Java PostCommit is the right way to go but I also don't have
>>> 

Re: Gradle status

[Romain Manni-Bucau]

2018-03-22 17:45 GMT+01:00 Lukasz Cwik :

> what do we do? "Gradle migration will happen incrementally."
>
> "last months prooved beam cant maintain 2 systems, easier with that state
> is then to drop gradle since it is a 0 investment compared to the opposite"
> Its unfortunate that you feel this way but many people do not share your
> opinion.
>

And a lot do so when a project is 50-50 it is time to act.

Incrementally kind of means never (makes 4 months and nothing really
changed in PRs and habits, gradle maintener(s) are still alone)


>
>
> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau 
> wrote:
>
>> @Valentyn: concretely any user can PR and be part of that process so
>> anyone can do it wrong (me first)
>> @Luskasz, Hennking: fine but what do we do? last months prooved beam cant
>> maintain 2 systems, easier with that state is then to drop gradle since it
>> is a 0 investment compared to the opposite
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>>
>>> Romain, from the previous discussions several people agreed that running
>>> a fixit that migrated Maven to Gradle over a 1 or 2 day period was
>>> worthwhile but there was nobody in the community with the time commitment
>>> to organize and run it so the status quo plan remained where the Gradle
>>> migration will happen incrementally.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde 
>>> wrote:
>>>
 My understanding was the same as Ismaël's. I don't think breaking the
 build with a large known gaps (but not fully known cost) is practical.
 Also, most items in the jira are not even assigned yet.


 On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Not really Ismaël, this thread was about to do it at once and have 1
> day to fix it all.
>
> As mentionned at the very beginning nobody maintains the 2 system so
> it must stop after months so either we drop maven or gradle *at once*
> or we keep a state where each dev does what he wants and the build
> system just doesn't work.
>
> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :
>
>> I don't think that removing all maven descriptors was the expected
>> path, no ? Or even a good idea at this moment.
>>
>> I understood that what we were going to do was to replace
>> incrementally the CI until we cover the whole maven functionality and
>> then remove it, from looking at the JIRA ticket
>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
>> covering the complete maven functionality in particular for the
>> release part that could be the biggest pain point.
>>
>>
>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>  wrote:
>> > hey guys,
>> >
>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
>> monday?
>> >
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >
>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
>> >>
>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik 
>> wrote:
>> >>>
>> >>> Based upon your description it seems as though you would rather
>> have a
>> >>> way to run existing postcommits without it impacting
>> dashboard/health
>> >>> stats/notifications/ (We have just run the PostCommits on PRs
>> for
>> >>> additional validation (like upgrading the Dataflow container
>> image)).
>> >>
>> >>
>> >> Yes, that is exactly what I have described.
>> >>
>> >>> I don't think that keeping the current Java PreCommit as a proxy
>> for the
>> >>> the Java PostCommit is the right way to go but I also don't have
>> the time to
>> >>> implement what your actually asking for.
>> >>
>> >>
>> >> Mostly I thought this might be very easy based on the fact that
>> they are
>> >> nearly identical. If not, oh well.
>> >>
>> >> Kenn
>> >>
>> >>
>> >>> It seems more likely that migrating the PostCommit to Gradle will
>> be less
>> >>> work then adding the functionality but your argument where the
>> PreCommit is
>> >>> a proxy for the Java PostCommit also applies to the
>> ValidatesRunner
>> >>> PostCommits and so forth requiring even more migration to happen
>> before you
>> >>> don't have to worry about maintaining Maven/breaking post commits.
>> >>>

Re: Gradle status

[Lukasz Cwik]

what do we do? "Gradle migration will happen incrementally."

"last months prooved beam cant maintain 2 systems, easier with that state
is then to drop gradle since it is a 0 investment compared to the opposite"
Its unfortunate that you feel this way but many people do not share your
opinion.


On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau 
wrote:

> @Valentyn: concretely any user can PR and be part of that process so
> anyone can do it wrong (me first)
> @Luskasz, Hennking: fine but what do we do? last months prooved beam cant
> maintain 2 systems, easier with that state is then to drop gradle since it
> is a 0 investment compared to the opposite
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>
>> Romain, from the previous discussions several people agreed that running
>> a fixit that migrated Maven to Gradle over a 1 or 2 day period was
>> worthwhile but there was nobody in the community with the time commitment
>> to organize and run it so the status quo plan remained where the Gradle
>> migration will happen incrementally.
>>
>>
>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde  wrote:
>>
>>> My understanding was the same as Ismaël's. I don't think breaking the
>>> build with a large known gaps (but not fully known cost) is practical.
>>> Also, most items in the jira are not even assigned yet.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Not really Ismaël, this thread was about to do it at once and have 1
 day to fix it all.

 As mentionned at the very beginning nobody maintains the 2 system so it
 must stop after months so either we drop maven or gradle *at once*
 or we keep a state where each dev does what he wants and the build
 system just doesn't work.

 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :

> I don't think that removing all maven descriptors was the expected
> path, no ? Or even a good idea at this moment.
>
> I understood that what we were going to do was to replace
> incrementally the CI until we cover the whole maven functionality and
> then remove it, from looking at the JIRA ticket
> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
> covering the complete maven functionality in particular for the
> release part that could be the biggest pain point.
>
>
> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>  wrote:
> > hey guys,
> >
> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
> monday?
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
> >>
> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik 
> wrote:
> >>>
> >>> Based upon your description it seems as though you would rather
> have a
> >>> way to run existing postcommits without it impacting
> dashboard/health
> >>> stats/notifications/ (We have just run the PostCommits on PRs
> for
> >>> additional validation (like upgrading the Dataflow container
> image)).
> >>
> >>
> >> Yes, that is exactly what I have described.
> >>
> >>> I don't think that keeping the current Java PreCommit as a proxy
> for the
> >>> the Java PostCommit is the right way to go but I also don't have
> the time to
> >>> implement what your actually asking for.
> >>
> >>
> >> Mostly I thought this might be very easy based on the fact that
> they are
> >> nearly identical. If not, oh well.
> >>
> >> Kenn
> >>
> >>
> >>> It seems more likely that migrating the PostCommit to Gradle will
> be less
> >>> work then adding the functionality but your argument where the
> PreCommit is
> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
> >>> PostCommits and so forth requiring even more migration to happen
> before you
> >>> don't have to worry about maintaining Maven/breaking post commits.
> >>>
> >>> I'm fine with leaving both the Java/Gradle PreCommits running for
> now and
> >>> hopefully as more of the PostCommits are migrated off we will be
> able to
> >>> remove it.
> >>>
> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles 
> wrote:
> 
>  Separate history (for easy dashboarding, health stats, etc) and
>  

Re: Gradle status

[Romain Manni-Bucau]

@Valentyn: concretely any user can PR and be part of that process so anyone
can do it wrong (me first)
@Luskasz, Hennking: fine but what do we do? last months prooved beam cant
maintain 2 systems, easier with that state is then to drop gradle since it
is a 0 investment compared to the opposite


Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book


2018-03-22 17:24 GMT+01:00 Lukasz Cwik :

> Romain, from the previous discussions several people agreed that running a
> fixit that migrated Maven to Gradle over a 1 or 2 day period was worthwhile
> but there was nobody in the community with the time commitment to organize
> and run it so the status quo plan remained where the Gradle migration will
> happen incrementally.
>
>
> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde  wrote:
>
>> My understanding was the same as Ismaël's. I don't think breaking the
>> build with a large known gaps (but not fully known cost) is practical.
>> Also, most items in the jira are not even assigned yet.
>>
>>
>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau 
>> wrote:
>>
>>> Not really Ismaël, this thread was about to do it at once and have 1 day
>>> to fix it all.
>>>
>>> As mentionned at the very beginning nobody maintains the 2 system so it
>>> must stop after months so either we drop maven or gradle *at once*
>>> or we keep a state where each dev does what he wants and the build
>>> system just doesn't work.
>>>
>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :
>>>
 I don't think that removing all maven descriptors was the expected
 path, no ? Or even a good idea at this moment.

 I understood that what we were going to do was to replace
 incrementally the CI until we cover the whole maven functionality and
 then remove it, from looking at the JIRA ticket
 https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
 covering the complete maven functionality in particular for the
 release part that could be the biggest pain point.


 On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
  wrote:
 > hey guys,
 >
 > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
 monday?
 >
 >
 > Romain Manni-Bucau
 > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
 >
 > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
 >>
 >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik 
 wrote:
 >>>
 >>> Based upon your description it seems as though you would rather
 have a
 >>> way to run existing postcommits without it impacting
 dashboard/health
 >>> stats/notifications/ (We have just run the PostCommits on PRs
 for
 >>> additional validation (like upgrading the Dataflow container
 image)).
 >>
 >>
 >> Yes, that is exactly what I have described.
 >>
 >>> I don't think that keeping the current Java PreCommit as a proxy
 for the
 >>> the Java PostCommit is the right way to go but I also don't have
 the time to
 >>> implement what your actually asking for.
 >>
 >>
 >> Mostly I thought this might be very easy based on the fact that they
 are
 >> nearly identical. If not, oh well.
 >>
 >> Kenn
 >>
 >>
 >>> It seems more likely that migrating the PostCommit to Gradle will
 be less
 >>> work then adding the functionality but your argument where the
 PreCommit is
 >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
 >>> PostCommits and so forth requiring even more migration to happen
 before you
 >>> don't have to worry about maintaining Maven/breaking post commits.
 >>>
 >>> I'm fine with leaving both the Java/Gradle PreCommits running for
 now and
 >>> hopefully as more of the PostCommits are migrated off we will be
 able to
 >>> remove it.
 >>>
 >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles 
 wrote:
 
  Separate history (for easy dashboarding, health stats, etc) and
  notification (email to dev@ for postcommits, nothing for
 precommits) for pre
  & post commit targets.
 
  A post commit failure is always a problem to be triaged at high
  priority, while a precommit failure is just a natural occurrence.
 
  On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik 
 wrote:
 >
 > Ken, I'm probably not seeing something but how does using the
 PreCommit
 > as a proxy improve upon just 

Re: Gradle status

[Lukasz Cwik]

Romain, from the previous discussions several people agreed that running a
fixit that migrated Maven to Gradle over a 1 or 2 day period was worthwhile
but there was nobody in the community with the time commitment to organize
and run it so the status quo plan remained where the Gradle migration will
happen incrementally.


On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde  wrote:

> My understanding was the same as Ismaël's. I don't think breaking the
> build with a large known gaps (but not fully known cost) is practical.
> Also, most items in the jira are not even assigned yet.
>
>
> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau 
> wrote:
>
>> Not really Ismaël, this thread was about to do it at once and have 1 day
>> to fix it all.
>>
>> As mentionned at the very beginning nobody maintains the 2 system so it
>> must stop after months so either we drop maven or gradle *at once*
>> or we keep a state where each dev does what he wants and the build system
>> just doesn't work.
>>
>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :
>>
>>> I don't think that removing all maven descriptors was the expected
>>> path, no ? Or even a good idea at this moment.
>>>
>>> I understood that what we were going to do was to replace
>>> incrementally the CI until we cover the whole maven functionality and
>>> then remove it, from looking at the JIRA ticket
>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
>>> covering the complete maven functionality in particular for the
>>> release part that could be the biggest pain point.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>>  wrote:
>>> > hey guys,
>>> >
>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
>>> monday?
>>> >
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >
>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
>>> >>
>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik  wrote:
>>> >>>
>>> >>> Based upon your description it seems as though you would rather have
>>> a
>>> >>> way to run existing postcommits without it impacting dashboard/health
>>> >>> stats/notifications/ (We have just run the PostCommits on PRs for
>>> >>> additional validation (like upgrading the Dataflow container image)).
>>> >>
>>> >>
>>> >> Yes, that is exactly what I have described.
>>> >>
>>> >>> I don't think that keeping the current Java PreCommit as a proxy for
>>> the
>>> >>> the Java PostCommit is the right way to go but I also don't have the
>>> time to
>>> >>> implement what your actually asking for.
>>> >>
>>> >>
>>> >> Mostly I thought this might be very easy based on the fact that they
>>> are
>>> >> nearly identical. If not, oh well.
>>> >>
>>> >> Kenn
>>> >>
>>> >>
>>> >>> It seems more likely that migrating the PostCommit to Gradle will be
>>> less
>>> >>> work then adding the functionality but your argument where the
>>> PreCommit is
>>> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
>>> >>> PostCommits and so forth requiring even more migration to happen
>>> before you
>>> >>> don't have to worry about maintaining Maven/breaking post commits.
>>> >>>
>>> >>> I'm fine with leaving both the Java/Gradle PreCommits running for
>>> now and
>>> >>> hopefully as more of the PostCommits are migrated off we will be
>>> able to
>>> >>> remove it.
>>> >>>
>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles 
>>> wrote:
>>> 
>>>  Separate history (for easy dashboarding, health stats, etc) and
>>>  notification (email to dev@ for postcommits, nothing for
>>> precommits) for pre
>>>  & post commit targets.
>>> 
>>>  A post commit failure is always a problem to be triaged at high
>>>  priority, while a precommit failure is just a natural occurrence.
>>> 
>>>  On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik 
>>> wrote:
>>> >
>>> > Ken, I'm probably not seeing something but how does using the
>>> PreCommit
>>> > as a proxy improve upon just running the post commit via the
>>> phrase it
>>> > already supports ('Run Java PostCommit')?
>>> >
>>> > On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles 
>>> > wrote:
>>> >>
>>> >> Indeed, we've already had the discussion a couple of times and I
>>> think
>>> >> the criteria are clearly met. Incremental progress is a good
>>> thing and we
>>> >> shouldn't block it.
>>> >>
>>> >> OTOH I see where Romain is coming from and I have a good example
>>> that
>>> >> supports a slightly different action. Consider
>>> >> https://github.com/apache/beam/pull/4740 which fixes some errors
>>> in how we
>>> >> use dependency mechanisms.
>>> >>
>>> >> This PR is green except that I need to fix some Maven pom slightly
>>> >> more. That is throwaway 

Re: Gradle status

[Henning Rohde]

My understanding was the same as Ismaël's. I don't think breaking the build
with a large known gaps (but not fully known cost) is practical. Also, most
items in the jira are not even assigned yet.


On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau 
wrote:

> Not really Ismaël, this thread was about to do it at once and have 1 day
> to fix it all.
>
> As mentionned at the very beginning nobody maintains the 2 system so it
> must stop after months so either we drop maven or gradle *at once*
> or we keep a state where each dev does what he wants and the build system
> just doesn't work.
>
> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :
>
>> I don't think that removing all maven descriptors was the expected
>> path, no ? Or even a good idea at this moment.
>>
>> I understood that what we were going to do was to replace
>> incrementally the CI until we cover the whole maven functionality and
>> then remove it, from looking at the JIRA ticket
>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
>> covering the complete maven functionality in particular for the
>> release part that could be the biggest pain point.
>>
>>
>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>  wrote:
>> > hey guys,
>> >
>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
>> monday?
>> >
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>> >
>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
>> >>
>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik  wrote:
>> >>>
>> >>> Based upon your description it seems as though you would rather have a
>> >>> way to run existing postcommits without it impacting dashboard/health
>> >>> stats/notifications/ (We have just run the PostCommits on PRs for
>> >>> additional validation (like upgrading the Dataflow container image)).
>> >>
>> >>
>> >> Yes, that is exactly what I have described.
>> >>
>> >>> I don't think that keeping the current Java PreCommit as a proxy for
>> the
>> >>> the Java PostCommit is the right way to go but I also don't have the
>> time to
>> >>> implement what your actually asking for.
>> >>
>> >>
>> >> Mostly I thought this might be very easy based on the fact that they
>> are
>> >> nearly identical. If not, oh well.
>> >>
>> >> Kenn
>> >>
>> >>
>> >>> It seems more likely that migrating the PostCommit to Gradle will be
>> less
>> >>> work then adding the functionality but your argument where the
>> PreCommit is
>> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
>> >>> PostCommits and so forth requiring even more migration to happen
>> before you
>> >>> don't have to worry about maintaining Maven/breaking post commits.
>> >>>
>> >>> I'm fine with leaving both the Java/Gradle PreCommits running for now
>> and
>> >>> hopefully as more of the PostCommits are migrated off we will be able
>> to
>> >>> remove it.
>> >>>
>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles 
>> wrote:
>> 
>>  Separate history (for easy dashboarding, health stats, etc) and
>>  notification (email to dev@ for postcommits, nothing for
>> precommits) for pre
>>  & post commit targets.
>> 
>>  A post commit failure is always a problem to be triaged at high
>>  priority, while a precommit failure is just a natural occurrence.
>> 
>>  On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik 
>> wrote:
>> >
>> > Ken, I'm probably not seeing something but how does using the
>> PreCommit
>> > as a proxy improve upon just running the post commit via the phrase
>> it
>> > already supports ('Run Java PostCommit')?
>> >
>> > On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles 
>> > wrote:
>> >>
>> >> Indeed, we've already had the discussion a couple of times and I
>> think
>> >> the criteria are clearly met. Incremental progress is a good thing
>> and we
>> >> shouldn't block it.
>> >>
>> >> OTOH I see where Romain is coming from and I have a good example
>> that
>> >> supports a slightly different action. Consider
>> >> https://github.com/apache/beam/pull/4740 which fixes some errors
>> in how we
>> >> use dependency mechanisms.
>> >>
>> >> This PR is green except that I need to fix some Maven pom slightly
>> >> more. That is throwaway work. I would love to just not have to do
>> it. But
>> >> removing the precommit does not actually make the PR OK to merge.
>> It would
>> >> cause postcommits to fail.
>> >>
>> >> We can hope such situations are rare. I think I tend to be hit by
>> this
>> >> more often than most, as I work with the project build health
>> quite a bit.
>> >>
>> >> Here is a proposal to support these things: instead of deleting the
>> >> job in #4814, move it to not run automatically but only via a
>> phrase. 

Re: Gradle status

[Romain Manni-Bucau]

Not really Ismaël, this thread was about to do it at once and have 1 day to
fix it all.

As mentionned at the very beginning nobody maintains the 2 system so it
must stop after months so either we drop maven or gradle *at once*
or we keep a state where each dev does what he wants and the build system
just doesn't work.

2018-03-22 15:42 GMT+01:00 Ismaël Mejía :

> I don't think that removing all maven descriptors was the expected
> path, no ? Or even a good idea at this moment.
>
> I understood that what we were going to do was to replace
> incrementally the CI until we cover the whole maven functionality and
> then remove it, from looking at the JIRA ticket
> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
> covering the complete maven functionality in particular for the
> release part that could be the biggest pain point.
>
>
> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>  wrote:
> > hey guys,
> >
> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
> monday?
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
> >>
> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik  wrote:
> >>>
> >>> Based upon your description it seems as though you would rather have a
> >>> way to run existing postcommits without it impacting dashboard/health
> >>> stats/notifications/ (We have just run the PostCommits on PRs for
> >>> additional validation (like upgrading the Dataflow container image)).
> >>
> >>
> >> Yes, that is exactly what I have described.
> >>
> >>> I don't think that keeping the current Java PreCommit as a proxy for
> the
> >>> the Java PostCommit is the right way to go but I also don't have the
> time to
> >>> implement what your actually asking for.
> >>
> >>
> >> Mostly I thought this might be very easy based on the fact that they are
> >> nearly identical. If not, oh well.
> >>
> >> Kenn
> >>
> >>
> >>> It seems more likely that migrating the PostCommit to Gradle will be
> less
> >>> work then adding the functionality but your argument where the
> PreCommit is
> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
> >>> PostCommits and so forth requiring even more migration to happen
> before you
> >>> don't have to worry about maintaining Maven/breaking post commits.
> >>>
> >>> I'm fine with leaving both the Java/Gradle PreCommits running for now
> and
> >>> hopefully as more of the PostCommits are migrated off we will be able
> to
> >>> remove it.
> >>>
> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles 
> wrote:
> 
>  Separate history (for easy dashboarding, health stats, etc) and
>  notification (email to dev@ for postcommits, nothing for precommits)
> for pre
>  & post commit targets.
> 
>  A post commit failure is always a problem to be triaged at high
>  priority, while a precommit failure is just a natural occurrence.
> 
>  On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik  wrote:
> >
> > Ken, I'm probably not seeing something but how does using the
> PreCommit
> > as a proxy improve upon just running the post commit via the phrase
> it
> > already supports ('Run Java PostCommit')?
> >
> > On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles 
> > wrote:
> >>
> >> Indeed, we've already had the discussion a couple of times and I
> think
> >> the criteria are clearly met. Incremental progress is a good thing
> and we
> >> shouldn't block it.
> >>
> >> OTOH I see where Romain is coming from and I have a good example
> that
> >> supports a slightly different action. Consider
> >> https://github.com/apache/beam/pull/4740 which fixes some errors
> in how we
> >> use dependency mechanisms.
> >>
> >> This PR is green except that I need to fix some Maven pom slightly
> >> more. That is throwaway work. I would love to just not have to do
> it. But
> >> removing the precommit does not actually make the PR OK to merge.
> It would
> >> cause postcommits to fail.
> >>
> >> We can hope such situations are rare. I think I tend to be hit by
> this
> >> more often than most, as I work with the project build health quite
> a bit.
> >>
> >> Here is a proposal to support these things: instead of deleting the
> >> job in #4814, move it to not run automatically but only via a
> phrase. In
> >> fact, you could migrate it to be the manually-invoked version of the
> >> postcommit job as we've discussed a couple times. Then if someone
> is working
> >> on the build in something like #4740 they can invoke it manually.
> >>
> >> Kenn
> >>
> >> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik 
> wrote:
> >>>
> >>> Based upon the criteria that was 

Re: Gradle status

[Ismaël Mejía]

I don't think that removing all maven descriptors was the expected
path, no ? Or even a good idea at this moment.

I understood that what we were going to do was to replace
incrementally the CI until we cover the whole maven functionality and
then remove it, from looking at the JIRA ticket
https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
covering the complete maven functionality in particular for the
release part that could be the biggest pain point.


On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
 wrote:
> hey guys,
>
> 2.4 is out, do we plan to drop all maven descriptors tomorrow or on monday?
>
>
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>
> 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
>>
>> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik  wrote:
>>>
>>> Based upon your description it seems as though you would rather have a
>>> way to run existing postcommits without it impacting dashboard/health
>>> stats/notifications/ (We have just run the PostCommits on PRs for
>>> additional validation (like upgrading the Dataflow container image)).
>>
>>
>> Yes, that is exactly what I have described.
>>
>>> I don't think that keeping the current Java PreCommit as a proxy for the
>>> the Java PostCommit is the right way to go but I also don't have the time to
>>> implement what your actually asking for.
>>
>>
>> Mostly I thought this might be very easy based on the fact that they are
>> nearly identical. If not, oh well.
>>
>> Kenn
>>
>>
>>> It seems more likely that migrating the PostCommit to Gradle will be less
>>> work then adding the functionality but your argument where the PreCommit is
>>> a proxy for the Java PostCommit also applies to the ValidatesRunner
>>> PostCommits and so forth requiring even more migration to happen before you
>>> don't have to worry about maintaining Maven/breaking post commits.
>>>
>>> I'm fine with leaving both the Java/Gradle PreCommits running for now and
>>> hopefully as more of the PostCommits are migrated off we will be able to
>>> remove it.
>>>
>>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles  wrote:

 Separate history (for easy dashboarding, health stats, etc) and
 notification (email to dev@ for postcommits, nothing for precommits) for 
 pre
 & post commit targets.

 A post commit failure is always a problem to be triaged at high
 priority, while a precommit failure is just a natural occurrence.

 On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik  wrote:
>
> Ken, I'm probably not seeing something but how does using the PreCommit
> as a proxy improve upon just running the post commit via the phrase it
> already supports ('Run Java PostCommit')?
>
> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles 
> wrote:
>>
>> Indeed, we've already had the discussion a couple of times and I think
>> the criteria are clearly met. Incremental progress is a good thing and we
>> shouldn't block it.
>>
>> OTOH I see where Romain is coming from and I have a good example that
>> supports a slightly different action. Consider
>> https://github.com/apache/beam/pull/4740 which fixes some errors in how 
>> we
>> use dependency mechanisms.
>>
>> This PR is green except that I need to fix some Maven pom slightly
>> more. That is throwaway work. I would love to just not have to do it. But
>> removing the precommit does not actually make the PR OK to merge. It 
>> would
>> cause postcommits to fail.
>>
>> We can hope such situations are rare. I think I tend to be hit by this
>> more often than most, as I work with the project build health quite a 
>> bit.
>>
>> Here is a proposal to support these things: instead of deleting the
>> job in #4814, move it to not run automatically but only via a phrase. In
>> fact, you could migrate it to be the manually-invoked version of the
>> postcommit job as we've discussed a couple times. Then if someone is 
>> working
>> on the build in something like #4740 they can invoke it manually.
>>
>> Kenn
>>
>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik  wrote:
>>>
>>> Based upon the criteria that was discussed on the mailing list[1], I
>>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>>
>>> 1:
>>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>>
>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau
>>>  wrote:

 Hi Kenneth,

 For now maven covers the full needs of beam. If we start to have
 this kind of PR we become dependent of the 2 builds which is what this
 thread is about avoiding 

Re: Gradle status

[Romain Manni-Bucau]

hey guys,

2.4 is out, do we plan to drop all maven descriptors tomorrow or on monday?


Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book


2018-03-09 21:42 GMT+01:00 Kenneth Knowles :

> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik  wrote:
>
>> Based upon your description it seems as though you would rather have a
>> way to run existing postcommits without it impacting dashboard/health
>> stats/notifications/ (We have just run the PostCommits on PRs for
>> additional validation (like upgrading the Dataflow container image)).
>>
>
> Yes, that is exactly what I have described.
>
> I don't think that keeping the current Java PreCommit as a proxy for the
>> the Java PostCommit is the right way to go but I also don't have the time
>> to implement what your actually asking for.
>>
>
> Mostly I thought this might be very easy based on the fact that they are
> nearly identical. If not, oh well.
>
> Kenn
>
>
> It seems more likely that migrating the PostCommit to Gradle will be less
>> work then adding the functionality but your argument where the PreCommit is
>> a proxy for the Java PostCommit also applies to the ValidatesRunner
>> PostCommits and so forth requiring even more migration to happen before you
>> don't have to worry about maintaining Maven/breaking post commits.
>>
>> I'm fine with leaving both the Java/Gradle PreCommits running for now and
>> hopefully as more of the PostCommits are migrated off we will be able to
>> remove it.
>>
>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles  wrote:
>>
>>> Separate history (for easy dashboarding, health stats, etc) and
>>> notification (email to dev@ for postcommits, nothing for precommits)
>>> for pre & post commit targets.
>>>
>>> A post commit failure is always a problem to be triaged at high
>>> priority, while a precommit failure is just a natural occurrence.
>>>
>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik  wrote:
>>>
 Ken, I'm probably not seeing something but how does using the PreCommit
 as a proxy improve upon just running the post commit via the phrase it
 already supports ('Run Java PostCommit')?

 On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles 
 wrote:

> Indeed, we've already had the discussion a couple of times and I think
> the criteria are clearly met. Incremental progress is a good thing and we
> shouldn't block it.
>
> OTOH I see where Romain is coming from and I have a good example that
> supports a slightly different action. Consider
> https://github.com/apache/beam/pull/4740 which fixes some errors in
> how we use dependency mechanisms.
>
> This PR is green except that I need to fix some Maven pom slightly
> more. That is throwaway work. I would love to just not have to do it. But
> removing the precommit does not actually make the PR OK to merge. It would
> cause postcommits to fail.
>
> We can hope such situations are rare. I think I tend to be hit by this
> more often than most, as I work with the project build health quite a bit.
>
> Here is a proposal to support these things: instead of deleting the
> job in #4814, move it to not run automatically but only via a phrase. In
> fact, you could migrate it to be the manually-invoked version of the
> postcommit job as we've discussed a couple times. Then if someone is
> working on the build in something like #4740 they can invoke it manually.
>
> Kenn
>
> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik  wrote:
>
>> Based upon the criteria that was discussed on the mailing list[1], I
>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>
>> 1: https://lists.apache.org/thread.html/
>> 7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%
>> 3Cdev.beam.apache.org%3E
>>
>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> Hi Kenneth,
>>>
>>> For now maven covers the full needs of beam. If we start to have
>>> this kind of PR we become dependent of the 2 builds which is what this
>>> thread is about avoiding so tempted to say it must be a PR drop 
>>> completely
>>> maven or nothing as mentionned before.
>>>
>>> Le 9 mars 2018 04:48, "Kenneth Knowles"  a écrit :
>>>
 I would like to briefly re-focus this discussion and suggest that
 we merge https://github.com/apache/beam/pull/4814.

 The only material objection I've heard is that it means the
 precommit no longer 

Re: Gradle status

[Kenneth Knowles]

On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik  wrote:

> Based upon your description it seems as though you would rather have a way
> to run existing postcommits without it impacting dashboard/health
> stats/notifications/ (We have just run the PostCommits on PRs for
> additional validation (like upgrading the Dataflow container image)).
>

Yes, that is exactly what I have described.

I don't think that keeping the current Java PreCommit as a proxy for the
> the Java PostCommit is the right way to go but I also don't have the time
> to implement what your actually asking for.
>

Mostly I thought this might be very easy based on the fact that they are
nearly identical. If not, oh well.

Kenn


It seems more likely that migrating the PostCommit to Gradle will be less
> work then adding the functionality but your argument where the PreCommit is
> a proxy for the Java PostCommit also applies to the ValidatesRunner
> PostCommits and so forth requiring even more migration to happen before you
> don't have to worry about maintaining Maven/breaking post commits.
>
> I'm fine with leaving both the Java/Gradle PreCommits running for now and
> hopefully as more of the PostCommits are migrated off we will be able to
> remove it.
>
> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles  wrote:
>
>> Separate history (for easy dashboarding, health stats, etc) and
>> notification (email to dev@ for postcommits, nothing for precommits) for
>> pre & post commit targets.
>>
>> A post commit failure is always a problem to be triaged at high priority,
>> while a precommit failure is just a natural occurrence.
>>
>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik  wrote:
>>
>>> Ken, I'm probably not seeing something but how does using the PreCommit
>>> as a proxy improve upon just running the post commit via the phrase it
>>> already supports ('Run Java PostCommit')?
>>>
>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles  wrote:
>>>
 Indeed, we've already had the discussion a couple of times and I think
 the criteria are clearly met. Incremental progress is a good thing and we
 shouldn't block it.

 OTOH I see where Romain is coming from and I have a good example that
 supports a slightly different action. Consider
 https://github.com/apache/beam/pull/4740 which fixes some errors in
 how we use dependency mechanisms.

 This PR is green except that I need to fix some Maven pom slightly
 more. That is throwaway work. I would love to just not have to do it. But
 removing the precommit does not actually make the PR OK to merge. It would
 cause postcommits to fail.

 We can hope such situations are rare. I think I tend to be hit by this
 more often than most, as I work with the project build health quite a bit.

 Here is a proposal to support these things: instead of deleting the job
 in #4814, move it to not run automatically but only via a phrase. In fact,
 you could migrate it to be the manually-invoked version of the postcommit
 job as we've discussed a couple times. Then if someone is working on the
 build in something like #4740 they can invoke it manually.

 Kenn

 On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik  wrote:

> Based upon the criteria that was discussed on the mailing list[1], I
> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>
> 1:
> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>
> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Hi Kenneth,
>>
>> For now maven covers the full needs of beam. If we start to have this
>> kind of PR we become dependent of the 2 builds which is what this thread 
>> is
>> about avoiding so tempted to say it must be a PR drop completely maven or
>> nothing as mentionned before.
>>
>> Le 9 mars 2018 04:48, "Kenneth Knowles"  a écrit :
>>
>>> I would like to briefly re-focus this discussion and suggest that we
>>> merge https://github.com/apache/beam/pull/4814.
>>>
>>> The only material objection I've heard is that it means the
>>> precommit no longer tests exactly what is built for release. It is a 
>>> valid
>>> concern, but we have mvn postcommit so the coverage is not lost. The
>>> benefits of quicker reviews and less Jenkins congestion seem worth it 
>>> to me.
>>>
>>> Kenn
>>>
>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 @Luskasz: not sure Im the best to host it since I know more gradle
 internals that user interface/ecosystem but happy to help. Will also
 require a "sudo" merger for this day to merge fixes asap - guess we can

Re: Gradle status

[Lukasz Cwik]

Based upon your description it seems as though you would rather have a way
to run existing postcommits without it impacting dashboard/health
stats/notifications/ (We have just run the PostCommits on PRs for
additional validation (like upgrading the Dataflow container image)).

I don't think that keeping the current Java PreCommit as a proxy for the
the Java PostCommit is the right way to go but I also don't have the time
to implement what your actually asking for.
It seems more likely that migrating the PostCommit to Gradle will be less
work then adding the functionality but your argument where the PreCommit is
a proxy for the Java PostCommit also applies to the ValidatesRunner
PostCommits and so forth requiring even more migration to happen before you
don't have to worry about maintaining Maven/breaking post commits.

I'm fine with leaving both the Java/Gradle PreCommits running for now and
hopefully as more of the PostCommits are migrated off we will be able to
remove it.

On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles  wrote:

> Separate history (for easy dashboarding, health stats, etc) and
> notification (email to dev@ for postcommits, nothing for precommits) for
> pre & post commit targets.
>
> A post commit failure is always a problem to be triaged at high priority,
> while a precommit failure is just a natural occurrence.
>
> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik  wrote:
>
>> Ken, I'm probably not seeing something but how does using the PreCommit
>> as a proxy improve upon just running the post commit via the phrase it
>> already supports ('Run Java PostCommit')?
>>
>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles  wrote:
>>
>>> Indeed, we've already had the discussion a couple of times and I think
>>> the criteria are clearly met. Incremental progress is a good thing and we
>>> shouldn't block it.
>>>
>>> OTOH I see where Romain is coming from and I have a good example that
>>> supports a slightly different action. Consider
>>> https://github.com/apache/beam/pull/4740 which fixes some errors in how
>>> we use dependency mechanisms.
>>>
>>> This PR is green except that I need to fix some Maven pom slightly more.
>>> That is throwaway work. I would love to just not have to do it. But
>>> removing the precommit does not actually make the PR OK to merge. It would
>>> cause postcommits to fail.
>>>
>>> We can hope such situations are rare. I think I tend to be hit by this
>>> more often than most, as I work with the project build health quite a bit.
>>>
>>> Here is a proposal to support these things: instead of deleting the job
>>> in #4814, move it to not run automatically but only via a phrase. In fact,
>>> you could migrate it to be the manually-invoked version of the postcommit
>>> job as we've discussed a couple times. Then if someone is working on the
>>> build in something like #4740 they can invoke it manually.
>>>
>>> Kenn
>>>
>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik  wrote:
>>>
 Based upon the criteria that was discussed on the mailing list[1], I
 would agree with Kenn about merging PR/4814 (drop Java Maven precommit).

 1: https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f
 6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E

 On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Hi Kenneth,
>
> For now maven covers the full needs of beam. If we start to have this
> kind of PR we become dependent of the 2 builds which is what this thread 
> is
> about avoiding so tempted to say it must be a PR drop completely maven or
> nothing as mentionned before.
>
> Le 9 mars 2018 04:48, "Kenneth Knowles"  a écrit :
>
>> I would like to briefly re-focus this discussion and suggest that we
>> merge https://github.com/apache/beam/pull/4814.
>>
>> The only material objection I've heard is that it means the precommit
>> no longer tests exactly what is built for release. It is a valid concern,
>> but we have mvn postcommit so the coverage is not lost. The benefits of
>> quicker reviews and less Jenkins congestion seem worth it to me.
>>
>> Kenn
>>
>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> @Luskasz: not sure Im the best to host it since I know more gradle
>>> internals that user interface/ecosystem but happy to help. Will also
>>> require a "sudo" merger for this day to merge fixes asap - guess we can
>>> bypass reviews or have a fast cycle plan for this day to avoid it to be 
>>> a
>>> week?
>>>
>>> Le 8 mars 2018 21:08, "Kenneth Knowles"  a écrit :
>>>
>>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven,
>>> and what I mean is that I have added enough hints that it works OOTB
>>> already. 

Re: Gradle status

[Romain Manni-Bucau]

Is having a 2 days period to "merge all mvn related pr we can" next week an
option? If so it can be enough and well fix gradle in the fix day (goal
being to reduce pr number a lot).

Le 9 mars 2018 20:40, "Kenneth Knowles"  a écrit :

> Separate history (for easy dashboarding, health stats, etc) and
> notification (email to dev@ for postcommits, nothing for precommits) for
> pre & post commit targets.
>
> A post commit failure is always a problem to be triaged at high priority,
> while a precommit failure is just a natural occurrence.
>
> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik  wrote:
>
>> Ken, I'm probably not seeing something but how does using the PreCommit
>> as a proxy improve upon just running the post commit via the phrase it
>> already supports ('Run Java PostCommit')?
>>
>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles  wrote:
>>
>>> Indeed, we've already had the discussion a couple of times and I think
>>> the criteria are clearly met. Incremental progress is a good thing and we
>>> shouldn't block it.
>>>
>>> OTOH I see where Romain is coming from and I have a good example that
>>> supports a slightly different action. Consider
>>> https://github.com/apache/beam/pull/4740 which fixes some errors in how
>>> we use dependency mechanisms.
>>>
>>> This PR is green except that I need to fix some Maven pom slightly more.
>>> That is throwaway work. I would love to just not have to do it. But
>>> removing the precommit does not actually make the PR OK to merge. It would
>>> cause postcommits to fail.
>>>
>>> We can hope such situations are rare. I think I tend to be hit by this
>>> more often than most, as I work with the project build health quite a bit.
>>>
>>> Here is a proposal to support these things: instead of deleting the job
>>> in #4814, move it to not run automatically but only via a phrase. In fact,
>>> you could migrate it to be the manually-invoked version of the postcommit
>>> job as we've discussed a couple times. Then if someone is working on the
>>> build in something like #4740 they can invoke it manually.
>>>
>>> Kenn
>>>
>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik  wrote:
>>>
 Based upon the criteria that was discussed on the mailing list[1], I
 would agree with Kenn about merging PR/4814 (drop Java Maven precommit).

 1: https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f
 6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E

 On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
 rmannibu...@gmail.com> wrote:

> Hi Kenneth,
>
> For now maven covers the full needs of beam. If we start to have this
> kind of PR we become dependent of the 2 builds which is what this thread 
> is
> about avoiding so tempted to say it must be a PR drop completely maven or
> nothing as mentionned before.
>
> Le 9 mars 2018 04:48, "Kenneth Knowles"  a écrit :
>
>> I would like to briefly re-focus this discussion and suggest that we
>> merge https://github.com/apache/beam/pull/4814.
>>
>> The only material objection I've heard is that it means the precommit
>> no longer tests exactly what is built for release. It is a valid concern,
>> but we have mvn postcommit so the coverage is not lost. The benefits of
>> quicker reviews and less Jenkins congestion seem worth it to me.
>>
>> Kenn
>>
>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
>> rmannibu...@gmail.com> wrote:
>>
>>> @Luskasz: not sure Im the best to host it since I know more gradle
>>> internals that user interface/ecosystem but happy to help. Will also
>>> require a "sudo" merger for this day to merge fixes asap - guess we can
>>> bypass reviews or have a fast cycle plan for this day to avoid it to be 
>>> a
>>> week?
>>>
>>> Le 8 mars 2018 21:08, "Kenneth Knowles"  a écrit :
>>>
>>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven,
>>> and what I mean is that I have added enough hints that it works OOTB
>>> already. The rest of my instructions are just how you should override
>>> IntelliJ's defaults to have a proper dev env - mostly just about storing
>>> files outside the git clone.
>>>
>>>
>>> Hmm it is always super slow here and not as integrated as maven
>>> which is smooth thanks to the combination of idea build and maven plugin
>>> config read. Import is also faster cause it just reads meta and doesnt 
>>> run
>>> anything. Hope it is a "a few times" issues at the moment but not yet 
>>> sure.
>>>
>>>
>>>
>>> @Łukasz - yea I just mean the ones that I find in my email and
>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which are
>>> the ones you classify as "totally not working". I would love to disable
>>> these.
>>>
>>> On 

Re: Gradle status

[Kenneth Knowles]

Separate history (for easy dashboarding, health stats, etc) and
notification (email to dev@ for postcommits, nothing for precommits) for
pre & post commit targets.

A post commit failure is always a problem to be triaged at high priority,
while a precommit failure is just a natural occurrence.

On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik  wrote:

> Ken, I'm probably not seeing something but how does using the PreCommit as
> a proxy improve upon just running the post commit via the phrase it already
> supports ('Run Java PostCommit')?
>
> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles  wrote:
>
>> Indeed, we've already had the discussion a couple of times and I think
>> the criteria are clearly met. Incremental progress is a good thing and we
>> shouldn't block it.
>>
>> OTOH I see where Romain is coming from and I have a good example that
>> supports a slightly different action. Consider
>> https://github.com/apache/beam/pull/4740 which fixes some errors in how
>> we use dependency mechanisms.
>>
>> This PR is green except that I need to fix some Maven pom slightly more.
>> That is throwaway work. I would love to just not have to do it. But
>> removing the precommit does not actually make the PR OK to merge. It would
>> cause postcommits to fail.
>>
>> We can hope such situations are rare. I think I tend to be hit by this
>> more often than most, as I work with the project build health quite a bit.
>>
>> Here is a proposal to support these things: instead of deleting the job
>> in #4814, move it to not run automatically but only via a phrase. In fact,
>> you could migrate it to be the manually-invoked version of the postcommit
>> job as we've discussed a couple times. Then if someone is working on the
>> build in something like #4740 they can invoke it manually.
>>
>> Kenn
>>
>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik  wrote:
>>
>>> Based upon the criteria that was discussed on the mailing list[1], I
>>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>>
>>> 1:
>>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>>
>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Hi Kenneth,

 For now maven covers the full needs of beam. If we start to have this
 kind of PR we become dependent of the 2 builds which is what this thread is
 about avoiding so tempted to say it must be a PR drop completely maven or
 nothing as mentionned before.

 Le 9 mars 2018 04:48, "Kenneth Knowles"  a écrit :

> I would like to briefly re-focus this discussion and suggest that we
> merge https://github.com/apache/beam/pull/4814.
>
> The only material objection I've heard is that it means the precommit
> no longer tests exactly what is built for release. It is a valid concern,
> but we have mvn postcommit so the coverage is not lost. The benefits of
> quicker reviews and less Jenkins congestion seem worth it to me.
>
> Kenn
>
> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> @Luskasz: not sure Im the best to host it since I know more gradle
>> internals that user interface/ecosystem but happy to help. Will also
>> require a "sudo" merger for this day to merge fixes asap - guess we can
>> bypass reviews or have a fast cycle plan for this day to avoid it to be a
>> week?
>>
>> Le 8 mars 2018 21:08, "Kenneth Knowles"  a écrit :
>>
>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven,
>> and what I mean is that I have added enough hints that it works OOTB
>> already. The rest of my instructions are just how you should override
>> IntelliJ's defaults to have a proper dev env - mostly just about storing
>> files outside the git clone.
>>
>>
>> Hmm it is always super slow here and not as integrated as maven which
>> is smooth thanks to the combination of idea build and maven plugin config
>> read. Import is also faster cause it just reads meta and doesnt run
>> anything. Hope it is a "a few times" issues at the moment but not yet 
>> sure.
>>
>>
>>
>> @Łukasz - yea I just mean the ones that I find in my email and
>> browsing https://builds.apache.org/view/A-D/view/Beam/ which are the
>> ones you classify as "totally not working". I would love to disable 
>> these.
>>
>> On the subject of running things on a pending PR - we should not run
>> postcommit jobs on PRs. We should make separate (optional) precommit jobs
>> that run the same build commands. That will give a more clear history and
>> allow trivial configuration of which jobs deserve alert emails and which
>> are not a problem. This is easy but I've been waiting to do it after