Oh wow! Thanks for taking over so fast :) Would really helpful if that would fix some flakiness also.
michel On Sun, Oct 28, 2018 at 9:37 PM Kenneth Knowles <[email protected]> wrote: > Opened https://github.com/apache/beam/pull/6864 with your change but > would love to get you GitHub contributor credit in the metrics FWIW. > > Kenn > > On Sun, Oct 28, 2018 at 12:52 PM Kenneth Knowles <[email protected]> wrote: > >> Nice! >> >> Do you have a PR open? Let's get this change in immediately. >> >> This is probably the root cause of the flood of failures in >> https://issues.apache.org/jira/browse/BEAM-5035 / >> https://issues.apache.org/jira/browse/BEAM-5207 / >> https://issues.apache.org/jira/browse/BEAM-5116 that led to us disabling >> parallel builds. >> >> I've been slowly working towards two other independently useful changes >> that I hoped would help ((1) don't depend on test jars and (2) vendor >> dependencies so we just don't shade at all). Your solution will get us a >> fast build sooner. >> >> Kenn >> >> On Sun, Oct 28, 2018 at 9:57 AM Michael Luckey <[email protected]> >> wrote: >> >>> Hi, >>> >>> currently we bundle our tests within packageTests task [1] and >>> additionally create a shadowed version with shadowTestJar. >>> >>> Unfortunately both task build the jar with the same classifier, which >>> results in each task overwriting the others output, preventing any >>> up-to-date checks from working. >>> >>> Changing the build slightly to produce different artifacts >>> >>> --- >>> a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy >>> >>> +++ >>> b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy >>> >>> @@ -588,7 +588,7 @@ class BeamModulePlugin implements Plugin<Project> { >>> >>> >>> >>> // Ensure that tests are packaged and part of the artifact set. >>> >>> project.task('packageTests', type: Jar) { >>> >>> - classifier = 'tests' >>> >>> + classifier = 'tests-unshaded' >>> >>> from project.sourceSets.test.output >>> >>> } >>> >>> project.artifacts.archives project.packageTests >>> >>> and so enabling up-to-date checking cuts build time [3] to half (on a >>> build ignoring python/go/website where up-to-date checks currently do not >>> work anyway) [4]. >>> >>> As I currently do not fully understand all consequences, I am wondering, >>> whether we could apply that simple fix or wether there is some hidden >>> magic, which relies on both artifacts be named the same. >>> >>> Anyone able to provide further insights here? >>> >>> Thx, >>> >>> michel >>> >>> >>> [1] >>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L609-L614 >>> [2] >>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L766-L780 >>> [3] >>> Profiled build: -x :beam-sdks-python:build -x :beam-sdks-go:build -x >>> :beam-website:build build >>> DescriptionDuration >>> Total Build Time 1m28.94s >>> Startup 1.657s >>> Settings and BuildSrc 1.870s >>> Loading Projects 0.035s >>> Configuring Projects 4.565s >>> Task Execution 1m17.11s >>> [4] >>> Profiled build: -x :beam-sdks-python:build -x :beam-sdks-go:build -x >>> :beam-website:build build >>> >>> DescriptionDuration >>> Total Build Time 46.781s >>> Startup 1.633s >>> Settings and BuildSrc 2.004s >>> Loading Projects 0.061s >>> Configuring Projects 4.883s >>> Task Execution 34.121s >>> >>> >>> >>>
