Dependency driven works, incremental works for most java modules. I use incremental almost all the time and just do one validation pass at the end before opening the PR where I use '--rerun-tasks' to be sure. Allows me to iterate on a task in seconds.
On Wed, Jan 24, 2018 at 4:07 PM, Kenneth Knowles <k...@google.com> wrote: > These are two different things: dependency-driven build (which works) and > incremental build (which seems not to, at least right now?). > > On Wed, Jan 24, 2018 at 2:24 PM, Romain Manni-Bucau <rmannibu...@gmail.com > > wrote: > >> Hmm, I'll try to refine it then next time we work with Ismael but can be >> a setup issue or a human (bad command) issue at the end. Thanks for the >> help, will make next iteration way easier probably :) >> >> >> Romain Manni-Bucau >> @rmannibucau <https://twitter.com/rmannibucau> | Blog >> <https://rmannibucau.metawerx.net/> | Old Blog >> <http://rmannibucau.wordpress.com> | Github >> <https://github.com/rmannibucau> | LinkedIn >> <https://www.linkedin.com/in/rmannibucau> >> >> 2018-01-24 23:05 GMT+01:00 Lukasz Cwik <lc...@google.com>: >> >>> Tasks always run any dependencies that are required. So if you ask to >>> run test it shouldn't run javadoc/checkstyle/... but should compile the >>> code and compile the code of all dependencies. test should never have a >>> dependency on checkstyle or javadoc or similar 'check' like tasks as they >>> shouldn't be needed. >>> >>> I set up the gradle build so that everytime you run a command in gradle, >>> it generates a task dependency tree dot file (look for visteg.dot inside >>> build/reports). I uploaded this one to imgur[1] for the >>> ':sdks:java:core:build' task to show what tasks are required. Note that >>> 'sdks:java:core:test' doesn't depend on checkstyle or spotless. >>> >>> 1: https://imgur.com/a/ZvYUX >>> >>> On Wed, Jan 24, 2018 at 12:50 PM, Romain Manni-Bucau < >>> rmannibu...@gmail.com> wrote: >>> >>>> Hmm, do I miss something or it only works for iterative runs when >>>> trying to identify an issue and not for the case you rebuild due to code >>>> changes (where you would need like 5-6 tasks at least - generate, compile, >>>> test, ...)? >>>> >>>> In case it is unclear: there are 2 needs: direct execution/task -> >>>> fulfilled and clarified now (just a doc issue I think), fast cycle skipping >>>> not mandatory tasks like style related ones >>>> >>>> >>>> Romain Manni-Bucau >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>> <http://rmannibucau.wordpress.com> | Github >>>> <https://github.com/rmannibucau> | LinkedIn >>>> <https://www.linkedin.com/in/rmannibucau> >>>> >>>> 2018-01-24 19:50 GMT+01:00 Lukasz Cwik <lc...@google.com>: >>>> >>>>> Gradle already has each task explicitly broken out. Kenn is pointing >>>>> out that you your development use case shouldn't use the './gradlew >>>>> :sdks:java:core:build' task since it is really an aggregator that >>>>> represents do everything within that project. This is the current list of >>>>> tasks available for :sdks:java:core: >>>>> :sdks:java:core:assemble - Assembles the outputs of this project. >>>>> :sdks:java:core:build - Assembles and tests this project. >>>>> :sdks:java:core:buildDependents - Assembles and tests this project >>>>> and all projects that depend on it. >>>>> :sdks:java:core:buildEnvironment - Displays all buildscript >>>>> dependencies declared in project :sdks:java:core. >>>>> :sdks:java:core:buildNeeded - Assembles and tests this project and >>>>> all projects it depends on. >>>>> :sdks:java:core:check - Runs all checks. >>>>> :sdks:java:core:checkstyleMain - Run Checkstyle analysis for main >>>>> classes >>>>> :sdks:java:core:checkstyleTest - Run Checkstyle analysis for test >>>>> classes >>>>> :sdks:java:core:classes - Assembles main classes. >>>>> :sdks:java:core:clean - Deletes the build directory. >>>>> :sdks:java:core:compileJava - Compiles main Java source. >>>>> :sdks:java:core:compileTestJava - Compiles test Java source. >>>>> :sdks:java:core:components - Displays the components produced by >>>>> project :sdks:java:core. [incubating] >>>>> :sdks:java:core:dependencies - Displays all dependencies declared in >>>>> project :sdks:java:core. >>>>> :sdks:java:core:dependencyInsight - Displays the insight into a >>>>> specific dependency in project :sdks:java:core. >>>>> :sdks:java:core:dependencyReport - Generates a report about your >>>>> library dependencies. >>>>> :sdks:java:core:dependentComponents - Displays the dependent >>>>> components of components in project :sdks:java:core. [incubating] >>>>> :sdks:java:core:findbugsMain - Run FindBugs analysis for main classes >>>>> :sdks:java:core:findbugsTest - Run FindBugs analysis for test classes >>>>> :sdks:java:core:generateAvroJava - Generates main Avro Java source >>>>> files from schema/protocol definition files. >>>>> :sdks:java:core:generateAvroProtocol - Generates main Avro protocol >>>>> definition files from IDL files. >>>>> :sdks:java:core:generateTestAvroJava - Generates test Avro Java >>>>> source files from schema/protocol definition files. >>>>> :sdks:java:core:generateTestAvroProtocol - Generates test Avro >>>>> protocol definition files from IDL files. >>>>> :sdks:java:core:help - Displays a help message. >>>>> :sdks:java:core:htmlDependencyReport - Generates an HTML report >>>>> about your library dependencies. >>>>> :sdks:java:core:install - Installs the archives artifacts into the >>>>> local Maven repository. >>>>> :sdks:java:core:jacocoTestCoverageVerification - Verifies code >>>>> coverage metrics based on specified rules for the test task. >>>>> :sdks:java:core:jacocoTestReport - Generates code coverage report >>>>> for the test task. >>>>> :sdks:java:core:jar - Assembles a jar archive containing the main >>>>> classes. >>>>> :sdks:java:core:javadoc - Generates Javadoc API documentation for the >>>>> main source code. >>>>> :sdks:java:core:knows - Do you know who knows? >>>>> :sdks:java:core:model - Displays the configuration model of project >>>>> :sdks:java:core. [incubating] >>>>> :sdks:java:core:packageTests - >>>>> :sdks:java:core:processResources - Processes main resources. >>>>> :sdks:java:core:processTestResources - Processes test resources. >>>>> :sdks:java:core:projectReport - Generates a report about your project. >>>>> :sdks:java:core:projects - Displays the sub-projects of project >>>>> :sdks:java:core. >>>>> :sdks:java:core:properties - Displays the properties of project >>>>> :sdks:java:core. >>>>> :sdks:java:core:propertyReport - Generates a report about your >>>>> properties. >>>>> :sdks:java:core:shadowJar - Create a combined JAR of project and >>>>> runtime dependencies >>>>> :sdks:java:core:shadowTestJar - >>>>> :sdks:java:core:spotlessApply - Applies code formatting steps to >>>>> sourcecode in-place. >>>>> :sdks:java:core:spotlessCheck - Checks that sourcecode satisfies >>>>> formatting steps. >>>>> :sdks:java:core:spotlessJava - >>>>> :sdks:java:core:spotlessJavaApply - >>>>> :sdks:java:core:spotlessJavaCheck - >>>>> :sdks:java:core:taskReport - Generates a report about your tasks. >>>>> :sdks:java:core:tasks - Displays the tasks runnable from project >>>>> :sdks:java:core. >>>>> :sdks:java:core:testClasses - Assembles test classes. >>>>> :sdks:java:core:test - Runs the unit tests. >>>>> :sdks:java:core:updateOfflineRepository - >>>>> >>>>> So if you specify the ':sdks:java:core:test' task then only the tests >>>>> run (and any dependent tasks which are not up to date), while >>>>> ':sdks:java:core:check' >>>>> runs the suite of all checks. If your working on two modules and want to >>>>> run the tests for both you really want to specify each explicit end goal >>>>> that you want like ':sdks:java:core:test' and ':sdks:java:harness:test'. >>>>> Unfortunately incremental builds (https://issues.apache.org/jir >>>>> a/browse/BEAM-3253) are unreliable in a few of the modules (like sql >>>>> and python) so you may find that you need to specify --rerun-tasks to make >>>>> sure that all tasks are rerun even if Gradle thinks they are up to date. >>>>> >>>>> On Tue, Jan 23, 2018 at 3:38 PM, Romain Manni-Bucau < >>>>> rmannibu...@gmail.com> wrote: >>>>> >>>>>> Can have mischecked gradle setup but I don't think we are here yet, >>>>>> if you are not bound in a module and work accross 2 modules and iterate >>>>>> between working on both and one, you will likely not bypass the "checks" >>>>>> in a satisfying way without a long -x command, is there a magic flag I >>>>>> missed? >>>>>> Also not sure about the last point and how gradle helps here - it is >>>>>> rather the opposite due to the way it loads it model IMHO - so not sure >>>>>> what would be the consequence in terms of action(s) but can have missed >>>>>> the >>>>>> point. >>>>>> >>>>>> >>>>>> >>>>>> Romain Manni-Bucau >>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>> <https://www.linkedin.com/in/rmannibucau> >>>>>> >>>>>> 2018-01-24 0:20 GMT+01:00 Kenneth Knowles <k...@google.com>: >>>>>> >>>>>>> On Tue, Jan 23, 2018 at 2:51 PM, Romain Manni-Bucau < >>>>>>> rmannibu...@gmail.com> wrote: >>>>>>> >>>>>>> Hmm, did you read it right Kenn? I think the idea was to skip all >>>>>>>> validation/sanity checks tasks at once (gradle xxxx -Pfast) instead of >>>>>>>> doing it manually (gradle -x findbugs -x checkstyle etc...) >>>>>>>> >>>>>>> >>>>>>> Yes, I read it right. We all want the same thing - not doing a bunch >>>>>>> of extra useless unrequested stuff when developing. The concept of >>>>>>> skipping >>>>>>> is backwards. We don't need configs that skip things, because in a >>>>>>> correct >>>>>>> dependency-driven build they are already not running. >>>>>>> >>>>>>> So since I don't want to pretend to know gradle's invocations yet I >>>>>>> will call it $TOOL. Here's a collection of imaginary commands: >>>>>>> >>>>>>> $TOOL :sdks:java:core:unittest # or $TOOL test :sdks:java:core >>>>>>> or whatever >>>>>>> $TOOL :sdks:java:core:findbugs >>>>>>> $TOOL :sdks:java:core:checkstyle >>>>>>> $TOOL :sdks:java:core:javadoc >>>>>>> >>>>>>> None of these causes any of the others to run. Anything else is a >>>>>>> bug. The `findbugs` and `test` cause a build of the needed jars and >>>>>>> nothing >>>>>>> else. >>>>>>> >>>>>>> Another example: >>>>>>> >>>>>>> $TOOL :runners:core-java:unittest >>>>>>> >>>>>>> This builds the model, the core SDK, and the runners-core module, >>>>>>> then runs the unit tests of the runners-core module. It does not test >>>>>>> SDK >>>>>>> core, or run any javadoc, findbugs, or checkstyle on any module. >>>>>>> Anything >>>>>>> else is a bug. >>>>>>> >>>>>>> Now, to build a precommit that is easy to reproduce on one line, you >>>>>>> could build a compound task >>>>>>> >>>>>>> $TOOL :sdks:java:core:precommit # runs a selection of targets >>>>>>> that we define >>>>>>> >>>>>>> At this point you might want to skip things from the :verify task >>>>>>> here. But really, you probably just want to run the things you are >>>>>>> interested in and you don't need custom hooks in the aggregated task. >>>>>>> >>>>>>> My understanding is that gradle can support all of this, if we are >>>>>>> disciplined. Getting to this point is the main/only reason I supported >>>>>>> gradle. >>>>>>> >>>>>>> Kenn >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> Kenn >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >> >>>>>>>>>> >> diff --git a/examples/java/build.gradle >>>>>>>>>> b/examples/java/build.gradle >>>>>>>>>> >> index 0fc0b17df..001bd8b38 100644 >>>>>>>>>> >> --- a/examples/java/build.gradle >>>>>>>>>> >> +++ b/examples/java/build.gradle >>>>>>>>>> >> @@ -130,7 +130,7 @@ def preCommitAdditionalFlags = [ >>>>>>>>>> >> dataflowStreamingRunner: [ "--streaming=true" ], >>>>>>>>>> >> ] >>>>>>>>>> >> for (String runner : preCommitRunners) { >>>>>>>>>> >> - tasks.create(name: runner + "PreCommit", type: Test) { >>>>>>>>>> >> + tasks.create(name: runner + "PreCommit", type: Test, >>>>>>>>>> description: "Run tests >>>>>>>>>> >> for runner ${runner.replace('Runner', '')}") { >>>>>>>>>> >> def preCommitBeamTestPipelineOptions = [ >>>>>>>>>> >> "--project=apache-beam-testing", >>>>>>>>>> >> "--tempRoot=gs://temp-storage-for-end-to-end-tests", >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> Romain Manni-Bucau >>>>>>>>>> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>>>>> >> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>>>>> >> <http://rmannibucau.wordpress.com> | Github < >>>>>>>>>> https://github.com/rmannibucau> | >>>>>>>>>> >> LinkedIn <https://www.linkedin.com/in/rmannibucau> >>>>>>>>>> >> >>>>>>>>>> >> 2018-01-23 8:45 GMT+01:00 Jean-Baptiste Onofré < >>>>>>>>>> j...@nanthrax.net >>>>>>>>>> >> <mailto:j...@nanthrax.net>>: >>>>>>>>>> >> >>>>>>>>>> >> Hi Romain, >>>>>>>>>> >> >>>>>>>>>> >> I think we are pretty close: agree to add some explicit >>>>>>>>>> tasks & projects names. >>>>>>>>>> >> >>>>>>>>>> >> We can add additional tasks like skipAudit, for instance. >>>>>>>>>> >> >>>>>>>>>> >> As reminder, gradle tasks provides the list of tasks and >>>>>>>>>> gradle projects >>>>>>>>>> >> provides the list of projects/modules. >>>>>>>>>> >> >>>>>>>>>> >> Regards >>>>>>>>>> >> JB >>>>>>>>>> >> >>>>>>>>>> >> On 01/23/2018 08:12 AM, Romain Manni-Bucau wrote: >>>>>>>>>> >> > Hmm, I have to admit docs dont have my favor cause they >>>>>>>>>> are easily outdated and >>>>>>>>>> >> > hard to search but you hit a good point. Starting by >>>>>>>>>> renaming properly the tasks >>>>>>>>>> >> > and maybe writing what is done in build files - since it >>>>>>>>>> is code and even "api >>>>>>>>>> >> > for dev", it requires as much comments than the main api >>>>>>>>>> - can be better to start. >>>>>>>>>> >> > >>>>>>>>>> >> > Also a big switch flag to bypass checkstyle/findbugs/... >>>>>>>>>> can be good while in >>>>>>>>>> >> > dev since these phases cost a looot for nothing while >>>>>>>>>> you validates your code in >>>>>>>>>> >> > runners modules for instance. >>>>>>>>>> >> > >>>>>>>>>> >> > Le 23 janv. 2018 07:15, "Kenneth Knowles" < >>>>>>>>>> k...@google.com <mailto:k...@google.com> >>>>>>>>>> >> > <mailto:k...@google.com <mailto:k...@google.com>>> a >>>>>>>>>> écrit : >>>>>>>>>> >> > >>>>>>>>>> >> > On Mon, Jan 22, 2018 at 10:03 PM, Romain Manni-Bucau >>>>>>>>>> <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> >>>>>>>>>> >> > <mailto:rmannibu...@gmail.com <mailto: >>>>>>>>>> rmannibu...@gmail.com>>> wrote: >>>>>>>>>> >> > >>>>>>>>>> >> > @Kenneth: why not dropping the doc for a script >>>>>>>>>> with comments in the >>>>>>>>>> >> > project? A "RUNME.sh" ;). >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> >> > That's cool, too, but also any shell one liner can >>>>>>>>>> be a gradle one >>>>>>>>>> >> liner or >>>>>>>>>> >> > mvn two/three liner :-). it is just trading one >>>>>>>>>> command that you cannot >>>>>>>>>> >> > guess easily for a different one that you still >>>>>>>>>> can't guess easily. >>>>>>>>>> >> > >>>>>>>>>> >> > For example, are the SparkRunner ValidatesRunner >>>>>>>>>> tests in the >>>>>>>>>> >> SparkRunner or >>>>>>>>>> >> > the core SDK or a third module that integrates the >>>>>>>>>> two? And why would you >>>>>>>>>> >> > know that the example ITs are called >>>>>>>>>> "sparkRunnerPreCommit"? It >>>>>>>>>> >> doesn't even >>>>>>>>>> >> > make sense really to have "precommit" or >>>>>>>>>> "postcommit" except as aliases to >>>>>>>>>> >> > make it easy to repro Jenkins' behavior - they have >>>>>>>>>> no other intrinsic >>>>>>>>>> >> meaning. >>>>>>>>>> >> > >>>>>>>>>> >> > So I was proposing a mapping from "full sentence + >>>>>>>>>> description" to one >>>>>>>>>> >> liner >>>>>>>>>> >> > to help people navigate the targets that we set up. >>>>>>>>>> Some web page or doc >>>>>>>>>> >> > that people can just quickly scan to find out to do >>>>>>>>>> common things, easier >>>>>>>>>> >> > than groovy or XML. >>>>>>>>>> >> > >>>>>>>>>> >> > Kenn >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> >> >>>>>>>>>> >> -- >>>>>>>>>> >> Jean-Baptiste Onofré >>>>>>>>>> >> jbono...@apache.org <mailto:jbono...@apache.org> >>>>>>>>>> >> http://blog.nanthrax.net >>>>>>>>>> >> Talend - http://www.talend.com >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> > >>>>>>>>>> > -- >>>>>>>>>> > Jean-Baptiste Onofré >>>>>>>>>> > jbono...@apache.org >>>>>>>>>> > http://blog.nanthrax.net >>>>>>>>>> > Talend - http://www.talend.com >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
