These are two different things: dependency-driven build (which works) and incremental build (which seems not to, at least right now?).
On Wed, Jan 24, 2018 at 2:24 PM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Hmm, I'll try to refine it then next time we work with Ismael but can be a > setup issue or a human (bad command) issue at the end. Thanks for the help, > will make next iteration way easier probably :) > > > Romain Manni-Bucau > @rmannibucau <https://twitter.com/rmannibucau> | Blog > <https://rmannibucau.metawerx.net/> | Old Blog > <http://rmannibucau.wordpress.com> | Github > <https://github.com/rmannibucau> | LinkedIn > <https://www.linkedin.com/in/rmannibucau> > > 2018-01-24 23:05 GMT+01:00 Lukasz Cwik <lc...@google.com>: > >> Tasks always run any dependencies that are required. So if you ask to run >> test it shouldn't run javadoc/checkstyle/... but should compile the code >> and compile the code of all dependencies. test should never have a >> dependency on checkstyle or javadoc or similar 'check' like tasks as they >> shouldn't be needed. >> >> I set up the gradle build so that everytime you run a command in gradle, >> it generates a task dependency tree dot file (look for visteg.dot inside >> build/reports). I uploaded this one to imgur[1] for the >> ':sdks:java:core:build' task to show what tasks are required. Note that >> 'sdks:java:core:test' doesn't depend on checkstyle or spotless. >> >> 1: https://imgur.com/a/ZvYUX >> >> On Wed, Jan 24, 2018 at 12:50 PM, Romain Manni-Bucau < >> rmannibu...@gmail.com> wrote: >> >>> Hmm, do I miss something or it only works for iterative runs when trying >>> to identify an issue and not for the case you rebuild due to code changes >>> (where you would need like 5-6 tasks at least - generate, compile, test, >>> ...)? >>> >>> In case it is unclear: there are 2 needs: direct execution/task -> >>> fulfilled and clarified now (just a doc issue I think), fast cycle skipping >>> not mandatory tasks like style related ones >>> >>> >>> Romain Manni-Bucau >>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>> <https://rmannibucau.metawerx.net/> | Old Blog >>> <http://rmannibucau.wordpress.com> | Github >>> <https://github.com/rmannibucau> | LinkedIn >>> <https://www.linkedin.com/in/rmannibucau> >>> >>> 2018-01-24 19:50 GMT+01:00 Lukasz Cwik <lc...@google.com>: >>> >>>> Gradle already has each task explicitly broken out. Kenn is pointing >>>> out that you your development use case shouldn't use the './gradlew >>>> :sdks:java:core:build' task since it is really an aggregator that >>>> represents do everything within that project. This is the current list of >>>> tasks available for :sdks:java:core: >>>> :sdks:java:core:assemble - Assembles the outputs of this project. >>>> :sdks:java:core:build - Assembles and tests this project. >>>> :sdks:java:core:buildDependents - Assembles and tests this project >>>> and all projects that depend on it. >>>> :sdks:java:core:buildEnvironment - Displays all buildscript >>>> dependencies declared in project :sdks:java:core. >>>> :sdks:java:core:buildNeeded - Assembles and tests this project and all >>>> projects it depends on. >>>> :sdks:java:core:check - Runs all checks. >>>> :sdks:java:core:checkstyleMain - Run Checkstyle analysis for main >>>> classes >>>> :sdks:java:core:checkstyleTest - Run Checkstyle analysis for test >>>> classes >>>> :sdks:java:core:classes - Assembles main classes. >>>> :sdks:java:core:clean - Deletes the build directory. >>>> :sdks:java:core:compileJava - Compiles main Java source. >>>> :sdks:java:core:compileTestJava - Compiles test Java source. >>>> :sdks:java:core:components - Displays the components produced by >>>> project :sdks:java:core. [incubating] >>>> :sdks:java:core:dependencies - Displays all dependencies declared in >>>> project :sdks:java:core. >>>> :sdks:java:core:dependencyInsight - Displays the insight into a >>>> specific dependency in project :sdks:java:core. >>>> :sdks:java:core:dependencyReport - Generates a report about your >>>> library dependencies. >>>> :sdks:java:core:dependentComponents - Displays the dependent >>>> components of components in project :sdks:java:core. [incubating] >>>> :sdks:java:core:findbugsMain - Run FindBugs analysis for main classes >>>> :sdks:java:core:findbugsTest - Run FindBugs analysis for test classes >>>> :sdks:java:core:generateAvroJava - Generates main Avro Java source >>>> files from schema/protocol definition files. >>>> :sdks:java:core:generateAvroProtocol - Generates main Avro protocol >>>> definition files from IDL files. >>>> :sdks:java:core:generateTestAvroJava - Generates test Avro Java >>>> source files from schema/protocol definition files. >>>> :sdks:java:core:generateTestAvroProtocol - Generates test Avro >>>> protocol definition files from IDL files. >>>> :sdks:java:core:help - Displays a help message. >>>> :sdks:java:core:htmlDependencyReport - Generates an HTML report about >>>> your library dependencies. >>>> :sdks:java:core:install - Installs the archives artifacts into the >>>> local Maven repository. >>>> :sdks:java:core:jacocoTestCoverageVerification - Verifies code >>>> coverage metrics based on specified rules for the test task. >>>> :sdks:java:core:jacocoTestReport - Generates code coverage report for >>>> the test task. >>>> :sdks:java:core:jar - Assembles a jar archive containing the main >>>> classes. >>>> :sdks:java:core:javadoc - Generates Javadoc API documentation for the >>>> main source code. >>>> :sdks:java:core:knows - Do you know who knows? >>>> :sdks:java:core:model - Displays the configuration model of project >>>> :sdks:java:core. [incubating] >>>> :sdks:java:core:packageTests - >>>> :sdks:java:core:processResources - Processes main resources. >>>> :sdks:java:core:processTestResources - Processes test resources. >>>> :sdks:java:core:projectReport - Generates a report about your project. >>>> :sdks:java:core:projects - Displays the sub-projects of project >>>> :sdks:java:core. >>>> :sdks:java:core:properties - Displays the properties of project >>>> :sdks:java:core. >>>> :sdks:java:core:propertyReport - Generates a report about your >>>> properties. >>>> :sdks:java:core:shadowJar - Create a combined JAR of project and >>>> runtime dependencies >>>> :sdks:java:core:shadowTestJar - >>>> :sdks:java:core:spotlessApply - Applies code formatting steps to >>>> sourcecode in-place. >>>> :sdks:java:core:spotlessCheck - Checks that sourcecode satisfies >>>> formatting steps. >>>> :sdks:java:core:spotlessJava - >>>> :sdks:java:core:spotlessJavaApply - >>>> :sdks:java:core:spotlessJavaCheck - >>>> :sdks:java:core:taskReport - Generates a report about your tasks. >>>> :sdks:java:core:tasks - Displays the tasks runnable from project >>>> :sdks:java:core. >>>> :sdks:java:core:testClasses - Assembles test classes. >>>> :sdks:java:core:test - Runs the unit tests. >>>> :sdks:java:core:updateOfflineRepository - >>>> >>>> So if you specify the ':sdks:java:core:test' task then only the tests >>>> run (and any dependent tasks which are not up to date), while >>>> ':sdks:java:core:check' >>>> runs the suite of all checks. If your working on two modules and want to >>>> run the tests for both you really want to specify each explicit end goal >>>> that you want like ':sdks:java:core:test' and ':sdks:java:harness:test'. >>>> Unfortunately incremental builds (https://issues.apache.org/jir >>>> a/browse/BEAM-3253) are unreliable in a few of the modules (like sql >>>> and python) so you may find that you need to specify --rerun-tasks to make >>>> sure that all tasks are rerun even if Gradle thinks they are up to date. >>>> >>>> On Tue, Jan 23, 2018 at 3:38 PM, Romain Manni-Bucau < >>>> rmannibu...@gmail.com> wrote: >>>> >>>>> Can have mischecked gradle setup but I don't think we are here yet, if >>>>> you are not bound in a module and work accross 2 modules and iterate >>>>> between working on both and one, you will likely not bypass the "checks" >>>>> in a satisfying way without a long -x command, is there a magic flag I >>>>> missed? >>>>> Also not sure about the last point and how gradle helps here - it is >>>>> rather the opposite due to the way it loads it model IMHO - so not sure >>>>> what would be the consequence in terms of action(s) but can have missed >>>>> the >>>>> point. >>>>> >>>>> >>>>> >>>>> Romain Manni-Bucau >>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>> <http://rmannibucau.wordpress.com> | Github >>>>> <https://github.com/rmannibucau> | LinkedIn >>>>> <https://www.linkedin.com/in/rmannibucau> >>>>> >>>>> 2018-01-24 0:20 GMT+01:00 Kenneth Knowles <k...@google.com>: >>>>> >>>>>> On Tue, Jan 23, 2018 at 2:51 PM, Romain Manni-Bucau < >>>>>> rmannibu...@gmail.com> wrote: >>>>>> >>>>>> Hmm, did you read it right Kenn? I think the idea was to skip all >>>>>>> validation/sanity checks tasks at once (gradle xxxx -Pfast) instead of >>>>>>> doing it manually (gradle -x findbugs -x checkstyle etc...) >>>>>>> >>>>>> >>>>>> Yes, I read it right. We all want the same thing - not doing a bunch >>>>>> of extra useless unrequested stuff when developing. The concept of >>>>>> skipping >>>>>> is backwards. We don't need configs that skip things, because in a >>>>>> correct >>>>>> dependency-driven build they are already not running. >>>>>> >>>>>> So since I don't want to pretend to know gradle's invocations yet I >>>>>> will call it $TOOL. Here's a collection of imaginary commands: >>>>>> >>>>>> $TOOL :sdks:java:core:unittest # or $TOOL test :sdks:java:core >>>>>> or whatever >>>>>> $TOOL :sdks:java:core:findbugs >>>>>> $TOOL :sdks:java:core:checkstyle >>>>>> $TOOL :sdks:java:core:javadoc >>>>>> >>>>>> None of these causes any of the others to run. Anything else is a >>>>>> bug. The `findbugs` and `test` cause a build of the needed jars and >>>>>> nothing >>>>>> else. >>>>>> >>>>>> Another example: >>>>>> >>>>>> $TOOL :runners:core-java:unittest >>>>>> >>>>>> This builds the model, the core SDK, and the runners-core module, >>>>>> then runs the unit tests of the runners-core module. It does not test SDK >>>>>> core, or run any javadoc, findbugs, or checkstyle on any module. Anything >>>>>> else is a bug. >>>>>> >>>>>> Now, to build a precommit that is easy to reproduce on one line, you >>>>>> could build a compound task >>>>>> >>>>>> $TOOL :sdks:java:core:precommit # runs a selection of targets >>>>>> that we define >>>>>> >>>>>> At this point you might want to skip things from the :verify task >>>>>> here. But really, you probably just want to run the things you are >>>>>> interested in and you don't need custom hooks in the aggregated task. >>>>>> >>>>>> My understanding is that gradle can support all of this, if we are >>>>>> disciplined. Getting to this point is the main/only reason I supported >>>>>> gradle. >>>>>> >>>>>> Kenn >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>>> Kenn >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >> >>>>>>>>> >> diff --git a/examples/java/build.gradle >>>>>>>>> b/examples/java/build.gradle >>>>>>>>> >> index 0fc0b17df..001bd8b38 100644 >>>>>>>>> >> --- a/examples/java/build.gradle >>>>>>>>> >> +++ b/examples/java/build.gradle >>>>>>>>> >> @@ -130,7 +130,7 @@ def preCommitAdditionalFlags = [ >>>>>>>>> >> dataflowStreamingRunner: [ "--streaming=true" ], >>>>>>>>> >> ] >>>>>>>>> >> for (String runner : preCommitRunners) { >>>>>>>>> >> - tasks.create(name: runner + "PreCommit", type: Test) { >>>>>>>>> >> + tasks.create(name: runner + "PreCommit", type: Test, >>>>>>>>> description: "Run tests >>>>>>>>> >> for runner ${runner.replace('Runner', '')}") { >>>>>>>>> >> def preCommitBeamTestPipelineOptions = [ >>>>>>>>> >> "--project=apache-beam-testing", >>>>>>>>> >> "--tempRoot=gs://temp-storage-for-end-to-end-tests", >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> Romain Manni-Bucau >>>>>>>>> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>>>> >> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>>>> >> <http://rmannibucau.wordpress.com> | Github < >>>>>>>>> https://github.com/rmannibucau> | >>>>>>>>> >> LinkedIn <https://www.linkedin.com/in/rmannibucau> >>>>>>>>> >> >>>>>>>>> >> 2018-01-23 8:45 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net >>>>>>>>> >> <mailto:j...@nanthrax.net>>: >>>>>>>>> >> >>>>>>>>> >> Hi Romain, >>>>>>>>> >> >>>>>>>>> >> I think we are pretty close: agree to add some explicit >>>>>>>>> tasks & projects names. >>>>>>>>> >> >>>>>>>>> >> We can add additional tasks like skipAudit, for instance. >>>>>>>>> >> >>>>>>>>> >> As reminder, gradle tasks provides the list of tasks and >>>>>>>>> gradle projects >>>>>>>>> >> provides the list of projects/modules. >>>>>>>>> >> >>>>>>>>> >> Regards >>>>>>>>> >> JB >>>>>>>>> >> >>>>>>>>> >> On 01/23/2018 08:12 AM, Romain Manni-Bucau wrote: >>>>>>>>> >> > Hmm, I have to admit docs dont have my favor cause they >>>>>>>>> are easily outdated and >>>>>>>>> >> > hard to search but you hit a good point. Starting by >>>>>>>>> renaming properly the tasks >>>>>>>>> >> > and maybe writing what is done in build files - since it >>>>>>>>> is code and even "api >>>>>>>>> >> > for dev", it requires as much comments than the main api >>>>>>>>> - can be better to start. >>>>>>>>> >> > >>>>>>>>> >> > Also a big switch flag to bypass checkstyle/findbugs/... >>>>>>>>> can be good while in >>>>>>>>> >> > dev since these phases cost a looot for nothing while you >>>>>>>>> validates your code in >>>>>>>>> >> > runners modules for instance. >>>>>>>>> >> > >>>>>>>>> >> > Le 23 janv. 2018 07:15, "Kenneth Knowles" <k...@google.com >>>>>>>>> <mailto:k...@google.com> >>>>>>>>> >> > <mailto:k...@google.com <mailto:k...@google.com>>> a écrit >>>>>>>>> : >>>>>>>>> >> > >>>>>>>>> >> > On Mon, Jan 22, 2018 at 10:03 PM, Romain Manni-Bucau < >>>>>>>>> rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> >>>>>>>>> >> > <mailto:rmannibu...@gmail.com <mailto: >>>>>>>>> rmannibu...@gmail.com>>> wrote: >>>>>>>>> >> > >>>>>>>>> >> > @Kenneth: why not dropping the doc for a script >>>>>>>>> with comments in the >>>>>>>>> >> > project? A "RUNME.sh" ;). >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> > That's cool, too, but also any shell one liner can be >>>>>>>>> a gradle one >>>>>>>>> >> liner or >>>>>>>>> >> > mvn two/three liner :-). it is just trading one >>>>>>>>> command that you cannot >>>>>>>>> >> > guess easily for a different one that you still can't >>>>>>>>> guess easily. >>>>>>>>> >> > >>>>>>>>> >> > For example, are the SparkRunner ValidatesRunner >>>>>>>>> tests in the >>>>>>>>> >> SparkRunner or >>>>>>>>> >> > the core SDK or a third module that integrates the >>>>>>>>> two? And why would you >>>>>>>>> >> > know that the example ITs are called >>>>>>>>> "sparkRunnerPreCommit"? It >>>>>>>>> >> doesn't even >>>>>>>>> >> > make sense really to have "precommit" or "postcommit" >>>>>>>>> except as aliases to >>>>>>>>> >> > make it easy to repro Jenkins' behavior - they have >>>>>>>>> no other intrinsic >>>>>>>>> >> meaning. >>>>>>>>> >> > >>>>>>>>> >> > So I was proposing a mapping from "full sentence + >>>>>>>>> description" to one >>>>>>>>> >> liner >>>>>>>>> >> > to help people navigate the targets that we set up. >>>>>>>>> Some web page or doc >>>>>>>>> >> > that people can just quickly scan to find out to do >>>>>>>>> common things, easier >>>>>>>>> >> > than groovy or XML. >>>>>>>>> >> > >>>>>>>>> >> > Kenn >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> >>>>>>>>> >> -- >>>>>>>>> >> Jean-Baptiste Onofré >>>>>>>>> >> jbono...@apache.org <mailto:jbono...@apache.org> >>>>>>>>> >> http://blog.nanthrax.net >>>>>>>>> >> Talend - http://www.talend.com >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> > >>>>>>>>> > -- >>>>>>>>> > Jean-Baptiste Onofré >>>>>>>>> > jbono...@apache.org >>>>>>>>> > http://blog.nanthrax.net >>>>>>>>> > Talend - http://www.talend.com >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
