Tasks always run any dependencies that are required. So if you ask to run test it shouldn't run javadoc/checkstyle/... but should compile the code and compile the code of all dependencies. test should never have a dependency on checkstyle or javadoc or similar 'check' like tasks as they shouldn't be needed.
I set up the gradle build so that everytime you run a command in gradle, it generates a task dependency tree dot file (look for visteg.dot inside build/reports). I uploaded this one to imgur[1] for the ':sdks:java:core:build' task to show what tasks are required. Note that 'sdks:java:core:test' doesn't depend on checkstyle or spotless. 1: https://imgur.com/a/ZvYUX On Wed, Jan 24, 2018 at 12:50 PM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Hmm, do I miss something or it only works for iterative runs when trying > to identify an issue and not for the case you rebuild due to code changes > (where you would need like 5-6 tasks at least - generate, compile, test, > ...)? > > In case it is unclear: there are 2 needs: direct execution/task -> > fulfilled and clarified now (just a doc issue I think), fast cycle skipping > not mandatory tasks like style related ones > > > Romain Manni-Bucau > @rmannibucau <https://twitter.com/rmannibucau> | Blog > <https://rmannibucau.metawerx.net/> | Old Blog > <http://rmannibucau.wordpress.com> | Github > <https://github.com/rmannibucau> | LinkedIn > <https://www.linkedin.com/in/rmannibucau> > > 2018-01-24 19:50 GMT+01:00 Lukasz Cwik <lc...@google.com>: > >> Gradle already has each task explicitly broken out. Kenn is pointing out >> that you your development use case shouldn't use the './gradlew >> :sdks:java:core:build' task since it is really an aggregator that >> represents do everything within that project. This is the current list of >> tasks available for :sdks:java:core: >> :sdks:java:core:assemble - Assembles the outputs of this project. >> :sdks:java:core:build - Assembles and tests this project. >> :sdks:java:core:buildDependents - Assembles and tests this project and >> all projects that depend on it. >> :sdks:java:core:buildEnvironment - Displays all buildscript >> dependencies declared in project :sdks:java:core. >> :sdks:java:core:buildNeeded - Assembles and tests this project and all >> projects it depends on. >> :sdks:java:core:check - Runs all checks. >> :sdks:java:core:checkstyleMain - Run Checkstyle analysis for main >> classes >> :sdks:java:core:checkstyleTest - Run Checkstyle analysis for test >> classes >> :sdks:java:core:classes - Assembles main classes. >> :sdks:java:core:clean - Deletes the build directory. >> :sdks:java:core:compileJava - Compiles main Java source. >> :sdks:java:core:compileTestJava - Compiles test Java source. >> :sdks:java:core:components - Displays the components produced by project >> :sdks:java:core. [incubating] >> :sdks:java:core:dependencies - Displays all dependencies declared in >> project :sdks:java:core. >> :sdks:java:core:dependencyInsight - Displays the insight into a >> specific dependency in project :sdks:java:core. >> :sdks:java:core:dependencyReport - Generates a report about your >> library dependencies. >> :sdks:java:core:dependentComponents - Displays the dependent components >> of components in project :sdks:java:core. [incubating] >> :sdks:java:core:findbugsMain - Run FindBugs analysis for main classes >> :sdks:java:core:findbugsTest - Run FindBugs analysis for test classes >> :sdks:java:core:generateAvroJava - Generates main Avro Java source >> files from schema/protocol definition files. >> :sdks:java:core:generateAvroProtocol - Generates main Avro protocol >> definition files from IDL files. >> :sdks:java:core:generateTestAvroJava - Generates test Avro Java source >> files from schema/protocol definition files. >> :sdks:java:core:generateTestAvroProtocol - Generates test Avro protocol >> definition files from IDL files. >> :sdks:java:core:help - Displays a help message. >> :sdks:java:core:htmlDependencyReport - Generates an HTML report about >> your library dependencies. >> :sdks:java:core:install - Installs the archives artifacts into the local >> Maven repository. >> :sdks:java:core:jacocoTestCoverageVerification - Verifies code coverage >> metrics based on specified rules for the test task. >> :sdks:java:core:jacocoTestReport - Generates code coverage report for >> the test task. >> :sdks:java:core:jar - Assembles a jar archive containing the main >> classes. >> :sdks:java:core:javadoc - Generates Javadoc API documentation for the >> main source code. >> :sdks:java:core:knows - Do you know who knows? >> :sdks:java:core:model - Displays the configuration model of project >> :sdks:java:core. [incubating] >> :sdks:java:core:packageTests - >> :sdks:java:core:processResources - Processes main resources. >> :sdks:java:core:processTestResources - Processes test resources. >> :sdks:java:core:projectReport - Generates a report about your project. >> :sdks:java:core:projects - Displays the sub-projects of project >> :sdks:java:core. >> :sdks:java:core:properties - Displays the properties of project >> :sdks:java:core. >> :sdks:java:core:propertyReport - Generates a report about your >> properties. >> :sdks:java:core:shadowJar - Create a combined JAR of project and runtime >> dependencies >> :sdks:java:core:shadowTestJar - >> :sdks:java:core:spotlessApply - Applies code formatting steps to >> sourcecode in-place. >> :sdks:java:core:spotlessCheck - Checks that sourcecode satisfies >> formatting steps. >> :sdks:java:core:spotlessJava - >> :sdks:java:core:spotlessJavaApply - >> :sdks:java:core:spotlessJavaCheck - >> :sdks:java:core:taskReport - Generates a report about your tasks. >> :sdks:java:core:tasks - Displays the tasks runnable from project >> :sdks:java:core. >> :sdks:java:core:testClasses - Assembles test classes. >> :sdks:java:core:test - Runs the unit tests. >> :sdks:java:core:updateOfflineRepository - >> >> So if you specify the ':sdks:java:core:test' task then only the tests >> run (and any dependent tasks which are not up to date), while >> ':sdks:java:core:check' >> runs the suite of all checks. If your working on two modules and want to >> run the tests for both you really want to specify each explicit end goal >> that you want like ':sdks:java:core:test' and ':sdks:java:harness:test'. >> Unfortunately incremental builds (https://issues.apache.org/jir >> a/browse/BEAM-3253) are unreliable in a few of the modules (like sql and >> python) so you may find that you need to specify --rerun-tasks to make sure >> that all tasks are rerun even if Gradle thinks they are up to date. >> >> On Tue, Jan 23, 2018 at 3:38 PM, Romain Manni-Bucau < >> rmannibu...@gmail.com> wrote: >> >>> Can have mischecked gradle setup but I don't think we are here yet, if >>> you are not bound in a module and work accross 2 modules and iterate >>> between working on both and one, you will likely not bypass the "checks" >>> in a satisfying way without a long -x command, is there a magic flag I >>> missed? >>> Also not sure about the last point and how gradle helps here - it is >>> rather the opposite due to the way it loads it model IMHO - so not sure >>> what would be the consequence in terms of action(s) but can have missed the >>> point. >>> >>> >>> >>> Romain Manni-Bucau >>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>> <https://rmannibucau.metawerx.net/> | Old Blog >>> <http://rmannibucau.wordpress.com> | Github >>> <https://github.com/rmannibucau> | LinkedIn >>> <https://www.linkedin.com/in/rmannibucau> >>> >>> 2018-01-24 0:20 GMT+01:00 Kenneth Knowles <k...@google.com>: >>> >>>> On Tue, Jan 23, 2018 at 2:51 PM, Romain Manni-Bucau < >>>> rmannibu...@gmail.com> wrote: >>>> >>>> Hmm, did you read it right Kenn? I think the idea was to skip all >>>>> validation/sanity checks tasks at once (gradle xxxx -Pfast) instead of >>>>> doing it manually (gradle -x findbugs -x checkstyle etc...) >>>>> >>>> >>>> Yes, I read it right. We all want the same thing - not doing a bunch of >>>> extra useless unrequested stuff when developing. The concept of skipping is >>>> backwards. We don't need configs that skip things, because in a correct >>>> dependency-driven build they are already not running. >>>> >>>> So since I don't want to pretend to know gradle's invocations yet I >>>> will call it $TOOL. Here's a collection of imaginary commands: >>>> >>>> $TOOL :sdks:java:core:unittest # or $TOOL test :sdks:java:core or >>>> whatever >>>> $TOOL :sdks:java:core:findbugs >>>> $TOOL :sdks:java:core:checkstyle >>>> $TOOL :sdks:java:core:javadoc >>>> >>>> None of these causes any of the others to run. Anything else is a bug. >>>> The `findbugs` and `test` cause a build of the needed jars and nothing >>>> else. >>>> >>>> Another example: >>>> >>>> $TOOL :runners:core-java:unittest >>>> >>>> This builds the model, the core SDK, and the runners-core module, then >>>> runs the unit tests of the runners-core module. It does not test SDK core, >>>> or run any javadoc, findbugs, or checkstyle on any module. Anything else is >>>> a bug. >>>> >>>> Now, to build a precommit that is easy to reproduce on one line, you >>>> could build a compound task >>>> >>>> $TOOL :sdks:java:core:precommit # runs a selection of targets that >>>> we define >>>> >>>> At this point you might want to skip things from the :verify task here. >>>> But really, you probably just want to run the things you are interested in >>>> and you don't need custom hooks in the aggregated task. >>>> >>>> My understanding is that gradle can support all of this, if we are >>>> disciplined. Getting to this point is the main/only reason I supported >>>> gradle. >>>> >>>> Kenn >>>> >>>> >>>> >>>>> >>>>> >>>> >>>> >>>>> >>>>>> Kenn >>>>>> >>>>>> >>>>>> >>>>>>> >> >>>>>>> >> diff --git a/examples/java/build.gradle >>>>>>> b/examples/java/build.gradle >>>>>>> >> index 0fc0b17df..001bd8b38 100644 >>>>>>> >> --- a/examples/java/build.gradle >>>>>>> >> +++ b/examples/java/build.gradle >>>>>>> >> @@ -130,7 +130,7 @@ def preCommitAdditionalFlags = [ >>>>>>> >> dataflowStreamingRunner: [ "--streaming=true" ], >>>>>>> >> ] >>>>>>> >> for (String runner : preCommitRunners) { >>>>>>> >> - tasks.create(name: runner + "PreCommit", type: Test) { >>>>>>> >> + tasks.create(name: runner + "PreCommit", type: Test, >>>>>>> description: "Run tests >>>>>>> >> for runner ${runner.replace('Runner', '')}") { >>>>>>> >> def preCommitBeamTestPipelineOptions = [ >>>>>>> >> "--project=apache-beam-testing", >>>>>>> >> "--tempRoot=gs://temp-storage-for-end-to-end-tests", >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> Romain Manni-Bucau >>>>>>> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>> >> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>> >> <http://rmannibucau.wordpress.com> | Github < >>>>>>> https://github.com/rmannibucau> | >>>>>>> >> LinkedIn <https://www.linkedin.com/in/rmannibucau> >>>>>>> >> >>>>>>> >> 2018-01-23 8:45 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net >>>>>>> >> <mailto:j...@nanthrax.net>>: >>>>>>> >> >>>>>>> >> Hi Romain, >>>>>>> >> >>>>>>> >> I think we are pretty close: agree to add some explicit tasks >>>>>>> & projects names. >>>>>>> >> >>>>>>> >> We can add additional tasks like skipAudit, for instance. >>>>>>> >> >>>>>>> >> As reminder, gradle tasks provides the list of tasks and >>>>>>> gradle projects >>>>>>> >> provides the list of projects/modules. >>>>>>> >> >>>>>>> >> Regards >>>>>>> >> JB >>>>>>> >> >>>>>>> >> On 01/23/2018 08:12 AM, Romain Manni-Bucau wrote: >>>>>>> >> > Hmm, I have to admit docs dont have my favor cause they are >>>>>>> easily outdated and >>>>>>> >> > hard to search but you hit a good point. Starting by >>>>>>> renaming properly the tasks >>>>>>> >> > and maybe writing what is done in build files - since it is >>>>>>> code and even "api >>>>>>> >> > for dev", it requires as much comments than the main api - >>>>>>> can be better to start. >>>>>>> >> > >>>>>>> >> > Also a big switch flag to bypass checkstyle/findbugs/... >>>>>>> can be good while in >>>>>>> >> > dev since these phases cost a looot for nothing while you >>>>>>> validates your code in >>>>>>> >> > runners modules for instance. >>>>>>> >> > >>>>>>> >> > Le 23 janv. 2018 07:15, "Kenneth Knowles" <k...@google.com >>>>>>> <mailto:k...@google.com> >>>>>>> >> > <mailto:k...@google.com <mailto:k...@google.com>>> a écrit : >>>>>>> >> > >>>>>>> >> > On Mon, Jan 22, 2018 at 10:03 PM, Romain Manni-Bucau < >>>>>>> rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> >>>>>>> >> > <mailto:rmannibu...@gmail.com <mailto: >>>>>>> rmannibu...@gmail.com>>> wrote: >>>>>>> >> > >>>>>>> >> > @Kenneth: why not dropping the doc for a script >>>>>>> with comments in the >>>>>>> >> > project? A "RUNME.sh" ;). >>>>>>> >> > >>>>>>> >> > >>>>>>> >> > That's cool, too, but also any shell one liner can be a >>>>>>> gradle one >>>>>>> >> liner or >>>>>>> >> > mvn two/three liner :-). it is just trading one command >>>>>>> that you cannot >>>>>>> >> > guess easily for a different one that you still can't >>>>>>> guess easily. >>>>>>> >> > >>>>>>> >> > For example, are the SparkRunner ValidatesRunner tests >>>>>>> in the >>>>>>> >> SparkRunner or >>>>>>> >> > the core SDK or a third module that integrates the two? >>>>>>> And why would you >>>>>>> >> > know that the example ITs are called >>>>>>> "sparkRunnerPreCommit"? It >>>>>>> >> doesn't even >>>>>>> >> > make sense really to have "precommit" or "postcommit" >>>>>>> except as aliases to >>>>>>> >> > make it easy to repro Jenkins' behavior - they have no >>>>>>> other intrinsic >>>>>>> >> meaning. >>>>>>> >> > >>>>>>> >> > So I was proposing a mapping from "full sentence + >>>>>>> description" to one >>>>>>> >> liner >>>>>>> >> > to help people navigate the targets that we set up. >>>>>>> Some web page or doc >>>>>>> >> > that people can just quickly scan to find out to do >>>>>>> common things, easier >>>>>>> >> > than groovy or XML. >>>>>>> >> > >>>>>>> >> > Kenn >>>>>>> >> > >>>>>>> >> > >>>>>>> >> >>>>>>> >> -- >>>>>>> >> Jean-Baptiste Onofré >>>>>>> >> jbono...@apache.org <mailto:jbono...@apache.org> >>>>>>> >> http://blog.nanthrax.net >>>>>>> >> Talend - http://www.talend.com >>>>>>> >> >>>>>>> >> >>>>>>> > >>>>>>> > -- >>>>>>> > Jean-Baptiste Onofré >>>>>>> > jbono...@apache.org >>>>>>> > http://blog.nanthrax.net >>>>>>> > Talend - http://www.talend.com >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
