Hmm, do I miss something or it only works for iterative runs when trying to identify an issue and not for the case you rebuild due to code changes (where you would need like 5-6 tasks at least - generate, compile, test, ...)?
In case it is unclear: there are 2 needs: direct execution/task -> fulfilled and clarified now (just a doc issue I think), fast cycle skipping not mandatory tasks like style related ones Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> 2018-01-24 19:50 GMT+01:00 Lukasz Cwik <lc...@google.com>: > Gradle already has each task explicitly broken out. Kenn is pointing out > that you your development use case shouldn't use the './gradlew > :sdks:java:core:build' task since it is really an aggregator that > represents do everything within that project. This is the current list of > tasks available for :sdks:java:core: > :sdks:java:core:assemble - Assembles the outputs of this project. > :sdks:java:core:build - Assembles and tests this project. > :sdks:java:core:buildDependents - Assembles and tests this project and > all projects that depend on it. > :sdks:java:core:buildEnvironment - Displays all buildscript dependencies > declared in project :sdks:java:core. > :sdks:java:core:buildNeeded - Assembles and tests this project and all > projects it depends on. > :sdks:java:core:check - Runs all checks. > :sdks:java:core:checkstyleMain - Run Checkstyle analysis for main classes > :sdks:java:core:checkstyleTest - Run Checkstyle analysis for test classes > :sdks:java:core:classes - Assembles main classes. > :sdks:java:core:clean - Deletes the build directory. > :sdks:java:core:compileJava - Compiles main Java source. > :sdks:java:core:compileTestJava - Compiles test Java source. > :sdks:java:core:components - Displays the components produced by project > :sdks:java:core. [incubating] > :sdks:java:core:dependencies - Displays all dependencies declared in > project :sdks:java:core. > :sdks:java:core:dependencyInsight - Displays the insight into a specific > dependency in project :sdks:java:core. > :sdks:java:core:dependencyReport - Generates a report about your library > dependencies. > :sdks:java:core:dependentComponents - Displays the dependent components > of components in project :sdks:java:core. [incubating] > :sdks:java:core:findbugsMain - Run FindBugs analysis for main classes > :sdks:java:core:findbugsTest - Run FindBugs analysis for test classes > :sdks:java:core:generateAvroJava - Generates main Avro Java source files > from schema/protocol definition files. > :sdks:java:core:generateAvroProtocol - Generates main Avro protocol > definition files from IDL files. > :sdks:java:core:generateTestAvroJava - Generates test Avro Java source > files from schema/protocol definition files. > :sdks:java:core:generateTestAvroProtocol - Generates test Avro protocol > definition files from IDL files. > :sdks:java:core:help - Displays a help message. > :sdks:java:core:htmlDependencyReport - Generates an HTML report about > your library dependencies. > :sdks:java:core:install - Installs the archives artifacts into the local > Maven repository. > :sdks:java:core:jacocoTestCoverageVerification - Verifies code coverage > metrics based on specified rules for the test task. > :sdks:java:core:jacocoTestReport - Generates code coverage report for > the test task. > :sdks:java:core:jar - Assembles a jar archive containing the main classes. > :sdks:java:core:javadoc - Generates Javadoc API documentation for the > main source code. > :sdks:java:core:knows - Do you know who knows? > :sdks:java:core:model - Displays the configuration model of project > :sdks:java:core. [incubating] > :sdks:java:core:packageTests - > :sdks:java:core:processResources - Processes main resources. > :sdks:java:core:processTestResources - Processes test resources. > :sdks:java:core:projectReport - Generates a report about your project. > :sdks:java:core:projects - Displays the sub-projects of project > :sdks:java:core. > :sdks:java:core:properties - Displays the properties of project > :sdks:java:core. > :sdks:java:core:propertyReport - Generates a report about your > properties. > :sdks:java:core:shadowJar - Create a combined JAR of project and runtime > dependencies > :sdks:java:core:shadowTestJar - > :sdks:java:core:spotlessApply - Applies code formatting steps to > sourcecode in-place. > :sdks:java:core:spotlessCheck - Checks that sourcecode satisfies > formatting steps. > :sdks:java:core:spotlessJava - > :sdks:java:core:spotlessJavaApply - > :sdks:java:core:spotlessJavaCheck - > :sdks:java:core:taskReport - Generates a report about your tasks. > :sdks:java:core:tasks - Displays the tasks runnable from project > :sdks:java:core. > :sdks:java:core:testClasses - Assembles test classes. > :sdks:java:core:test - Runs the unit tests. > :sdks:java:core:updateOfflineRepository - > > So if you specify the ':sdks:java:core:test' task then only the tests run > (and any dependent tasks which are not up to date), while > ':sdks:java:core:check' > runs the suite of all checks. If your working on two modules and want to > run the tests for both you really want to specify each explicit end goal > that you want like ':sdks:java:core:test' and ':sdks:java:harness:test'. > Unfortunately incremental builds (https://issues.apache.org/ > jira/browse/BEAM-3253) are unreliable in a few of the modules (like sql > and python) so you may find that you need to specify --rerun-tasks to make > sure that all tasks are rerun even if Gradle thinks they are up to date. > > On Tue, Jan 23, 2018 at 3:38 PM, Romain Manni-Bucau <rmannibu...@gmail.com > > wrote: > >> Can have mischecked gradle setup but I don't think we are here yet, if >> you are not bound in a module and work accross 2 modules and iterate >> between working on both and one, you will likely not bypass the "checks" >> in a satisfying way without a long -x command, is there a magic flag I >> missed? >> Also not sure about the last point and how gradle helps here - it is >> rather the opposite due to the way it loads it model IMHO - so not sure >> what would be the consequence in terms of action(s) but can have missed the >> point. >> >> >> >> Romain Manni-Bucau >> @rmannibucau <https://twitter.com/rmannibucau> | Blog >> <https://rmannibucau.metawerx.net/> | Old Blog >> <http://rmannibucau.wordpress.com> | Github >> <https://github.com/rmannibucau> | LinkedIn >> <https://www.linkedin.com/in/rmannibucau> >> >> 2018-01-24 0:20 GMT+01:00 Kenneth Knowles <k...@google.com>: >> >>> On Tue, Jan 23, 2018 at 2:51 PM, Romain Manni-Bucau < >>> rmannibu...@gmail.com> wrote: >>> >>> Hmm, did you read it right Kenn? I think the idea was to skip all >>>> validation/sanity checks tasks at once (gradle xxxx -Pfast) instead of >>>> doing it manually (gradle -x findbugs -x checkstyle etc...) >>>> >>> >>> Yes, I read it right. We all want the same thing - not doing a bunch of >>> extra useless unrequested stuff when developing. The concept of skipping is >>> backwards. We don't need configs that skip things, because in a correct >>> dependency-driven build they are already not running. >>> >>> So since I don't want to pretend to know gradle's invocations yet I will >>> call it $TOOL. Here's a collection of imaginary commands: >>> >>> $TOOL :sdks:java:core:unittest # or $TOOL test :sdks:java:core or >>> whatever >>> $TOOL :sdks:java:core:findbugs >>> $TOOL :sdks:java:core:checkstyle >>> $TOOL :sdks:java:core:javadoc >>> >>> None of these causes any of the others to run. Anything else is a bug. >>> The `findbugs` and `test` cause a build of the needed jars and nothing else. >>> >>> Another example: >>> >>> $TOOL :runners:core-java:unittest >>> >>> This builds the model, the core SDK, and the runners-core module, then >>> runs the unit tests of the runners-core module. It does not test SDK core, >>> or run any javadoc, findbugs, or checkstyle on any module. Anything else is >>> a bug. >>> >>> Now, to build a precommit that is easy to reproduce on one line, you >>> could build a compound task >>> >>> $TOOL :sdks:java:core:precommit # runs a selection of targets that >>> we define >>> >>> At this point you might want to skip things from the :verify task here. >>> But really, you probably just want to run the things you are interested in >>> and you don't need custom hooks in the aggregated task. >>> >>> My understanding is that gradle can support all of this, if we are >>> disciplined. Getting to this point is the main/only reason I supported >>> gradle. >>> >>> Kenn >>> >>> >>> >>>> >>>> >>> >>> >>>> >>>>> Kenn >>>>> >>>>> >>>>> >>>>>> >> >>>>>> >> diff --git a/examples/java/build.gradle >>>>>> b/examples/java/build.gradle >>>>>> >> index 0fc0b17df..001bd8b38 100644 >>>>>> >> --- a/examples/java/build.gradle >>>>>> >> +++ b/examples/java/build.gradle >>>>>> >> @@ -130,7 +130,7 @@ def preCommitAdditionalFlags = [ >>>>>> >> dataflowStreamingRunner: [ "--streaming=true" ], >>>>>> >> ] >>>>>> >> for (String runner : preCommitRunners) { >>>>>> >> - tasks.create(name: runner + "PreCommit", type: Test) { >>>>>> >> + tasks.create(name: runner + "PreCommit", type: Test, >>>>>> description: "Run tests >>>>>> >> for runner ${runner.replace('Runner', '')}") { >>>>>> >> def preCommitBeamTestPipelineOptions = [ >>>>>> >> "--project=apache-beam-testing", >>>>>> >> "--tempRoot=gs://temp-storage-for-end-to-end-tests", >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> Romain Manni-Bucau >>>>>> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>> >> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>> >> <http://rmannibucau.wordpress.com> | Github < >>>>>> https://github.com/rmannibucau> | >>>>>> >> LinkedIn <https://www.linkedin.com/in/rmannibucau> >>>>>> >> >>>>>> >> 2018-01-23 8:45 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net >>>>>> >> <mailto:j...@nanthrax.net>>: >>>>>> >> >>>>>> >> Hi Romain, >>>>>> >> >>>>>> >> I think we are pretty close: agree to add some explicit tasks >>>>>> & projects names. >>>>>> >> >>>>>> >> We can add additional tasks like skipAudit, for instance. >>>>>> >> >>>>>> >> As reminder, gradle tasks provides the list of tasks and >>>>>> gradle projects >>>>>> >> provides the list of projects/modules. >>>>>> >> >>>>>> >> Regards >>>>>> >> JB >>>>>> >> >>>>>> >> On 01/23/2018 08:12 AM, Romain Manni-Bucau wrote: >>>>>> >> > Hmm, I have to admit docs dont have my favor cause they are >>>>>> easily outdated and >>>>>> >> > hard to search but you hit a good point. Starting by >>>>>> renaming properly the tasks >>>>>> >> > and maybe writing what is done in build files - since it is >>>>>> code and even "api >>>>>> >> > for dev", it requires as much comments than the main api - >>>>>> can be better to start. >>>>>> >> > >>>>>> >> > Also a big switch flag to bypass checkstyle/findbugs/... can >>>>>> be good while in >>>>>> >> > dev since these phases cost a looot for nothing while you >>>>>> validates your code in >>>>>> >> > runners modules for instance. >>>>>> >> > >>>>>> >> > Le 23 janv. 2018 07:15, "Kenneth Knowles" <k...@google.com >>>>>> <mailto:k...@google.com> >>>>>> >> > <mailto:k...@google.com <mailto:k...@google.com>>> a écrit : >>>>>> >> > >>>>>> >> > On Mon, Jan 22, 2018 at 10:03 PM, Romain Manni-Bucau < >>>>>> rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> >>>>>> >> > <mailto:rmannibu...@gmail.com <mailto: >>>>>> rmannibu...@gmail.com>>> wrote: >>>>>> >> > >>>>>> >> > @Kenneth: why not dropping the doc for a script with >>>>>> comments in the >>>>>> >> > project? A "RUNME.sh" ;). >>>>>> >> > >>>>>> >> > >>>>>> >> > That's cool, too, but also any shell one liner can be a >>>>>> gradle one >>>>>> >> liner or >>>>>> >> > mvn two/three liner :-). it is just trading one command >>>>>> that you cannot >>>>>> >> > guess easily for a different one that you still can't >>>>>> guess easily. >>>>>> >> > >>>>>> >> > For example, are the SparkRunner ValidatesRunner tests >>>>>> in the >>>>>> >> SparkRunner or >>>>>> >> > the core SDK or a third module that integrates the two? >>>>>> And why would you >>>>>> >> > know that the example ITs are called >>>>>> "sparkRunnerPreCommit"? It >>>>>> >> doesn't even >>>>>> >> > make sense really to have "precommit" or "postcommit" >>>>>> except as aliases to >>>>>> >> > make it easy to repro Jenkins' behavior - they have no >>>>>> other intrinsic >>>>>> >> meaning. >>>>>> >> > >>>>>> >> > So I was proposing a mapping from "full sentence + >>>>>> description" to one >>>>>> >> liner >>>>>> >> > to help people navigate the targets that we set up. Some >>>>>> web page or doc >>>>>> >> > that people can just quickly scan to find out to do >>>>>> common things, easier >>>>>> >> > than groovy or XML. >>>>>> >> > >>>>>> >> > Kenn >>>>>> >> > >>>>>> >> > >>>>>> >> >>>>>> >> -- >>>>>> >> Jean-Baptiste Onofré >>>>>> >> jbono...@apache.org <mailto:jbono...@apache.org> >>>>>> >> http://blog.nanthrax.net >>>>>> >> Talend - http://www.talend.com >>>>>> >> >>>>>> >> >>>>>> > >>>>>> > -- >>>>>> > Jean-Baptiste Onofré >>>>>> > jbono...@apache.org >>>>>> > http://blog.nanthrax.net >>>>>> > Talend - http://www.talend.com >>>>>> >>>>> >>>>> >>>> >>> >> >
