Hmm, I'll try to refine it then next time we work with Ismael but can be a setup issue or a human (bad command) issue at the end. Thanks for the help, will make next iteration way easier probably :)
Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> 2018-01-24 23:05 GMT+01:00 Lukasz Cwik <lc...@google.com>: > Tasks always run any dependencies that are required. So if you ask to run > test it shouldn't run javadoc/checkstyle/... but should compile the code > and compile the code of all dependencies. test should never have a > dependency on checkstyle or javadoc or similar 'check' like tasks as they > shouldn't be needed. > > I set up the gradle build so that everytime you run a command in gradle, > it generates a task dependency tree dot file (look for visteg.dot inside > build/reports). I uploaded this one to imgur[1] for the > ':sdks:java:core:build' task to show what tasks are required. Note that > 'sdks:java:core:test' doesn't depend on checkstyle or spotless. > > 1: https://imgur.com/a/ZvYUX > > On Wed, Jan 24, 2018 at 12:50 PM, Romain Manni-Bucau < > rmannibu...@gmail.com> wrote: > >> Hmm, do I miss something or it only works for iterative runs when trying >> to identify an issue and not for the case you rebuild due to code changes >> (where you would need like 5-6 tasks at least - generate, compile, test, >> ...)? >> >> In case it is unclear: there are 2 needs: direct execution/task -> >> fulfilled and clarified now (just a doc issue I think), fast cycle skipping >> not mandatory tasks like style related ones >> >> >> Romain Manni-Bucau >> @rmannibucau <https://twitter.com/rmannibucau> | Blog >> <https://rmannibucau.metawerx.net/> | Old Blog >> <http://rmannibucau.wordpress.com> | Github >> <https://github.com/rmannibucau> | LinkedIn >> <https://www.linkedin.com/in/rmannibucau> >> >> 2018-01-24 19:50 GMT+01:00 Lukasz Cwik <lc...@google.com>: >> >>> Gradle already has each task explicitly broken out. Kenn is pointing out >>> that you your development use case shouldn't use the './gradlew >>> :sdks:java:core:build' task since it is really an aggregator that >>> represents do everything within that project. This is the current list of >>> tasks available for :sdks:java:core: >>> :sdks:java:core:assemble - Assembles the outputs of this project. >>> :sdks:java:core:build - Assembles and tests this project. >>> :sdks:java:core:buildDependents - Assembles and tests this project and >>> all projects that depend on it. >>> :sdks:java:core:buildEnvironment - Displays all buildscript >>> dependencies declared in project :sdks:java:core. >>> :sdks:java:core:buildNeeded - Assembles and tests this project and all >>> projects it depends on. >>> :sdks:java:core:check - Runs all checks. >>> :sdks:java:core:checkstyleMain - Run Checkstyle analysis for main >>> classes >>> :sdks:java:core:checkstyleTest - Run Checkstyle analysis for test >>> classes >>> :sdks:java:core:classes - Assembles main classes. >>> :sdks:java:core:clean - Deletes the build directory. >>> :sdks:java:core:compileJava - Compiles main Java source. >>> :sdks:java:core:compileTestJava - Compiles test Java source. >>> :sdks:java:core:components - Displays the components produced by >>> project :sdks:java:core. [incubating] >>> :sdks:java:core:dependencies - Displays all dependencies declared in >>> project :sdks:java:core. >>> :sdks:java:core:dependencyInsight - Displays the insight into a >>> specific dependency in project :sdks:java:core. >>> :sdks:java:core:dependencyReport - Generates a report about your >>> library dependencies. >>> :sdks:java:core:dependentComponents - Displays the dependent >>> components of components in project :sdks:java:core. [incubating] >>> :sdks:java:core:findbugsMain - Run FindBugs analysis for main classes >>> :sdks:java:core:findbugsTest - Run FindBugs analysis for test classes >>> :sdks:java:core:generateAvroJava - Generates main Avro Java source >>> files from schema/protocol definition files. >>> :sdks:java:core:generateAvroProtocol - Generates main Avro protocol >>> definition files from IDL files. >>> :sdks:java:core:generateTestAvroJava - Generates test Avro Java source >>> files from schema/protocol definition files. >>> :sdks:java:core:generateTestAvroProtocol - Generates test Avro >>> protocol definition files from IDL files. >>> :sdks:java:core:help - Displays a help message. >>> :sdks:java:core:htmlDependencyReport - Generates an HTML report about >>> your library dependencies. >>> :sdks:java:core:install - Installs the archives artifacts into the >>> local Maven repository. >>> :sdks:java:core:jacocoTestCoverageVerification - Verifies code >>> coverage metrics based on specified rules for the test task. >>> :sdks:java:core:jacocoTestReport - Generates code coverage report for >>> the test task. >>> :sdks:java:core:jar - Assembles a jar archive containing the main >>> classes. >>> :sdks:java:core:javadoc - Generates Javadoc API documentation for the >>> main source code. >>> :sdks:java:core:knows - Do you know who knows? >>> :sdks:java:core:model - Displays the configuration model of project >>> :sdks:java:core. [incubating] >>> :sdks:java:core:packageTests - >>> :sdks:java:core:processResources - Processes main resources. >>> :sdks:java:core:processTestResources - Processes test resources. >>> :sdks:java:core:projectReport - Generates a report about your project. >>> :sdks:java:core:projects - Displays the sub-projects of project >>> :sdks:java:core. >>> :sdks:java:core:properties - Displays the properties of project >>> :sdks:java:core. >>> :sdks:java:core:propertyReport - Generates a report about your >>> properties. >>> :sdks:java:core:shadowJar - Create a combined JAR of project and >>> runtime dependencies >>> :sdks:java:core:shadowTestJar - >>> :sdks:java:core:spotlessApply - Applies code formatting steps to >>> sourcecode in-place. >>> :sdks:java:core:spotlessCheck - Checks that sourcecode satisfies >>> formatting steps. >>> :sdks:java:core:spotlessJava - >>> :sdks:java:core:spotlessJavaApply - >>> :sdks:java:core:spotlessJavaCheck - >>> :sdks:java:core:taskReport - Generates a report about your tasks. >>> :sdks:java:core:tasks - Displays the tasks runnable from project >>> :sdks:java:core. >>> :sdks:java:core:testClasses - Assembles test classes. >>> :sdks:java:core:test - Runs the unit tests. >>> :sdks:java:core:updateOfflineRepository - >>> >>> So if you specify the ':sdks:java:core:test' task then only the tests >>> run (and any dependent tasks which are not up to date), while >>> ':sdks:java:core:check' >>> runs the suite of all checks. If your working on two modules and want to >>> run the tests for both you really want to specify each explicit end goal >>> that you want like ':sdks:java:core:test' and ':sdks:java:harness:test'. >>> Unfortunately incremental builds (https://issues.apache.org/jir >>> a/browse/BEAM-3253) are unreliable in a few of the modules (like sql >>> and python) so you may find that you need to specify --rerun-tasks to make >>> sure that all tasks are rerun even if Gradle thinks they are up to date. >>> >>> On Tue, Jan 23, 2018 at 3:38 PM, Romain Manni-Bucau < >>> rmannibu...@gmail.com> wrote: >>> >>>> Can have mischecked gradle setup but I don't think we are here yet, if >>>> you are not bound in a module and work accross 2 modules and iterate >>>> between working on both and one, you will likely not bypass the "checks" >>>> in a satisfying way without a long -x command, is there a magic flag I >>>> missed? >>>> Also not sure about the last point and how gradle helps here - it is >>>> rather the opposite due to the way it loads it model IMHO - so not sure >>>> what would be the consequence in terms of action(s) but can have missed the >>>> point. >>>> >>>> >>>> >>>> Romain Manni-Bucau >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>> <http://rmannibucau.wordpress.com> | Github >>>> <https://github.com/rmannibucau> | LinkedIn >>>> <https://www.linkedin.com/in/rmannibucau> >>>> >>>> 2018-01-24 0:20 GMT+01:00 Kenneth Knowles <k...@google.com>: >>>> >>>>> On Tue, Jan 23, 2018 at 2:51 PM, Romain Manni-Bucau < >>>>> rmannibu...@gmail.com> wrote: >>>>> >>>>> Hmm, did you read it right Kenn? I think the idea was to skip all >>>>>> validation/sanity checks tasks at once (gradle xxxx -Pfast) instead of >>>>>> doing it manually (gradle -x findbugs -x checkstyle etc...) >>>>>> >>>>> >>>>> Yes, I read it right. We all want the same thing - not doing a bunch >>>>> of extra useless unrequested stuff when developing. The concept of >>>>> skipping >>>>> is backwards. We don't need configs that skip things, because in a correct >>>>> dependency-driven build they are already not running. >>>>> >>>>> So since I don't want to pretend to know gradle's invocations yet I >>>>> will call it $TOOL. Here's a collection of imaginary commands: >>>>> >>>>> $TOOL :sdks:java:core:unittest # or $TOOL test :sdks:java:core or >>>>> whatever >>>>> $TOOL :sdks:java:core:findbugs >>>>> $TOOL :sdks:java:core:checkstyle >>>>> $TOOL :sdks:java:core:javadoc >>>>> >>>>> None of these causes any of the others to run. Anything else is a bug. >>>>> The `findbugs` and `test` cause a build of the needed jars and nothing >>>>> else. >>>>> >>>>> Another example: >>>>> >>>>> $TOOL :runners:core-java:unittest >>>>> >>>>> This builds the model, the core SDK, and the runners-core module, then >>>>> runs the unit tests of the runners-core module. It does not test SDK core, >>>>> or run any javadoc, findbugs, or checkstyle on any module. Anything else >>>>> is >>>>> a bug. >>>>> >>>>> Now, to build a precommit that is easy to reproduce on one line, you >>>>> could build a compound task >>>>> >>>>> $TOOL :sdks:java:core:precommit # runs a selection of targets >>>>> that we define >>>>> >>>>> At this point you might want to skip things from the :verify task >>>>> here. But really, you probably just want to run the things you are >>>>> interested in and you don't need custom hooks in the aggregated task. >>>>> >>>>> My understanding is that gradle can support all of this, if we are >>>>> disciplined. Getting to this point is the main/only reason I supported >>>>> gradle. >>>>> >>>>> Kenn >>>>> >>>>> >>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>>> >>>>>>> Kenn >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >> >>>>>>>> >> diff --git a/examples/java/build.gradle >>>>>>>> b/examples/java/build.gradle >>>>>>>> >> index 0fc0b17df..001bd8b38 100644 >>>>>>>> >> --- a/examples/java/build.gradle >>>>>>>> >> +++ b/examples/java/build.gradle >>>>>>>> >> @@ -130,7 +130,7 @@ def preCommitAdditionalFlags = [ >>>>>>>> >> dataflowStreamingRunner: [ "--streaming=true" ], >>>>>>>> >> ] >>>>>>>> >> for (String runner : preCommitRunners) { >>>>>>>> >> - tasks.create(name: runner + "PreCommit", type: Test) { >>>>>>>> >> + tasks.create(name: runner + "PreCommit", type: Test, >>>>>>>> description: "Run tests >>>>>>>> >> for runner ${runner.replace('Runner', '')}") { >>>>>>>> >> def preCommitBeamTestPipelineOptions = [ >>>>>>>> >> "--project=apache-beam-testing", >>>>>>>> >> "--tempRoot=gs://temp-storage-for-end-to-end-tests", >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> Romain Manni-Bucau >>>>>>>> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>>> >> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>>> >> <http://rmannibucau.wordpress.com> | Github < >>>>>>>> https://github.com/rmannibucau> | >>>>>>>> >> LinkedIn <https://www.linkedin.com/in/rmannibucau> >>>>>>>> >> >>>>>>>> >> 2018-01-23 8:45 GMT+01:00 Jean-Baptiste Onofré <j...@nanthrax.net >>>>>>>> >> <mailto:j...@nanthrax.net>>: >>>>>>>> >> >>>>>>>> >> Hi Romain, >>>>>>>> >> >>>>>>>> >> I think we are pretty close: agree to add some explicit >>>>>>>> tasks & projects names. >>>>>>>> >> >>>>>>>> >> We can add additional tasks like skipAudit, for instance. >>>>>>>> >> >>>>>>>> >> As reminder, gradle tasks provides the list of tasks and >>>>>>>> gradle projects >>>>>>>> >> provides the list of projects/modules. >>>>>>>> >> >>>>>>>> >> Regards >>>>>>>> >> JB >>>>>>>> >> >>>>>>>> >> On 01/23/2018 08:12 AM, Romain Manni-Bucau wrote: >>>>>>>> >> > Hmm, I have to admit docs dont have my favor cause they >>>>>>>> are easily outdated and >>>>>>>> >> > hard to search but you hit a good point. Starting by >>>>>>>> renaming properly the tasks >>>>>>>> >> > and maybe writing what is done in build files - since it >>>>>>>> is code and even "api >>>>>>>> >> > for dev", it requires as much comments than the main api - >>>>>>>> can be better to start. >>>>>>>> >> > >>>>>>>> >> > Also a big switch flag to bypass checkstyle/findbugs/... >>>>>>>> can be good while in >>>>>>>> >> > dev since these phases cost a looot for nothing while you >>>>>>>> validates your code in >>>>>>>> >> > runners modules for instance. >>>>>>>> >> > >>>>>>>> >> > Le 23 janv. 2018 07:15, "Kenneth Knowles" <k...@google.com >>>>>>>> <mailto:k...@google.com> >>>>>>>> >> > <mailto:k...@google.com <mailto:k...@google.com>>> a écrit : >>>>>>>> >> > >>>>>>>> >> > On Mon, Jan 22, 2018 at 10:03 PM, Romain Manni-Bucau < >>>>>>>> rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> >>>>>>>> >> > <mailto:rmannibu...@gmail.com <mailto: >>>>>>>> rmannibu...@gmail.com>>> wrote: >>>>>>>> >> > >>>>>>>> >> > @Kenneth: why not dropping the doc for a script >>>>>>>> with comments in the >>>>>>>> >> > project? A "RUNME.sh" ;). >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > That's cool, too, but also any shell one liner can be >>>>>>>> a gradle one >>>>>>>> >> liner or >>>>>>>> >> > mvn two/three liner :-). it is just trading one >>>>>>>> command that you cannot >>>>>>>> >> > guess easily for a different one that you still can't >>>>>>>> guess easily. >>>>>>>> >> > >>>>>>>> >> > For example, are the SparkRunner ValidatesRunner tests >>>>>>>> in the >>>>>>>> >> SparkRunner or >>>>>>>> >> > the core SDK or a third module that integrates the >>>>>>>> two? And why would you >>>>>>>> >> > know that the example ITs are called >>>>>>>> "sparkRunnerPreCommit"? It >>>>>>>> >> doesn't even >>>>>>>> >> > make sense really to have "precommit" or "postcommit" >>>>>>>> except as aliases to >>>>>>>> >> > make it easy to repro Jenkins' behavior - they have no >>>>>>>> other intrinsic >>>>>>>> >> meaning. >>>>>>>> >> > >>>>>>>> >> > So I was proposing a mapping from "full sentence + >>>>>>>> description" to one >>>>>>>> >> liner >>>>>>>> >> > to help people navigate the targets that we set up. >>>>>>>> Some web page or doc >>>>>>>> >> > that people can just quickly scan to find out to do >>>>>>>> common things, easier >>>>>>>> >> > than groovy or XML. >>>>>>>> >> > >>>>>>>> >> > Kenn >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> >>>>>>>> >> -- >>>>>>>> >> Jean-Baptiste Onofré >>>>>>>> >> jbono...@apache.org <mailto:jbono...@apache.org> >>>>>>>> >> http://blog.nanthrax.net >>>>>>>> >> Talend - http://www.talend.com >>>>>>>> >> >>>>>>>> >> >>>>>>>> > >>>>>>>> > -- >>>>>>>> > Jean-Baptiste Onofré >>>>>>>> > jbono...@apache.org >>>>>>>> > http://blog.nanthrax.net >>>>>>>> > Talend - http://www.talend.com >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
