Well it is more about consistency and reliability than speed here. Comoilabtion result is just corrupted :(
Le 25 janv. 2018 20:33, "Lukasz Cwik" <lc...@google.com> a écrit : > You can only get incremental support at the build system level, not at the > individual tool level like javac. The task the represents compilation would > need to be broken up into smaller tasks with smaller source sets to speed > up compilation of really large modules. > > On Wed, Jan 24, 2018 at 11:12 PM, Romain Manni-Bucau < > rmannibu...@gmail.com> wrote: > >> One pitfall I noted with javac wrappers is that if you dont clean - and >> loose javac incremental support then deleted classes stay here and are >> considered in the output. Common example is a nested class which has been >> deleted leading to a corrupted enclosing class or a test which is ran but >> deleted from java/. Any known way to protect us from it and keep the >> uncremental support for big modules? >> >> Le 25 janv. 2018 01:22, "Lukasz Cwik" <lc...@google.com> a écrit : >> >>> Dependency driven works, incremental works for most java modules. >>> I use incremental almost all the time and just do one validation pass at >>> the end before opening the PR where I use '--rerun-tasks' to be sure. >>> Allows me to iterate on a task in seconds. >>> >>> On Wed, Jan 24, 2018 at 4:07 PM, Kenneth Knowles <k...@google.com> wrote: >>> >>>> These are two different things: dependency-driven build (which works) >>>> and incremental build (which seems not to, at least right now?). >>>> >>>> On Wed, Jan 24, 2018 at 2:24 PM, Romain Manni-Bucau < >>>> rmannibu...@gmail.com> wrote: >>>> >>>>> Hmm, I'll try to refine it then next time we work with Ismael but can >>>>> be a setup issue or a human (bad command) issue at the end. Thanks for the >>>>> help, will make next iteration way easier probably :) >>>>> >>>>> >>>>> Romain Manni-Bucau >>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>> <http://rmannibucau.wordpress.com> | Github >>>>> <https://github.com/rmannibucau> | LinkedIn >>>>> <https://www.linkedin.com/in/rmannibucau> >>>>> >>>>> 2018-01-24 23:05 GMT+01:00 Lukasz Cwik <lc...@google.com>: >>>>> >>>>>> Tasks always run any dependencies that are required. So if you ask to >>>>>> run test it shouldn't run javadoc/checkstyle/... but should compile the >>>>>> code and compile the code of all dependencies. test should never have a >>>>>> dependency on checkstyle or javadoc or similar 'check' like tasks as they >>>>>> shouldn't be needed. >>>>>> >>>>>> I set up the gradle build so that everytime you run a command in >>>>>> gradle, it generates a task dependency tree dot file (look for visteg.dot >>>>>> inside build/reports). I uploaded this one to imgur[1] for the >>>>>> ':sdks:java:core:build' task to show what tasks are required. Note that >>>>>> 'sdks:java:core:test' doesn't depend on checkstyle or spotless. >>>>>> >>>>>> 1: https://imgur.com/a/ZvYUX >>>>>> >>>>>> On Wed, Jan 24, 2018 at 12:50 PM, Romain Manni-Bucau < >>>>>> rmannibu...@gmail.com> wrote: >>>>>> >>>>>>> Hmm, do I miss something or it only works for iterative runs when >>>>>>> trying to identify an issue and not for the case you rebuild due to code >>>>>>> changes (where you would need like 5-6 tasks at least - generate, >>>>>>> compile, >>>>>>> test, ...)? >>>>>>> >>>>>>> In case it is unclear: there are 2 needs: direct execution/task -> >>>>>>> fulfilled and clarified now (just a doc issue I think), fast cycle >>>>>>> skipping >>>>>>> not mandatory tasks like style related ones >>>>>>> >>>>>>> >>>>>>> Romain Manni-Bucau >>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>>> <https://www.linkedin.com/in/rmannibucau> >>>>>>> >>>>>>> 2018-01-24 19:50 GMT+01:00 Lukasz Cwik <lc...@google.com>: >>>>>>> >>>>>>>> Gradle already has each task explicitly broken out. Kenn is >>>>>>>> pointing out that you your development use case shouldn't use the >>>>>>>> './gradlew :sdks:java:core:build' task since it is really an aggregator >>>>>>>> that represents do everything within that project. This is the current >>>>>>>> list >>>>>>>> of tasks available for :sdks:java:core: >>>>>>>> :sdks:java:core:assemble - Assembles the outputs of this project. >>>>>>>> :sdks:java:core:build - Assembles and tests this project. >>>>>>>> :sdks:java:core:buildDependents - Assembles and tests this >>>>>>>> project and all projects that depend on it. >>>>>>>> :sdks:java:core:buildEnvironment - Displays all buildscript >>>>>>>> dependencies declared in project :sdks:java:core. >>>>>>>> :sdks:java:core:buildNeeded - Assembles and tests this project and >>>>>>>> all projects it depends on. >>>>>>>> :sdks:java:core:check - Runs all checks. >>>>>>>> :sdks:java:core:checkstyleMain - Run Checkstyle analysis for main >>>>>>>> classes >>>>>>>> :sdks:java:core:checkstyleTest - Run Checkstyle analysis for test >>>>>>>> classes >>>>>>>> :sdks:java:core:classes - Assembles main classes. >>>>>>>> :sdks:java:core:clean - Deletes the build directory. >>>>>>>> :sdks:java:core:compileJava - Compiles main Java source. >>>>>>>> :sdks:java:core:compileTestJava - Compiles test Java source. >>>>>>>> :sdks:java:core:components - Displays the components produced by >>>>>>>> project :sdks:java:core. [incubating] >>>>>>>> :sdks:java:core:dependencies - Displays all dependencies declared >>>>>>>> in project :sdks:java:core. >>>>>>>> :sdks:java:core:dependencyInsight - Displays the insight into a >>>>>>>> specific dependency in project :sdks:java:core. >>>>>>>> :sdks:java:core:dependencyReport - Generates a report about your >>>>>>>> library dependencies. >>>>>>>> :sdks:java:core:dependentComponents - Displays the dependent >>>>>>>> components of components in project :sdks:java:core. [incubating] >>>>>>>> :sdks:java:core:findbugsMain - Run FindBugs analysis for main >>>>>>>> classes >>>>>>>> :sdks:java:core:findbugsTest - Run FindBugs analysis for test >>>>>>>> classes >>>>>>>> :sdks:java:core:generateAvroJava - Generates main Avro Java >>>>>>>> source files from schema/protocol definition files. >>>>>>>> :sdks:java:core:generateAvroProtocol - Generates main Avro >>>>>>>> protocol definition files from IDL files. >>>>>>>> :sdks:java:core:generateTestAvroJava - Generates test Avro Java >>>>>>>> source files from schema/protocol definition files. >>>>>>>> :sdks:java:core:generateTestAvroProtocol - Generates test Avro >>>>>>>> protocol definition files from IDL files. >>>>>>>> :sdks:java:core:help - Displays a help message. >>>>>>>> :sdks:java:core:htmlDependencyReport - Generates an HTML report >>>>>>>> about your library dependencies. >>>>>>>> :sdks:java:core:install - Installs the archives artifacts into the >>>>>>>> local Maven repository. >>>>>>>> :sdks:java:core:jacocoTestCoverageVerification - Verifies code >>>>>>>> coverage metrics based on specified rules for the test task. >>>>>>>> :sdks:java:core:jacocoTestReport - Generates code coverage report >>>>>>>> for the test task. >>>>>>>> :sdks:java:core:jar - Assembles a jar archive containing the main >>>>>>>> classes. >>>>>>>> :sdks:java:core:javadoc - Generates Javadoc API documentation for >>>>>>>> the main source code. >>>>>>>> :sdks:java:core:knows - Do you know who knows? >>>>>>>> :sdks:java:core:model - Displays the configuration model of >>>>>>>> project :sdks:java:core. [incubating] >>>>>>>> :sdks:java:core:packageTests - >>>>>>>> :sdks:java:core:processResources - Processes main resources. >>>>>>>> :sdks:java:core:processTestResources - Processes test resources. >>>>>>>> :sdks:java:core:projectReport - Generates a report about your >>>>>>>> project. >>>>>>>> :sdks:java:core:projects - Displays the sub-projects of project >>>>>>>> :sdks:java:core. >>>>>>>> :sdks:java:core:properties - Displays the properties of project >>>>>>>> :sdks:java:core. >>>>>>>> :sdks:java:core:propertyReport - Generates a report about your >>>>>>>> properties. >>>>>>>> :sdks:java:core:shadowJar - Create a combined JAR of project and >>>>>>>> runtime dependencies >>>>>>>> :sdks:java:core:shadowTestJar - >>>>>>>> :sdks:java:core:spotlessApply - Applies code formatting steps to >>>>>>>> sourcecode in-place. >>>>>>>> :sdks:java:core:spotlessCheck - Checks that sourcecode satisfies >>>>>>>> formatting steps. >>>>>>>> :sdks:java:core:spotlessJava - >>>>>>>> :sdks:java:core:spotlessJavaApply - >>>>>>>> :sdks:java:core:spotlessJavaCheck - >>>>>>>> :sdks:java:core:taskReport - Generates a report about your tasks. >>>>>>>> :sdks:java:core:tasks - Displays the tasks runnable from project >>>>>>>> :sdks:java:core. >>>>>>>> :sdks:java:core:testClasses - Assembles test classes. >>>>>>>> :sdks:java:core:test - Runs the unit tests. >>>>>>>> :sdks:java:core:updateOfflineRepository - >>>>>>>> >>>>>>>> So if you specify the ':sdks:java:core:test' task then only the >>>>>>>> tests run (and any dependent tasks which are not up to date), while >>>>>>>> ':sdks:java:core:check' >>>>>>>> runs the suite of all checks. If your working on two modules and want >>>>>>>> to >>>>>>>> run the tests for both you really want to specify each explicit end >>>>>>>> goal >>>>>>>> that you want like ':sdks:java:core:test' and >>>>>>>> ':sdks:java:harness:test'. >>>>>>>> Unfortunately incremental builds (https://issues.apache.org/jir >>>>>>>> a/browse/BEAM-3253) are unreliable in a few of the modules (like >>>>>>>> sql and python) so you may find that you need to specify --rerun-tasks >>>>>>>> to >>>>>>>> make sure that all tasks are rerun even if Gradle thinks they are up to >>>>>>>> date. >>>>>>>> >>>>>>>> On Tue, Jan 23, 2018 at 3:38 PM, Romain Manni-Bucau < >>>>>>>> rmannibu...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Can have mischecked gradle setup but I don't think we are here >>>>>>>>> yet, if you are not bound in a module and work accross 2 modules and >>>>>>>>> iterate between working on both and one, you will likely not bypass >>>>>>>>> the >>>>>>>>> "checks" in a satisfying way without a long -x command, is there a >>>>>>>>> magic >>>>>>>>> flag I missed? >>>>>>>>> Also not sure about the last point and how gradle helps here - it >>>>>>>>> is rather the opposite due to the way it loads it model IMHO - so not >>>>>>>>> sure >>>>>>>>> what would be the consequence in terms of action(s) but can have >>>>>>>>> missed the >>>>>>>>> point. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Romain Manni-Bucau >>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>>>>> <https://www.linkedin.com/in/rmannibucau> >>>>>>>>> >>>>>>>>> 2018-01-24 0:20 GMT+01:00 Kenneth Knowles <k...@google.com>: >>>>>>>>> >>>>>>>>>> On Tue, Jan 23, 2018 at 2:51 PM, Romain Manni-Bucau < >>>>>>>>>> rmannibu...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> Hmm, did you read it right Kenn? I think the idea was to skip all >>>>>>>>>>> validation/sanity checks tasks at once (gradle xxxx -Pfast) instead >>>>>>>>>>> of >>>>>>>>>>> doing it manually (gradle -x findbugs -x checkstyle etc...) >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Yes, I read it right. We all want the same thing - not doing a >>>>>>>>>> bunch of extra useless unrequested stuff when developing. The >>>>>>>>>> concept of >>>>>>>>>> skipping is backwards. We don't need configs that skip things, >>>>>>>>>> because in a >>>>>>>>>> correct dependency-driven build they are already not running. >>>>>>>>>> >>>>>>>>>> So since I don't want to pretend to know gradle's invocations yet >>>>>>>>>> I will call it $TOOL. Here's a collection of imaginary commands: >>>>>>>>>> >>>>>>>>>> $TOOL :sdks:java:core:unittest # or $TOOL test >>>>>>>>>> :sdks:java:core or whatever >>>>>>>>>> $TOOL :sdks:java:core:findbugs >>>>>>>>>> $TOOL :sdks:java:core:checkstyle >>>>>>>>>> $TOOL :sdks:java:core:javadoc >>>>>>>>>> >>>>>>>>>> None of these causes any of the others to run. Anything else is a >>>>>>>>>> bug. The `findbugs` and `test` cause a build of the needed jars and >>>>>>>>>> nothing >>>>>>>>>> else. >>>>>>>>>> >>>>>>>>>> Another example: >>>>>>>>>> >>>>>>>>>> $TOOL :runners:core-java:unittest >>>>>>>>>> >>>>>>>>>> This builds the model, the core SDK, and the runners-core module, >>>>>>>>>> then runs the unit tests of the runners-core module. It does not >>>>>>>>>> test SDK >>>>>>>>>> core, or run any javadoc, findbugs, or checkstyle on any module. >>>>>>>>>> Anything >>>>>>>>>> else is a bug. >>>>>>>>>> >>>>>>>>>> Now, to build a precommit that is easy to reproduce on one line, >>>>>>>>>> you could build a compound task >>>>>>>>>> >>>>>>>>>> $TOOL :sdks:java:core:precommit # runs a selection of >>>>>>>>>> targets that we define >>>>>>>>>> >>>>>>>>>> At this point you might want to skip things from the :verify task >>>>>>>>>> here. But really, you probably just want to run the things you are >>>>>>>>>> interested in and you don't need custom hooks in the aggregated task. >>>>>>>>>> >>>>>>>>>> My understanding is that gradle can support all of this, if we >>>>>>>>>> are disciplined. Getting to this point is the main/only reason I >>>>>>>>>> supported >>>>>>>>>> gradle. >>>>>>>>>> >>>>>>>>>> Kenn >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Kenn >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> diff --git a/examples/java/build.gradle >>>>>>>>>>>>> b/examples/java/build.gradle >>>>>>>>>>>>> >> index 0fc0b17df..001bd8b38 100644 >>>>>>>>>>>>> >> --- a/examples/java/build.gradle >>>>>>>>>>>>> >> +++ b/examples/java/build.gradle >>>>>>>>>>>>> >> @@ -130,7 +130,7 @@ def preCommitAdditionalFlags = [ >>>>>>>>>>>>> >> dataflowStreamingRunner: [ "--streaming=true" ], >>>>>>>>>>>>> >> ] >>>>>>>>>>>>> >> for (String runner : preCommitRunners) { >>>>>>>>>>>>> >> - tasks.create(name: runner + "PreCommit", type: Test) { >>>>>>>>>>>>> >> + tasks.create(name: runner + "PreCommit", type: Test, >>>>>>>>>>>>> description: "Run tests >>>>>>>>>>>>> >> for runner ${runner.replace('Runner', '')}") { >>>>>>>>>>>>> >> def preCommitBeamTestPipelineOptions = [ >>>>>>>>>>>>> >> "--project=apache-beam-testing", >>>>>>>>>>>>> >> "--tempRoot=gs://temp-storage >>>>>>>>>>>>> -for-end-to-end-tests", >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Romain Manni-Bucau >>>>>>>>>>>>> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>>>>>>>> >> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>>>>>>>> >> <http://rmannibucau.wordpress.com> | Github < >>>>>>>>>>>>> https://github.com/rmannibucau> | >>>>>>>>>>>>> >> LinkedIn <https://www.linkedin.com/in/rmannibucau> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> 2018-01-23 8:45 GMT+01:00 Jean-Baptiste Onofré < >>>>>>>>>>>>> j...@nanthrax.net >>>>>>>>>>>>> >> <mailto:j...@nanthrax.net>>: >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Hi Romain, >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> I think we are pretty close: agree to add some explicit >>>>>>>>>>>>> tasks & projects names. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> We can add additional tasks like skipAudit, for >>>>>>>>>>>>> instance. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> As reminder, gradle tasks provides the list of tasks >>>>>>>>>>>>> and gradle projects >>>>>>>>>>>>> >> provides the list of projects/modules. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Regards >>>>>>>>>>>>> >> JB >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> On 01/23/2018 08:12 AM, Romain Manni-Bucau wrote: >>>>>>>>>>>>> >> > Hmm, I have to admit docs dont have my favor cause >>>>>>>>>>>>> they are easily outdated and >>>>>>>>>>>>> >> > hard to search but you hit a good point. Starting by >>>>>>>>>>>>> renaming properly the tasks >>>>>>>>>>>>> >> > and maybe writing what is done in build files - since >>>>>>>>>>>>> it is code and even "api >>>>>>>>>>>>> >> > for dev", it requires as much comments than the main >>>>>>>>>>>>> api - can be better to start. >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> > Also a big switch flag to bypass >>>>>>>>>>>>> checkstyle/findbugs/... can be good while in >>>>>>>>>>>>> >> > dev since these phases cost a looot for nothing while >>>>>>>>>>>>> you validates your code in >>>>>>>>>>>>> >> > runners modules for instance. >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> > Le 23 janv. 2018 07:15, "Kenneth Knowles" < >>>>>>>>>>>>> k...@google.com <mailto:k...@google.com> >>>>>>>>>>>>> >> > <mailto:k...@google.com <mailto:k...@google.com>>> a >>>>>>>>>>>>> écrit : >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> > On Mon, Jan 22, 2018 at 10:03 PM, Romain >>>>>>>>>>>>> Manni-Bucau <rmannibu...@gmail.com <mailto: >>>>>>>>>>>>> rmannibu...@gmail.com> >>>>>>>>>>>>> >> > <mailto:rmannibu...@gmail.com <mailto: >>>>>>>>>>>>> rmannibu...@gmail.com>>> wrote: >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> > @Kenneth: why not dropping the doc for a >>>>>>>>>>>>> script with comments in the >>>>>>>>>>>>> >> > project? A "RUNME.sh" ;). >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> > That's cool, too, but also any shell one liner >>>>>>>>>>>>> can be a gradle one >>>>>>>>>>>>> >> liner or >>>>>>>>>>>>> >> > mvn two/three liner :-). it is just trading one >>>>>>>>>>>>> command that you cannot >>>>>>>>>>>>> >> > guess easily for a different one that you still >>>>>>>>>>>>> can't guess easily. >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> > For example, are the SparkRunner ValidatesRunner >>>>>>>>>>>>> tests in the >>>>>>>>>>>>> >> SparkRunner or >>>>>>>>>>>>> >> > the core SDK or a third module that integrates >>>>>>>>>>>>> the two? And why would you >>>>>>>>>>>>> >> > know that the example ITs are called >>>>>>>>>>>>> "sparkRunnerPreCommit"? It >>>>>>>>>>>>> >> doesn't even >>>>>>>>>>>>> >> > make sense really to have "precommit" or >>>>>>>>>>>>> "postcommit" except as aliases to >>>>>>>>>>>>> >> > make it easy to repro Jenkins' behavior - they >>>>>>>>>>>>> have no other intrinsic >>>>>>>>>>>>> >> meaning. >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> > So I was proposing a mapping from "full sentence >>>>>>>>>>>>> + description" to one >>>>>>>>>>>>> >> liner >>>>>>>>>>>>> >> > to help people navigate the targets that we set >>>>>>>>>>>>> up. Some web page or doc >>>>>>>>>>>>> >> > that people can just quickly scan to find out to >>>>>>>>>>>>> do common things, easier >>>>>>>>>>>>> >> > than groovy or XML. >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> > Kenn >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> -- >>>>>>>>>>>>> >> Jean-Baptiste Onofré >>>>>>>>>>>>> >> jbono...@apache.org <mailto:jbono...@apache.org> >>>>>>>>>>>>> >> http://blog.nanthrax.net >>>>>>>>>>>>> >> Talend - http://www.talend.com >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> > >>>>>>>>>>>>> > -- >>>>>>>>>>>>> > Jean-Baptiste Onofré >>>>>>>>>>>>> > jbono...@apache.org >>>>>>>>>>>>> > http://blog.nanthrax.net >>>>>>>>>>>>> > Talend - http://www.talend.com >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >
