You can only get incremental support at the build system level, not at the individual tool level like javac. The task the represents compilation would need to be broken up into smaller tasks with smaller source sets to speed up compilation of really large modules.
On Wed, Jan 24, 2018 at 11:12 PM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > One pitfall I noted with javac wrappers is that if you dont clean - and > loose javac incremental support then deleted classes stay here and are > considered in the output. Common example is a nested class which has been > deleted leading to a corrupted enclosing class or a test which is ran but > deleted from java/. Any known way to protect us from it and keep the > uncremental support for big modules? > > Le 25 janv. 2018 01:22, "Lukasz Cwik" <lc...@google.com> a écrit : > >> Dependency driven works, incremental works for most java modules. >> I use incremental almost all the time and just do one validation pass at >> the end before opening the PR where I use '--rerun-tasks' to be sure. >> Allows me to iterate on a task in seconds. >> >> On Wed, Jan 24, 2018 at 4:07 PM, Kenneth Knowles <k...@google.com> wrote: >> >>> These are two different things: dependency-driven build (which works) >>> and incremental build (which seems not to, at least right now?). >>> >>> On Wed, Jan 24, 2018 at 2:24 PM, Romain Manni-Bucau < >>> rmannibu...@gmail.com> wrote: >>> >>>> Hmm, I'll try to refine it then next time we work with Ismael but can >>>> be a setup issue or a human (bad command) issue at the end. Thanks for the >>>> help, will make next iteration way easier probably :) >>>> >>>> >>>> Romain Manni-Bucau >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>> <http://rmannibucau.wordpress.com> | Github >>>> <https://github.com/rmannibucau> | LinkedIn >>>> <https://www.linkedin.com/in/rmannibucau> >>>> >>>> 2018-01-24 23:05 GMT+01:00 Lukasz Cwik <lc...@google.com>: >>>> >>>>> Tasks always run any dependencies that are required. So if you ask to >>>>> run test it shouldn't run javadoc/checkstyle/... but should compile the >>>>> code and compile the code of all dependencies. test should never have a >>>>> dependency on checkstyle or javadoc or similar 'check' like tasks as they >>>>> shouldn't be needed. >>>>> >>>>> I set up the gradle build so that everytime you run a command in >>>>> gradle, it generates a task dependency tree dot file (look for visteg.dot >>>>> inside build/reports). I uploaded this one to imgur[1] for the >>>>> ':sdks:java:core:build' task to show what tasks are required. Note that >>>>> 'sdks:java:core:test' doesn't depend on checkstyle or spotless. >>>>> >>>>> 1: https://imgur.com/a/ZvYUX >>>>> >>>>> On Wed, Jan 24, 2018 at 12:50 PM, Romain Manni-Bucau < >>>>> rmannibu...@gmail.com> wrote: >>>>> >>>>>> Hmm, do I miss something or it only works for iterative runs when >>>>>> trying to identify an issue and not for the case you rebuild due to code >>>>>> changes (where you would need like 5-6 tasks at least - generate, >>>>>> compile, >>>>>> test, ...)? >>>>>> >>>>>> In case it is unclear: there are 2 needs: direct execution/task -> >>>>>> fulfilled and clarified now (just a doc issue I think), fast cycle >>>>>> skipping >>>>>> not mandatory tasks like style related ones >>>>>> >>>>>> >>>>>> Romain Manni-Bucau >>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>> <https://www.linkedin.com/in/rmannibucau> >>>>>> >>>>>> 2018-01-24 19:50 GMT+01:00 Lukasz Cwik <lc...@google.com>: >>>>>> >>>>>>> Gradle already has each task explicitly broken out. Kenn is pointing >>>>>>> out that you your development use case shouldn't use the './gradlew >>>>>>> :sdks:java:core:build' task since it is really an aggregator that >>>>>>> represents do everything within that project. This is the current list >>>>>>> of >>>>>>> tasks available for :sdks:java:core: >>>>>>> :sdks:java:core:assemble - Assembles the outputs of this project. >>>>>>> :sdks:java:core:build - Assembles and tests this project. >>>>>>> :sdks:java:core:buildDependents - Assembles and tests this project >>>>>>> and all projects that depend on it. >>>>>>> :sdks:java:core:buildEnvironment - Displays all buildscript >>>>>>> dependencies declared in project :sdks:java:core. >>>>>>> :sdks:java:core:buildNeeded - Assembles and tests this project and >>>>>>> all projects it depends on. >>>>>>> :sdks:java:core:check - Runs all checks. >>>>>>> :sdks:java:core:checkstyleMain - Run Checkstyle analysis for main >>>>>>> classes >>>>>>> :sdks:java:core:checkstyleTest - Run Checkstyle analysis for test >>>>>>> classes >>>>>>> :sdks:java:core:classes - Assembles main classes. >>>>>>> :sdks:java:core:clean - Deletes the build directory. >>>>>>> :sdks:java:core:compileJava - Compiles main Java source. >>>>>>> :sdks:java:core:compileTestJava - Compiles test Java source. >>>>>>> :sdks:java:core:components - Displays the components produced by >>>>>>> project :sdks:java:core. [incubating] >>>>>>> :sdks:java:core:dependencies - Displays all dependencies declared >>>>>>> in project :sdks:java:core. >>>>>>> :sdks:java:core:dependencyInsight - Displays the insight into a >>>>>>> specific dependency in project :sdks:java:core. >>>>>>> :sdks:java:core:dependencyReport - Generates a report about your >>>>>>> library dependencies. >>>>>>> :sdks:java:core:dependentComponents - Displays the dependent >>>>>>> components of components in project :sdks:java:core. [incubating] >>>>>>> :sdks:java:core:findbugsMain - Run FindBugs analysis for main >>>>>>> classes >>>>>>> :sdks:java:core:findbugsTest - Run FindBugs analysis for test >>>>>>> classes >>>>>>> :sdks:java:core:generateAvroJava - Generates main Avro Java source >>>>>>> files from schema/protocol definition files. >>>>>>> :sdks:java:core:generateAvroProtocol - Generates main Avro >>>>>>> protocol definition files from IDL files. >>>>>>> :sdks:java:core:generateTestAvroJava - Generates test Avro Java >>>>>>> source files from schema/protocol definition files. >>>>>>> :sdks:java:core:generateTestAvroProtocol - Generates test Avro >>>>>>> protocol definition files from IDL files. >>>>>>> :sdks:java:core:help - Displays a help message. >>>>>>> :sdks:java:core:htmlDependencyReport - Generates an HTML report >>>>>>> about your library dependencies. >>>>>>> :sdks:java:core:install - Installs the archives artifacts into the >>>>>>> local Maven repository. >>>>>>> :sdks:java:core:jacocoTestCoverageVerification - Verifies code >>>>>>> coverage metrics based on specified rules for the test task. >>>>>>> :sdks:java:core:jacocoTestReport - Generates code coverage report >>>>>>> for the test task. >>>>>>> :sdks:java:core:jar - Assembles a jar archive containing the main >>>>>>> classes. >>>>>>> :sdks:java:core:javadoc - Generates Javadoc API documentation for >>>>>>> the main source code. >>>>>>> :sdks:java:core:knows - Do you know who knows? >>>>>>> :sdks:java:core:model - Displays the configuration model of project >>>>>>> :sdks:java:core. [incubating] >>>>>>> :sdks:java:core:packageTests - >>>>>>> :sdks:java:core:processResources - Processes main resources. >>>>>>> :sdks:java:core:processTestResources - Processes test resources. >>>>>>> :sdks:java:core:projectReport - Generates a report about your >>>>>>> project. >>>>>>> :sdks:java:core:projects - Displays the sub-projects of project >>>>>>> :sdks:java:core. >>>>>>> :sdks:java:core:properties - Displays the properties of project >>>>>>> :sdks:java:core. >>>>>>> :sdks:java:core:propertyReport - Generates a report about your >>>>>>> properties. >>>>>>> :sdks:java:core:shadowJar - Create a combined JAR of project and >>>>>>> runtime dependencies >>>>>>> :sdks:java:core:shadowTestJar - >>>>>>> :sdks:java:core:spotlessApply - Applies code formatting steps to >>>>>>> sourcecode in-place. >>>>>>> :sdks:java:core:spotlessCheck - Checks that sourcecode satisfies >>>>>>> formatting steps. >>>>>>> :sdks:java:core:spotlessJava - >>>>>>> :sdks:java:core:spotlessJavaApply - >>>>>>> :sdks:java:core:spotlessJavaCheck - >>>>>>> :sdks:java:core:taskReport - Generates a report about your tasks. >>>>>>> :sdks:java:core:tasks - Displays the tasks runnable from project >>>>>>> :sdks:java:core. >>>>>>> :sdks:java:core:testClasses - Assembles test classes. >>>>>>> :sdks:java:core:test - Runs the unit tests. >>>>>>> :sdks:java:core:updateOfflineRepository - >>>>>>> >>>>>>> So if you specify the ':sdks:java:core:test' task then only the >>>>>>> tests run (and any dependent tasks which are not up to date), while >>>>>>> ':sdks:java:core:check' >>>>>>> runs the suite of all checks. If your working on two modules and want to >>>>>>> run the tests for both you really want to specify each explicit end goal >>>>>>> that you want like ':sdks:java:core:test' and ':sdks:java:harness:test'. >>>>>>> Unfortunately incremental builds (https://issues.apache.org/jir >>>>>>> a/browse/BEAM-3253) are unreliable in a few of the modules (like >>>>>>> sql and python) so you may find that you need to specify --rerun-tasks >>>>>>> to >>>>>>> make sure that all tasks are rerun even if Gradle thinks they are up to >>>>>>> date. >>>>>>> >>>>>>> On Tue, Jan 23, 2018 at 3:38 PM, Romain Manni-Bucau < >>>>>>> rmannibu...@gmail.com> wrote: >>>>>>> >>>>>>>> Can have mischecked gradle setup but I don't think we are here yet, >>>>>>>> if you are not bound in a module and work accross 2 modules and iterate >>>>>>>> between working on both and one, you will likely not bypass the >>>>>>>> "checks" >>>>>>>> in a satisfying way without a long -x command, is there a magic flag I >>>>>>>> missed? >>>>>>>> Also not sure about the last point and how gradle helps here - it >>>>>>>> is rather the opposite due to the way it loads it model IMHO - so not >>>>>>>> sure >>>>>>>> what would be the consequence in terms of action(s) but can have >>>>>>>> missed the >>>>>>>> point. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Romain Manni-Bucau >>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>>>> <https://www.linkedin.com/in/rmannibucau> >>>>>>>> >>>>>>>> 2018-01-24 0:20 GMT+01:00 Kenneth Knowles <k...@google.com>: >>>>>>>> >>>>>>>>> On Tue, Jan 23, 2018 at 2:51 PM, Romain Manni-Bucau < >>>>>>>>> rmannibu...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Hmm, did you read it right Kenn? I think the idea was to skip all >>>>>>>>>> validation/sanity checks tasks at once (gradle xxxx -Pfast) instead >>>>>>>>>> of >>>>>>>>>> doing it manually (gradle -x findbugs -x checkstyle etc...) >>>>>>>>>> >>>>>>>>> >>>>>>>>> Yes, I read it right. We all want the same thing - not doing a >>>>>>>>> bunch of extra useless unrequested stuff when developing. The concept >>>>>>>>> of >>>>>>>>> skipping is backwards. We don't need configs that skip things, >>>>>>>>> because in a >>>>>>>>> correct dependency-driven build they are already not running. >>>>>>>>> >>>>>>>>> So since I don't want to pretend to know gradle's invocations yet >>>>>>>>> I will call it $TOOL. Here's a collection of imaginary commands: >>>>>>>>> >>>>>>>>> $TOOL :sdks:java:core:unittest # or $TOOL test >>>>>>>>> :sdks:java:core or whatever >>>>>>>>> $TOOL :sdks:java:core:findbugs >>>>>>>>> $TOOL :sdks:java:core:checkstyle >>>>>>>>> $TOOL :sdks:java:core:javadoc >>>>>>>>> >>>>>>>>> None of these causes any of the others to run. Anything else is a >>>>>>>>> bug. The `findbugs` and `test` cause a build of the needed jars and >>>>>>>>> nothing >>>>>>>>> else. >>>>>>>>> >>>>>>>>> Another example: >>>>>>>>> >>>>>>>>> $TOOL :runners:core-java:unittest >>>>>>>>> >>>>>>>>> This builds the model, the core SDK, and the runners-core module, >>>>>>>>> then runs the unit tests of the runners-core module. It does not test >>>>>>>>> SDK >>>>>>>>> core, or run any javadoc, findbugs, or checkstyle on any module. >>>>>>>>> Anything >>>>>>>>> else is a bug. >>>>>>>>> >>>>>>>>> Now, to build a precommit that is easy to reproduce on one line, >>>>>>>>> you could build a compound task >>>>>>>>> >>>>>>>>> $TOOL :sdks:java:core:precommit # runs a selection of targets >>>>>>>>> that we define >>>>>>>>> >>>>>>>>> At this point you might want to skip things from the :verify task >>>>>>>>> here. But really, you probably just want to run the things you are >>>>>>>>> interested in and you don't need custom hooks in the aggregated task. >>>>>>>>> >>>>>>>>> My understanding is that gradle can support all of this, if we are >>>>>>>>> disciplined. Getting to this point is the main/only reason I supported >>>>>>>>> gradle. >>>>>>>>> >>>>>>>>> Kenn >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Kenn >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >> >>>>>>>>>>>> >> diff --git a/examples/java/build.gradle >>>>>>>>>>>> b/examples/java/build.gradle >>>>>>>>>>>> >> index 0fc0b17df..001bd8b38 100644 >>>>>>>>>>>> >> --- a/examples/java/build.gradle >>>>>>>>>>>> >> +++ b/examples/java/build.gradle >>>>>>>>>>>> >> @@ -130,7 +130,7 @@ def preCommitAdditionalFlags = [ >>>>>>>>>>>> >> dataflowStreamingRunner: [ "--streaming=true" ], >>>>>>>>>>>> >> ] >>>>>>>>>>>> >> for (String runner : preCommitRunners) { >>>>>>>>>>>> >> - tasks.create(name: runner + "PreCommit", type: Test) { >>>>>>>>>>>> >> + tasks.create(name: runner + "PreCommit", type: Test, >>>>>>>>>>>> description: "Run tests >>>>>>>>>>>> >> for runner ${runner.replace('Runner', '')}") { >>>>>>>>>>>> >> def preCommitBeamTestPipelineOptions = [ >>>>>>>>>>>> >> "--project=apache-beam-testing", >>>>>>>>>>>> >> "--tempRoot=gs://temp-storage-for-end-to-end-tests", >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> Romain Manni-Bucau >>>>>>>>>>>> >> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>>>>>>> >> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>>>>>>> >> <http://rmannibucau.wordpress.com> | Github < >>>>>>>>>>>> https://github.com/rmannibucau> | >>>>>>>>>>>> >> LinkedIn <https://www.linkedin.com/in/rmannibucau> >>>>>>>>>>>> >> >>>>>>>>>>>> >> 2018-01-23 8:45 GMT+01:00 Jean-Baptiste Onofré < >>>>>>>>>>>> j...@nanthrax.net >>>>>>>>>>>> >> <mailto:j...@nanthrax.net>>: >>>>>>>>>>>> >> >>>>>>>>>>>> >> Hi Romain, >>>>>>>>>>>> >> >>>>>>>>>>>> >> I think we are pretty close: agree to add some explicit >>>>>>>>>>>> tasks & projects names. >>>>>>>>>>>> >> >>>>>>>>>>>> >> We can add additional tasks like skipAudit, for instance. >>>>>>>>>>>> >> >>>>>>>>>>>> >> As reminder, gradle tasks provides the list of tasks and >>>>>>>>>>>> gradle projects >>>>>>>>>>>> >> provides the list of projects/modules. >>>>>>>>>>>> >> >>>>>>>>>>>> >> Regards >>>>>>>>>>>> >> JB >>>>>>>>>>>> >> >>>>>>>>>>>> >> On 01/23/2018 08:12 AM, Romain Manni-Bucau wrote: >>>>>>>>>>>> >> > Hmm, I have to admit docs dont have my favor cause >>>>>>>>>>>> they are easily outdated and >>>>>>>>>>>> >> > hard to search but you hit a good point. Starting by >>>>>>>>>>>> renaming properly the tasks >>>>>>>>>>>> >> > and maybe writing what is done in build files - since >>>>>>>>>>>> it is code and even "api >>>>>>>>>>>> >> > for dev", it requires as much comments than the main >>>>>>>>>>>> api - can be better to start. >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > Also a big switch flag to bypass >>>>>>>>>>>> checkstyle/findbugs/... can be good while in >>>>>>>>>>>> >> > dev since these phases cost a looot for nothing while >>>>>>>>>>>> you validates your code in >>>>>>>>>>>> >> > runners modules for instance. >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > Le 23 janv. 2018 07:15, "Kenneth Knowles" < >>>>>>>>>>>> k...@google.com <mailto:k...@google.com> >>>>>>>>>>>> >> > <mailto:k...@google.com <mailto:k...@google.com>>> a >>>>>>>>>>>> écrit : >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > On Mon, Jan 22, 2018 at 10:03 PM, Romain >>>>>>>>>>>> Manni-Bucau <rmannibu...@gmail.com <mailto: >>>>>>>>>>>> rmannibu...@gmail.com> >>>>>>>>>>>> >> > <mailto:rmannibu...@gmail.com <mailto: >>>>>>>>>>>> rmannibu...@gmail.com>>> wrote: >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > @Kenneth: why not dropping the doc for a >>>>>>>>>>>> script with comments in the >>>>>>>>>>>> >> > project? A "RUNME.sh" ;). >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > That's cool, too, but also any shell one liner can >>>>>>>>>>>> be a gradle one >>>>>>>>>>>> >> liner or >>>>>>>>>>>> >> > mvn two/three liner :-). it is just trading one >>>>>>>>>>>> command that you cannot >>>>>>>>>>>> >> > guess easily for a different one that you still >>>>>>>>>>>> can't guess easily. >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > For example, are the SparkRunner ValidatesRunner >>>>>>>>>>>> tests in the >>>>>>>>>>>> >> SparkRunner or >>>>>>>>>>>> >> > the core SDK or a third module that integrates the >>>>>>>>>>>> two? And why would you >>>>>>>>>>>> >> > know that the example ITs are called >>>>>>>>>>>> "sparkRunnerPreCommit"? It >>>>>>>>>>>> >> doesn't even >>>>>>>>>>>> >> > make sense really to have "precommit" or >>>>>>>>>>>> "postcommit" except as aliases to >>>>>>>>>>>> >> > make it easy to repro Jenkins' behavior - they >>>>>>>>>>>> have no other intrinsic >>>>>>>>>>>> >> meaning. >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > So I was proposing a mapping from "full sentence + >>>>>>>>>>>> description" to one >>>>>>>>>>>> >> liner >>>>>>>>>>>> >> > to help people navigate the targets that we set >>>>>>>>>>>> up. Some web page or doc >>>>>>>>>>>> >> > that people can just quickly scan to find out to >>>>>>>>>>>> do common things, easier >>>>>>>>>>>> >> > than groovy or XML. >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > Kenn >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> >>>>>>>>>>>> >> -- >>>>>>>>>>>> >> Jean-Baptiste Onofré >>>>>>>>>>>> >> jbono...@apache.org <mailto:jbono...@apache.org> >>>>>>>>>>>> >> http://blog.nanthrax.net >>>>>>>>>>>> >> Talend - http://www.talend.com >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> > >>>>>>>>>>>> > -- >>>>>>>>>>>> > Jean-Baptiste Onofré >>>>>>>>>>>> > jbono...@apache.org >>>>>>>>>>>> > http://blog.nanthrax.net >>>>>>>>>>>> > Talend - http://www.talend.com >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>
