That makes sense. The point is that we have to compare equivalently. I'm also curious about Gradle PoC assuming it does the same actions as Maven.
Regards JB On Nov 3, 2017, 20:41, at 20:41, Kenneth Knowles <k...@google.com.INVALID> wrote: >I'm confident that any choice will speed things up dramatically even >beyond >a fast profile, even if the new tool runs all the extra stuff. But that >is >a question that we can answer empirically anyhow. Let's see how it >goes! > >Incidentally, my experiments with Bazel have led me to the conclusion >that >it is not the right choice for us so I'm not going to be proposing any >completed POC of that right now. I'm interested in the outcome of the >Gradle POC. > >Kenn > > >On Fri, Nov 3, 2017 at 3:30 AM, Jean-Baptiste Onofré <j...@nanthrax.net> >wrote: > >> Hi >> >> It's what I said in a previous e-mail: I don't think that just >changing >> the build tool will improve a lot the build time. >> >> We already know (and discussed while ago) that plugins like findbugs, >> checkstyle, etc are taking time. >> >> So, I think we can already have a fast profile. >> >> Regards >> JB >> >> On Nov 3, 2017, 11:16, at 11:16, Romain Manni-Bucau ><rmannibu...@gmail.com> >> wrote: >> >Hi guys, >> > >> >when you check the duration of each mojo of the build (almost since >> >python part of the build just breaks it locally) you see that there >is >> >no real link with maven for the perf issues beam can encounter: >> >https://gist.github.com/rmannibucau/f65fdde28d5dab0fdac50633f84554c9 >> >(generated from the profiling of tesla-profile and parsed with >> >>https://gist.github.com/rmannibucau/e329d54b8af6c009f46fd151d10037ad) >> > >> >Before PoC-ing other tools which will end up to either have the same >> >issues if the other builds do the same things (test, checkstyle, >> >enforcer, findbugs, ...) or have a less reliable build (trying to >skip >> >some parts of the build if "untouched" - note that this is a very >hard >> >issue since static code anaylizis doesn't give you any guarantee of >> >what it does with modern code - then maybe some action can be taken >on >> >the current build: >> > >> >- testing https://github.com/vackosar/gitflow-incremental-builder or >> >https://github.com/khmarbaise/incremental-module-builder maybe or do >> >the same kind of extension including the beam needs (/!\ the >previous >> >warning is still accurate and requires a full run at some point to >> >validate the graph detection algorithm didn't get abused by some >> >indirect code dependency) >> >- maybe try to get rid of some shades (it is a bit crazy ATM to have >> >so much shades no?) >> >- the CI can have profiles based on a PR convention (name of the >> >branch?) to select the build profile, for instance >> >fb/elasticsearch_super-nice-PR would build only the elasticsearch >> >modules, jenkins/travis have this ability since they support >scripting >> >- document how to setup a "fastBuild" profile in its settings.xml >> >which bypasses checkstyle, enforcer plugin, findbugs, etc... for >fast >> >development iterations >> > >> > >> > >> > >> >Romain Manni-Bucau >> >@rmannibucau | Blog | Old Blog | Github | LinkedIn >> > >> > >> >2017-11-01 21:02 GMT+01:00 Kenneth Knowles <k...@google.com.invalid>: >> >> I have started one, here: >> >https://github.com/kennknowles/beam/commits/bazel. >> >> It is not nearly as far along as Luke's. For the POC I am just >> >putting >> >> things in one root BUILD, and learning where we might find the >> >necessary >> >> plugins as I go. I am happy to grant push access to this branch. >It >> >would >> >> be superb if you had some time to work through the Python steps. >> >> >> >> On Wed, Nov 1, 2017 at 10:09 AM, Ahmet Altay >> ><al...@google.com.invalid> >> >> wrote: >> >> >> >>> Has anyone started a POC with Bazel? I would be interested in >> >helping that >> >>> effort. >> >>> >> >>> On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik >> ><lc...@google.com.invalid> >> >>> wrote: >> >>> >> >>> > I have started a POC for using Gradle here: >> >>> > https://github.com/lukecwik/incubator-beam/tree/gradle >> >>> > >> >>> > Things that work: >> >>> > * compiling all Java code (src/main and src/test) >> >>> > * generating source from protos >> >>> > * generating source from avro >> >>> > * running rat, checkstyle >> >>> > >> >>> > Partially working: >> >>> > * generating maven pom (albeit with wrong dependencies for some >> >>> > subprojects) >> >>> > * running tests (~80% pass, remainder seem to be dependency >> >related but >> >>> are >> >>> > uninvestigated) >> >>> > >> >>> > Things that don't work: >> >>> > * anything Python/Go/Docker compilation related >> >>> > * many tests fail because I messed up dependencies >> >>> > * anything shading related >> >>> > * minor plugins like eclipse code formatter/... >> >>> > * running @NeedsRunner/@ValidatesRunner/integration tests >> >>> > >> >>> > Feel free to reach out to me on Slack if you would like to try >to >> >tackle >> >>> a >> >>> > piece of the POC to prevent duplication of effort from anyone >> >working on >> >>> > it. >> >>> > >> >>> > >> >>> > >> >>> > On Tue, Oct 31, 2017 at 10:25 PM, Jean-Baptiste Onofré >> ><j...@nanthrax.net> >> >>> > wrote: >> >>> > >> >>> > > Agree to move forward on a PoC. >> >>> > > >> >>> > > Thanks Reuven for bringing discussion on the mailing list ! >> >>> > > >> >>> > > Regards >> >>> > > JB >> >>> > > >> >>> > > On Nov 1, 2017, 03:20, at 03:20, Reuven Lax >> ><re...@google.com.INVALID> >> >>> > > wrote: >> >>> > > >Some good discussion here, and thanks to JB and Romain for >> >adding to >> >>> > > >it! >> >>> > > > >> >>> > > >JB makes the good point that we still need to release Maven >> >artifacts, >> >>> > > >as >> >>> > > >many Beam users want to develop using Maven. So none of this >> >>> discussion >> >>> > > >will affect our release process, as we still need Maven >> >"releases." >> >>> > > > >> >>> > > >At this point, if people are interested, I see no harm in >> >prototyping. >> >>> > > >Having working alternatives will give us a better basis for >> >comparison >> >>> > > >to >> >>> > > >understand whether these other build systems give us >anything >> >over >> >>> what >> >>> > > >Maven does. >> >>> > > > >> >>> > > >Reuven >> >>> > > > >> >>> > > >On Tue, Oct 31, 2017 at 11:05 AM, Charles Chen >> ><c...@google.com.invalid >> >>> > >> >>> > > >wrote: >> >>> > > > >> >>> > > >> As a contributor to the Beam Python SDK, I noticed that >many >> >of the >> >>> > > >points >> >>> > > >> above regarding Maven and Gradle pertain mostly to Java >SDK >> >>> > > >development. >> >>> > > >> For Python development, Maven is much less natural, and we >> >end up >> >>> > > >just >> >>> > > >> shelling out to perform builds and tests. For Python SDK >> >(and >> >>> > > >upcoming Go >> >>> > > >> SDK development), an option to use Bazel would be quite >> >useful. >> >>> > > >> >> >>> > > >> On Tue, Oct 31, 2017 at 10:42 AM Robert Bradshaw >> >>> > > >> <rober...@google.com.invalid> wrote: >> >>> > > >> >> >>> > > >> > +1, Maven is both a build tool and a repository, and the >> >latter is >> >>> > > >> > essential to keep. Both Gradel and Bazel can interface >with >> >this >> >>> > > >> > repository. >> >>> > > >> > >> >>> > > >> > I am, however, very supportive of moving away from Maven >to >> >a tool >> >>> > > >> > that supports correct incremental, hermetic, >> >dependency-driven, >> >>> > > >> > multi-langauge, and hopefully fast builds for our own >> >development. >> >>> > > >> > >> >>> > > >> > On Tue, Oct 31, 2017 at 10:00 AM, Kenneth Knowles >> >>> > > >> > <k...@google.com.invalid> wrote: >> >>> > > >> > > Echoing what JB and Reuven said, we absolutely must >> >provide >> >>> maven >> >>> > > >> central >> >>> > > >> > > artifacts for Java users, just as we provide pypi >> >artifacts for >> >>> > > >Python >> >>> > > >> > > users. >> >>> > > >> > > >> >>> > > >> > > I see Maven as still a viable tool for single-module >Java >> >>> builds, >> >>> > > >> > > especially considering its rich plugin ecosystem. >> >>> > > >> > > >> >>> > > >> > > On Mon, Oct 30, 2017 at 11:27 PM, Reuven Lax >> >>> > > ><re...@google.com.invalid >> >>> > > >> > >> >>> > > >> > > wrote: >> >>> > > >> > > >> >>> > > >> > >> I think that's a very good point. No matter what >build >> >system >> >>> we >> >>> > > >use >> >>> > > >> for >> >>> > > >> > >> our own personal development, we still need to >release >> >Maven >> >>> > > >artifacts >> >>> > > >> > and >> >>> > > >> > >> releases as we need to support our users using Maven. >> >>> > > >> > >> >> >>> > > >> > >> On Mon, Oct 30, 2017 at 11:26 PM, Jean-Baptiste >Onofré < >> >>> > > >> j...@nanthrax.net >> >>> > > >> > > >> >>> > > >> > >> wrote: >> >>> > > >> > >> >> >>> > > >> > >> > Generally speaking, it's interesting to evaluate >> >>> alternatives, >> >>> > > >> > especially >> >>> > > >> > >> > Gradle. My point is also to keep Maven artifacts >and >> >>> > > >"releases" as >> >>> > > >> > most >> >>> > > >> > >> of >> >>> > > >> > >> > our users will use Maven. >> >>> > > >> > >> > For incremental build, afair, there's some >> >enhancements on >> >>> > > >Maven >> >>> > > >> but I >> >>> > > >> > >> > have to take a look. >> >>> > > >> > >> > >> >>> > > >> > >> > Regards >> >>> > > >> > >> > JB >> >>> > > >> > >> > >> >>> > > >> > >> > On Oct 31, 2017, 07:22, at 07:22, Eugene Kirpichov >> >>> > > >> > >> > <kirpic...@google.com.INVALID> wrote: >> >>> > > >> > >> > >Hi! >> >>> > > >> > >> > > >> >>> > > >> > >> > >Many of these points sound valid, but AFAICT Maven >> >doesn't >> >>> > > >really >> >>> > > >> do >> >>> > > >> > >> > >incremental builds [1]. The best it can do is, it >> >seems, >> >>> > > >recompile >> >>> > > >> > only >> >>> > > >> > >> > >changed files, but Java compilation is a tiny part >of >> >the >> >>> > > >overall >> >>> > > >> > >> > >build. >> >>> > > >> > >> > > >> >>> > > >> > >> > >Almost all time is taken by other plugins, such as >> >unit >> >>> > > >testing or >> >>> > > >> > >> > >findbugs >> >>> > > >> > >> > >- and Maven does not seem to currently support >> >features such >> >>> > > >as "do >> >>> > > >> > not >> >>> > > >> > >> > >rerun unit tests of a module if the code didn't >> >change". >> >>> > > >> > >> > > >> >>> > > >> > >> > >The fact that the surefire plugin has existed for >>11 >> >years >> >>> > > >> (version >> >>> > > >> > >> > >2.0 >> >>> > > >> > >> > >was released in 2006) and still doesn't have this >> >feature >> >>> > > >makes me >> >>> > > >> > >> > >think >> >>> > > >> > >> > >that it's unlikely to be supported in the next few >> >years >> >>> > > >either. >> >>> > > >> > >> > > >> >>> > > >> > >> > >I suspect most PRs affect a very small number of >> >modules, so >> >>> > > >I >> >>> > > >> think >> >>> > > >> > >> > >the >> >>> > > >> > >> > >performance advantage of a build system truly >> >supporting >> >>> > > >> incremental >> >>> > > >> > >> > >builds >> >>> > > >> > >> > >may be so overwhelming as to trump many other >> >factors. Of >> >>> > > >course, >> >>> > > >> > we'd >> >>> > > >> > >> > >need >> >>> > > >> > >> > >to prototype and have hard numbers in hand to >discuss >> >this >> >>> > > >with >> >>> > > >> more >> >>> > > >> > >> > >substance. >> >>> > > >> > >> > > >> >>> > > >> > >> > >[1] >> >>> > > >> > >> > >> >>https://stackoverflow.com/questions/8918165/does-maven- >> >>> > > >> > >> > support-incremental-builds >> >>> > > >> > >> > > >> >>> > > >> > >> > >On Mon, Oct 30, 2017 at 10:57 PM Romain >Manni-Bucau >> >>> > > >> > >> > ><rmannibu...@gmail.com> >> >>> > > >> > >> > >wrote: >> >>> > > >> > >> > > >> >>> > > >> > >> > >> Hi >> >>> > > >> > >> > >> >> >>> > > >> > >> > >> Even if not a commiter or even PMC, I'd like to >> >mention a >> >>> > > >few >> >>> > > >> > points >> >>> > > >> > >> > >from >> >>> > > >> > >> > >> an external eye: >> >>> > > >> > >> > >> >> >>> > > >> > >> > >> - Maven stays the most common build tool and >easier >> >one >> >>> for >> >>> > > >any >> >>> > > >> > user. >> >>> > > >> > >> > >It >> >>> > > >> > >> > >> means it is the best one to hope contributions >> >IMHO. >> >>> > > >> > >> > >> - Maven has incremental support but if there is >any >> >>> blocker >> >>> > > >the >> >>> > > >> > >> > >community >> >>> > > >> > >> > >> is probably ready to enhance it (has been done >for >> >>> compiler >> >>> > > >> plugin >> >>> > > >> > >> > >for >> >>> > > >> > >> > >> instance) >> >>> > > >> > >> > >> - Gradle hides issues easily with its daemon so >a >> >build >> >>> > > >without >> >>> > > >> > >> > >daemon is >> >>> > > >> > >> > >> needed >> >>> > > >> > >> > >> - Gradle doesnt isolate plugins well enough so >> >ensure your >> >>> > > >> planned >> >>> > > >> > >> > >plugins >> >>> > > >> > >> > >> doesnt conflict >> >>> > > >> > >> > >> - Only Maven is correctly supported in >mainstream >> >and >> >>> > > >OS/free IDE >> >>> > > >> > >> > >> >> >>> > > >> > >> > >> This is the reasons why I think Maven is better >- >> >not even >> >>> > > >> entering >> >>> > > >> > >> > >into >> >>> > > >> > >> > >> the ASF points. >> >>> > > >> > >> > >> >> >>> > > >> > >> > >> Now Maven is not perfect but some quick >> >enhancements can >> >>> be >> >>> > > >done: >> >>> > > >> > >> > >> >> >>> > > >> > >> > >> - A fast build profile can be created >> >>> > > >> > >> > >> - Takari scheduler can be used yo enhance the >> >parallel >> >>> > > >build >> >>> > > >> > >> > >> - Scripts can be provided to build a subpart of >the >> >>> project >> >>> > > >> > >> > >> - A beam extension can surely be done to >optimize >> >or >> >>> > > >compute the >> >>> > > >> > >> > >reactors >> >>> > > >> > >> > >> more easily based on module names >> >>> > > >> > >> > >> >> >>> > > >> > >> > >> Romain >> >>> > > >> > >> > >> >> >>> > > >> > >> > >> Le 31 oct. 2017 06:42, "Jean-Baptiste Onofré" >> >>> > > ><j...@nanthrax.net> >> >>> > > >> a >> >>> > > >> > >> > >écrit : >> >>> > > >> > >> > >> >> >>> > > >> > >> > >> -0 >> >>> > > >> > >> > >> >> >>> > > >> > >> > >> For the following reasons reasons: >> >>> > > >> > >> > >> - maven is a Apache project and we can have >> >>> > > >support/improvement >> >>> > > >> > >> > >> - I don't see how another build tool would speed >up >> >the >> >>> > > >build by >> >>> > > >> > >> > >itself >> >>> > > >> > >> > >> - Apache default release process is based on >Maven >> >>> > > >> > >> > >> >> >>> > > >> > >> > >> On the other hand, Gradle could be interesting. >> >Anyway >> >>> it's >> >>> > > >> > something >> >>> > > >> > >> > >to >> >>> > > >> > >> > >> evaluate. >> >>> > > >> > >> > >> >> >>> > > >> > >> > >> Regards >> >>> > > >> > >> > >> JB >> >>> > > >> > >> > >> >> >>> > > >> > >> > >> >> >>> > > >> > >> > >> On Oct 30, 2017, 18:46, at 18:46, Ted Yu >> >>> > > ><yuzhih...@gmail.com> >> >>> > > >> > wrote: >> >>> > > >> > >> > >> >I agree with Ben's comment. >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> >Recently I have been using gradle in another >> >Apache >> >>> > > >project and >> >>> > > >> > >> > >found >> >>> > > >> > >> > >> >it >> >>> > > >> > >> > >> >interesting. >> >>> > > >> > >> > >> > >> >>> > > >> > >> > >> >Cheers >> >>> > > >> > >> > >> >> >>> > > >> > >> > >> >>> > > >> > >> >> >>> > > >> > >> >>> > > >> >> >>> > > >> >>> > >> >>> >>
