Just to answer a previous question: An ASF github search gives me these stats:
- mvn (org:apache apache filename:pom.xml path:/): 731 - gradle (org:apache apache filename:build.gradle path:/): 31 Which is consistent with what I saw in enterprises and private repos. So way different from the whole github stats which is not done on a representative sample if not pre-filtered. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-28 9:08 GMT+01:00 Robert Bradshaw <rober...@google.com>: > It's great to see all the discussion going on here. > > I think it's important to point out that merging a parallel set of > gradle build scripts is a separate (and much less disruptive) step > than, say, switching over the default (or even recommended) > build/release process to use them, let alone removing the maven build > files entirely. The latter two should definitely be gated by a formal > vote (each, probably), with the current state the gradle files can > mostly be ignored by most people. In particular, this is the kind of > change that needs to be in master to be evaluated--if it's on a branch > we can't very well see how it impacts presubmits, and most importantly > people can't try it out for real development. > > I agree that the choice of build tool may attract some contributors > and discourage others. Having builds that are fast, correct, and > reproducible will probably matter more to potential contributors than > the particular command to run. While maven can surely be improved, I > doubt a 2x improvement (and many more times that for incremental > builds) is low-hanging fruit, and many of the issues seem quite > fundamental (e.g. all the special treatment we need for NeedsRunner > tests, and having to do a (global-by-default) mvn install to skip > tests of dependencies when testing a leaf module). > > Getting data on what other apache projects use could be interesting, > but unless we gather why such choices were made I don't know that it'd > be that influential once we've established that both tools are widely > supported generally. > > To re-emphasize, we'll continue to produce and publish maven > artifacts, so our choice of build system won't matter for those only > using Beam as a dependency. > > > > On Mon, Nov 27, 2017 at 10:48 PM, Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: >> Yeah, especially, I think it would have been great to have a vote before >> merging on master. >> >> Not a big deal, however, I'm really community focus ;) >> >> Regards >> JB >> >> On 11/28/2017 07:36 AM, Reuven Lax wrote: >>> >>> Agreed. I thinking having a formal vote before Luke had numbers and >>> results would have been too early. However now that we have such numbers, we >>> should think about having a vote. >>> >>> Also, while I disagree with Romain that Gradle is not "enterprise ready" >>> (it's heavily used by Netflix, LinkedIn, Siemens, and is the default build >>> framework for Android apps), it would be interesting to see if any other ASF >>> projects are using it. I don't think that should not make or break the >>> decision - we should do what's best for the Beam project, and "everyone else >>> is doing something" is rarely a good argument - it will provide good data >>> points for us to evaluate. >>> >>> Reuven >>> >>> On Mon, Nov 27, 2017 at 10:23 PM, Jean-Baptiste Onofré <j...@nanthrax.net >>> <mailto:j...@nanthrax.net>> wrote: >>> >>> Hi Luke, >>> >>> just curious (and maybe I missed it): did we do a formal vote to merge >>> the >>> gradle build ? >>> Gradle is now on master, we have some Jira to update the release guide >>> with >>> gradle. It's fine, but I remember only a discussion, not a vote. >>> >>> In order to embrace the community and avoid to have some contributors >>> "frustrated" (meaning that "this project doesn't care about >>> contributor, >>> they just do whatever they want"), I would have love to see a formal >>> vote >>> about Gradle more than just a discussion. >>> >>> My $0.01 >>> >>> Regards >>> JB >>> >>> On 11/27/2017 07:46 PM, Lukasz Cwik wrote: >>> >>> I have collected data by running several builds against master >>> using Gradle >>> and Maven without using Gradle's support for incremental builds. >>> >>> Gradle (mins) >>> min: 25.04 >>> max: 160.14 >>> median: 45.78 >>> average: 52.19 >>> stdev: 30.80 >>> >>> Maven (mins) >>> min: 56.86 >>> max: 216.55 (actually > 240 mins because this data does not >>> include >>> timeouts) >>> median: 87.93 >>> average: 109.10 >>> stdev: 48.01 >>> >>> I excluded a few timeouts (240 mins) that happened during the >>> Maven build >>> from its numbers but we can see conclusively that Gradle is twice >>> as fast >>> for the build when compared to Maven when run using Jenkins. >>> On my desktop, I have enabled incremental builds and have seen a >>> major >>> improvement on the above numbers but it doesn't yet work correctly >>> because >>> of incorrectly specified inputs/outputs for certain tasks. >>> >>> The data is available here >>> >>> https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing >>> >>> <https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing> >>> >>> With this data, I feel confident that we should swap and have >>> opened the >>> following issue https://issues.apache.org/jira/browse/BEAM-3249 >>> <https://issues.apache.org/jira/browse/BEAM-3249> and related >>> sub-tasks. >>> >>> On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré >>> <j...@nanthrax.net >>> <mailto:j...@nanthrax.net>> >>> wrote: >>> >>> Thanks for the update Luke. >>> >>> I'm updating my local working copy to do new tests. >>> >>> Regards >>> JB >>> >>> On 11/19/2017 08:21 PM, Lukasz Cwik wrote: >>> >>> The gradle build rules have been merged, I'm adding a >>> precommit[1] to >>> start >>> collecting data about the build times. It currently only >>> mirrors >>> the Java >>> mvn install precommit. I'll gather data over the next two >>> weeks and >>> provide >>> a summary here. >>> >>> You can rerun the precommit by issuing "Run Java Gradle >>> PreCommit" >>> >>> 1: https://github.com/apache/beam/pull/4146 >>> <https://github.com/apache/beam/pull/4146> >>> >>> >>> On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik >>> <lc...@google.com >>> <mailto:lc...@google.com>> wrote: >>> >>> There has been plenty of time for comments on the PR and >>> the >>> approach. >>> >>> >>> So far Ken Knowles has provided the most feedback on >>> the PR, >>> Ken would >>> you >>> like to finish the review? >>> >>> >>> >>> On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau < >>> rmannibu...@gmail.com <mailto:rmannibu...@gmail.com> >>> >>> wrote: >>> >>> >>> This is only a setup thing and better to not break the >>> master history for >>> >>> poc/tests, in particular when no very localized. >>> Alternative can be to >>> ask >>> another temp repo to infra and have a synchro >>> between >>> both but dont >>> think >>> it does worth it personally. >>> >>> >>> >>> Le 10 nov. 2017 18:57, "Lukasz Cwik" >>> <lc...@google.com.invalid> a >>> écrit : >>> >>> The reason to get it on master is because that is >>> where >>> all the PRs >>> >>> >>> are. An >>> >>> upstream branch without any development means >>> no data. >>> Also, our Jenkins setup via job-dsl doesn't >>> honor >>> using the Jenkins >>> configuration on the branch because the seed >>> job >>> always runs against >>> master. >>> >>> On Thu, Nov 9, 2017 at 9:59 PM, Romain >>> Manni-Bucau < >>> >>> rmannibu...@gmail.com >>> <mailto:rmannibu...@gmail.com>> >>> >>> wrote: >>> >>> What about pushing it on a "upstream" branch >>> and >>> testing it for 1 >>> >>> >>> week in >>> >>> >>> parallel of the maven reference build? If >>> gradle is >>> always 50% faster >>> >>> >>> on >>> >>> >>> jenkins then it could become master setup >>> without >>> much discussion I >>> >>> >>> guess. >>> >>> We can even have 2 jenkins jobs: one with >>> the >>> daemon etc and one >>> >>> without. >>> >>> >>> >>> Also noticed yesterday that gradle build >>> is >>> killing my machine (all 8 >>> >>> cores >>> >>> are 100%) during the first minutes vs >>> maven >>> build which let me do >>> >>> something >>> >>> else. Then all the consumed time which >>> makes >>> gradle not that fast is >>> >>> about >>> >>> python. Will try to send figures later >>> today. >>> >>> Le 10 nov. 2017 00:10, "Lukasz Cwik" >>> <lc...@google.com.invalid> a >>> >>> écrit >>> >>> >>> : >>> >>> >>> I wouldn't mind merging this change in so >>> I >>> could setup those Gradle >>> >>> Jenkins precommits. >>> >>> As per our contribution guidelines, >>> any >>> committer willing to sign >>> >>> off >>> >>> >>> on >>> >>> the PR? >>> >>> >>> On Thu, Nov 9, 2017 at 2:12 PM, Romain >>> Manni-Bucau < >>> >>> rmannibu...@gmail.com >>> <mailto:rmannibu...@gmail.com>> >>> >>> wrote: >>> >>> Le 9 nov. 2017 21:31, "Kenneth >>> Knowles" >>> <k...@google.com.invalid> >>> >>> >>> a >>> >>> >>> écrit : >>> >>> >>> >>> Keep in mind that a clean build is >>> unusual during development (it >>> >>> is >>> >>> >>> common >>> >>> >>> for mvn use and that is a bug) and >>> also >>> not necessary for >>> >>> precommits >>> >>> >>> if >>> >>> the >>> >>> >>> build tool is correct enough that >>> caching is safe. So while this >>> >>> number >>> >>> >>> matters, it is not the most important. >>> >>> >>> >>> Not sure, in dev you bypass the >>> build >>> tool most of the time >>> >>> anyway - >>> >>> >>> thanks >>> >>> >>> to IDE or other shortcuts - but >>> not on >>> PR and CI. Keep in mind >>> >>> that >>> >>> >>> not >>> >>> doing a clean and killing gradle daemon >>> makes >>> the build not >>> >>> >>> reproducible >>> >>> >>> and therefore useful :(. Starting to >>> build >>> from a subpart of the >>> >>> >>> reactor >>> >>> >>> - >>> >>> with the mentionned mvn plugin for >>> instance - can be nice on some >>> >>> CI >>> >>> >>> like >>> >>> >>> travis if the caching is well >>> configured but >>> still not a guarantee >>> >>> >>> the >>> >>> >>> build is "green". >>> >>> >>> My trade off is to ensure an easy >>> build >>> and relevant result over >>> >>> the >>> >>> >>> time >>> >>> >>> criteria. Do you share it as well or >>> prefer >>> time over other >>> >>> >>> criteria >>> >>> >>> - >>> >>> which leads to other conclusions and >>> options >>> indeed and can make >>> >>> >>> us >>> >>> >>> not >>> >>> understanding each other? >>> >>> >>> >>> On Thu, Nov 9, 2017 at 11:30 AM, >>> Romain >>> Manni-Bucau < >>> >>> rmannibu...@gmail.com >>> <mailto:rmannibu...@gmail.com> >>> >>> >>> wrote: >>> >>> >>> I will try next week yes but the 2 >>> runs >>> i did were 28mn vs 32mn >>> >>> >>> from >>> >>> >>> memory >>> >>> >>> - after having downloaded all >>> deps once. >>> >>> Le 9 nov. 2017 19:45, "Lukasz >>> Cwik" >>> <lc...@google.com.invalid> >>> >>> a >>> >>> >>> écrit : >>> >>> >>> >>> If Gradle was slow, do you >>> mind >>> running the build with >>> >>> >>> --profile >>> >>> >>> and >>> >>> >>> sharing that and also sharing the >>> Maven >>> build log? >>> >>> >>> On Thu, Nov 9, 2017 at >>> 10:43 AM, >>> Lukasz Cwik < >>> >>> lc...@google.com >>> <mailto:lc...@google.com>> >>> >>> >>> wrote: >>> >>> >>> >>> Romain, I don't understand >>> your >>> last comment, were you >>> >>> >>> trying >>> >>> >>> to >>> >>> say >>> >>> >>> that >>> >>> >>> you had the same Gradle >>> build >>> times like I did and it >>> was an >>> >>> >>> improvement >>> >>> >>> over Maven or that you did >>> not >>> and you experienced build >>> >>> >>> times >>> >>> >>> that >>> >>> >>> were >>> >>> >>> equivalent to Maven? >>> >>> >>> On Thu, Nov 9, 2017 at >>> 9:51 >>> AM, Romain Manni-Bucau >>> < >>> >>> rmannibu...@gmail.com >>> >>> <mailto:rmannibu...@gmail.com>> >>> >>> wrote: >>> >>> 2017-11-09 18:38 >>> GMT+01:00 >>> Kenneth Knowles >>> >>> >>> >>> <k...@google.com.invalid >>> >>> >>> : >>> >>> >>> On Thu, Nov 9, 2017 at >>> 9:11 AM, >>> Romain Manni-Bucau < >>> >>> >>> >>> rmannibu...@gmail.com >>> >>> <mailto:rmannibu...@gmail.com>> >>> >>> wrote: >>> >>> (this is >>> another >>> topic so we >>> can >>> maybe open >>> another >>> >>> >>> thread) >>> >>> >>> issue >>> >>> >>> is >>> >>> >>> not much about python but >>> more >>> about the fact the build >>> >>> >>> is >>> >>> >>> not >>> >>> >>> self >>> >>> >>> contained. it is a maven build >>> and >>> maven should be >>> >>> >>> sufficient >>> >>> >>> without >>> >>> >>> having to install python + >>> dependencies. >>> >>> >>> >>> >>> Let's leave >>> out the >>> topic of >>> whether our >>> build should >>> >>> install >>> >>> >>> things >>> >>> >>> like >>> >>> >>> JDKs, Python, >>> Golang, >>> Docker, >>> protoc, >>> findbugs, >>> RAT, etc. >>> >>> That >>> >>> >>> issue >>> >>> >>> is >>> >>> >>> somewhat independent >>> of >>> build tool, and the >>> new build >>> >>> >>> isn't >>> >>> >>> worse >>> >>> >>> than >>> >>> >>> the >>> >>> >>> old one as far >>> as it >>> goes. >>> >>> >>> >>> Yep, globally the >>> same >>> time with clean >>> and >>> killing the >>> >>> daemon. >>> >>> >>> >>> >>> Kenn >>> >>> >>> >>> I don't see >>> any >>> technical >>> >>> blockers >>> to do >>> it (except >>> time >>> ;)) but it >>> is >>> always a >>> >>> bit >>> >>> >>> annoying >>> >>> >>> to >>> >>> git clone then not be able >>> to build. >>> >>> >>> Romain >>> Manni-Bucau >>> >>> @rmannibucau | >>> Blog | Old Blog >>> | Github | >>> LinkedIn >>> >>> >>> 2017-11-09 >>> 18:07 >>> GMT+01:00 >>> Lukasz >>> Cwik >>> >>> >>> <lc...@google.com.invalid >>> >>> >>> : >>> >>> >>> Hmm, I have had good luck >>> when >>> following the Python >>> >>> >>> quick >>> >>> >>> start >>> >>> >>> setup >>> >>> >>> >>> <https://beam.apache.org/get-started/quickstart-py/ >>> >>> <https://beam.apache.org/get-started/quickstart-py/>> >>> >>> >>> on >>> >>> >>> multiple >>> >>> >>> machines >>> >>> >>> by >>> ensuring >>> the >>> >>> installed >>> >>> version of >>> >>> setuptools, >>> >>> virtualenv >>> >>> >>> and >>> >>> >>> pip >>> >>> >>> are >>> >>> >>> new >>> >>> >>> enough >>> versions. >>> >>> You >>> can >>> always >>> skip >>> the >>> Python >>> >>> portion of >>> the >>> build by >>> >>> excluding >>> >>> >>> the >>> >>> >>> build >>> >>> >>> task >>> as so: >>> >>> ./gradlew >>> build >>> -x >>> >>> ":beam-sdks-parent:beam-sdks- >>> >>> >>> python:build" >>> >>> >>> >>> On >>> Thu, Nov >>> 9, >>> 2017 at >>> 8:58 >>> AM, >>> Romain >>> >>> Manni-Bucau < >>> >>> >>> rmannibu...@gmail.com >>> >>> <mailto:rmannibu...@gmail.com>> >>> >>> wrote: >>> >>> The >>> 1.3.5 >>> file >>> is when >>> i >>> installed >>> the >>> python >>> >>> >>> >>> dependencies >>> >>> >>> manually >>> >>> >>> to make the build >>> passing >>> (the pip command never >>> >>> >>> passed >>> >>> >>> on >>> >>> my >>> >>> >>> computer >>> >>> >>> and therefore >>> the >>> build always >>> has >>> been broken >>> until >>> >>> >>> i >>> >>> >>> installed >>> >>> >>> it >>> >>> >>> manually - >>> independently >>> from the build tool). >>> >>> >>> >>> Romain >>> >>> Manni-Bucau >>> >>> @rmannibucau >>> | >>> Blog >>> | >>> Old >>> >>> Blog | >>> >>> Github | >>> >>> LinkedIn >>> >>> >>> >>> 2017-11-09 >>> >>> 17:51 >>> >>> GMT+01:00 Lukasz >>> >>> Cwik >>> >>> >>> <lc...@google.com.invalid >>> >>> >>> : >>> >>> >>> It turns out that >>> the >>> Apache Rat Ant >>> task and the >>> >>> >>> >>> Apache >>> >>> >>> Rat >>> >>> >>> Maven >>> >>> >>> plugin >>> >>> >>> differ >>> in >>> that >>> the >>> plugin >>> >>> automatically excludes >>> >>> >>> >>> certain >>> >>> >>> files >>> >>> >>> by >>> >>> >>> default >>> >>> >>> while >>> the >>> Ant >>> task >>> does >>> not. >>> >>> >>> See: >>> >>> http://creadur.apache.org/rat/ >>> >>> >>> apache-rat-plugin/check-mojo. >>> >>> >>> html#useDefaultExcludes >>> >>> >>> >>> >>> I >>> >>> fixed the >>> >>> list >>> >>> to >>> >>> exclude >>> >>> ".idea/" >>> >>> instead >>> >>> of >>> >>> >>> "idea/" >>> >>> >>> since >>> >>> >>> there >>> >>> >>> was a >>> >>> >>> typo. >>> >>> >>> >>> I >>> >>> have >>> >>> no >>> >>> idea >>> >>> what >>> >>> the >>> >>> file >>> >>> "=1.3.5" >>> >>> is. >>> >>> Can you >>> >>> >>> take a >>> >>> >>> look >>> >>> >>> at >>> >>> the >>> >>> >>> contents? >>> >>> >>> >>> On >>> >>> Thu, >>> >>> Nov >>> >>> 9, >>> >>> 2017 >>> >>> at >>> >>> 12:03 AM, >>> >>> Romain >>> >>> >>> Manni-Bucau >>> < >>> >>> >>> rmannibu...@gmail.com >>> <mailto:rmannibu...@gmail.com>> >>> >>> >>> >>> wrote: >>> >>> >>> Ok, >>> >>> the >>> >>> rat >>> >>> issues >>> >>> I >>> >>> got >>> >>> were: >>> >>> >>> >>> == >>> >>> File: >>> >>> /home/rmannibucau/1_dev/beam/.idea/* >>> >>> == >>> >>> File: >>> >>> /home/rmannibucau/1_dev/beam/ >>> >>> >>> sdks/python/=1.3.5 >>> >>> >>> >>> >>> The >>> >>> first >>> >>> one >>> >>> could >>> >>> be >>> >>> in >>> >>> my >>> >>> default >>> >>> exclude >>> >>> - >>> >>> >>> even >>> >>> >>> if >>> >>> eclipse/idea >>> >>> >>> files should >>> be in >>> the default >>> exclude >>> set of beam >>> >>> >>> >>> rat >>> >>> >>> config >>> >>> >>> IMHO, >>> >>> >>> the last one >>> is more >>> a "?" can >>> probably be >>> >>> >>> >>> exclude >>> >>> as >>> >>> >>> well >>> >>> >>> if >>> >>> created >>> >>> >>> by the build >>> at some >>> point. >>> >>> >>> >>> >>> Romain >>> >>> Manni-Bucau >>> >>> @rmannibucau >>> >>> | >>> Blog >>> >>> | Old >>> >>> Blog >>> >>> | Github >>> >>> | >>> >>> >>> LinkedIn >>> >>> >>> >>> >>> >>> 2017-11-08 >>> >>> 19:17 >>> >>> GMT+01:00 >>> >>> Jean-Baptiste >>> >>> Onofré >>> >>> < >>> >>> >>> j...@nanthrax.net >>> >>> <mailto:j...@nanthrax.net> >>> >>> >>> : >>> >>> >>> Thanks for >>> the >>> update. I >>> was >>> swamped on >>> some >>> >>> >>> >>> meetings. >>> >>> >>> I'm >>> >>> >>> back to >>> >>> >>> test >>> >>> >>> >>> the >>> >>> latest >>> >>> changes. >>> >>> >>> >>> >>> Regards >>> >>> JB
