Guys, just realized a lot of modules have threadCount=4 or so in the surefire/failsafe config. It makes it impossible to adapt the parallelism to the machine and therefore makes the parallelism inadapted and useless. Can it be a variable at least? -T1C (or -T2C) should allow to be smoother and aligned with the config machine.
Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn 2017-11-28 9:53 GMT+01:00 Romain Manni-Bucau <[email protected]>: > Just to answer a previous question: > > An ASF github search gives me these stats: > > - mvn (org:apache apache filename:pom.xml path:/): 731 > - gradle (org:apache apache filename:build.gradle path:/): 31 > > Which is consistent with what I saw in enterprises and private repos. > So way different from the whole github stats which is not done on a > representative sample if not pre-filtered. > > Romain Manni-Bucau > @rmannibucau | Blog | Old Blog | Github | LinkedIn > > > 2017-11-28 9:08 GMT+01:00 Robert Bradshaw <[email protected]>: >> It's great to see all the discussion going on here. >> >> I think it's important to point out that merging a parallel set of >> gradle build scripts is a separate (and much less disruptive) step >> than, say, switching over the default (or even recommended) >> build/release process to use them, let alone removing the maven build >> files entirely. The latter two should definitely be gated by a formal >> vote (each, probably), with the current state the gradle files can >> mostly be ignored by most people. In particular, this is the kind of >> change that needs to be in master to be evaluated--if it's on a branch >> we can't very well see how it impacts presubmits, and most importantly >> people can't try it out for real development. >> >> I agree that the choice of build tool may attract some contributors >> and discourage others. Having builds that are fast, correct, and >> reproducible will probably matter more to potential contributors than >> the particular command to run. While maven can surely be improved, I >> doubt a 2x improvement (and many more times that for incremental >> builds) is low-hanging fruit, and many of the issues seem quite >> fundamental (e.g. all the special treatment we need for NeedsRunner >> tests, and having to do a (global-by-default) mvn install to skip >> tests of dependencies when testing a leaf module). >> >> Getting data on what other apache projects use could be interesting, >> but unless we gather why such choices were made I don't know that it'd >> be that influential once we've established that both tools are widely >> supported generally. >> >> To re-emphasize, we'll continue to produce and publish maven >> artifacts, so our choice of build system won't matter for those only >> using Beam as a dependency. >> >> >> >> On Mon, Nov 27, 2017 at 10:48 PM, Jean-Baptiste Onofré <[email protected]> >> wrote: >>> Yeah, especially, I think it would have been great to have a vote before >>> merging on master. >>> >>> Not a big deal, however, I'm really community focus ;) >>> >>> Regards >>> JB >>> >>> On 11/28/2017 07:36 AM, Reuven Lax wrote: >>>> >>>> Agreed. I thinking having a formal vote before Luke had numbers and >>>> results would have been too early. However now that we have such numbers, >>>> we >>>> should think about having a vote. >>>> >>>> Also, while I disagree with Romain that Gradle is not "enterprise ready" >>>> (it's heavily used by Netflix, LinkedIn, Siemens, and is the default build >>>> framework for Android apps), it would be interesting to see if any other >>>> ASF >>>> projects are using it. I don't think that should not make or break the >>>> decision - we should do what's best for the Beam project, and "everyone >>>> else >>>> is doing something" is rarely a good argument - it will provide good data >>>> points for us to evaluate. >>>> >>>> Reuven >>>> >>>> On Mon, Nov 27, 2017 at 10:23 PM, Jean-Baptiste Onofré <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> Hi Luke, >>>> >>>> just curious (and maybe I missed it): did we do a formal vote to merge >>>> the >>>> gradle build ? >>>> Gradle is now on master, we have some Jira to update the release guide >>>> with >>>> gradle. It's fine, but I remember only a discussion, not a vote. >>>> >>>> In order to embrace the community and avoid to have some contributors >>>> "frustrated" (meaning that "this project doesn't care about >>>> contributor, >>>> they just do whatever they want"), I would have love to see a formal >>>> vote >>>> about Gradle more than just a discussion. >>>> >>>> My $0.01 >>>> >>>> Regards >>>> JB >>>> >>>> On 11/27/2017 07:46 PM, Lukasz Cwik wrote: >>>> >>>> I have collected data by running several builds against master >>>> using Gradle >>>> and Maven without using Gradle's support for incremental builds. >>>> >>>> Gradle (mins) >>>> min: 25.04 >>>> max: 160.14 >>>> median: 45.78 >>>> average: 52.19 >>>> stdev: 30.80 >>>> >>>> Maven (mins) >>>> min: 56.86 >>>> max: 216.55 (actually > 240 mins because this data does not >>>> include >>>> timeouts) >>>> median: 87.93 >>>> average: 109.10 >>>> stdev: 48.01 >>>> >>>> I excluded a few timeouts (240 mins) that happened during the >>>> Maven build >>>> from its numbers but we can see conclusively that Gradle is twice >>>> as fast >>>> for the build when compared to Maven when run using Jenkins. >>>> On my desktop, I have enabled incremental builds and have seen a >>>> major >>>> improvement on the above numbers but it doesn't yet work correctly >>>> because >>>> of incorrectly specified inputs/outputs for certain tasks. >>>> >>>> The data is available here >>>> >>>> https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing >>>> >>>> <https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing> >>>> >>>> With this data, I feel confident that we should swap and have >>>> opened the >>>> following issue https://issues.apache.org/jira/browse/BEAM-3249 >>>> <https://issues.apache.org/jira/browse/BEAM-3249> and related >>>> sub-tasks. >>>> >>>> On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré >>>> <[email protected] >>>> <mailto:[email protected]>> >>>> wrote: >>>> >>>> Thanks for the update Luke. >>>> >>>> I'm updating my local working copy to do new tests. >>>> >>>> Regards >>>> JB >>>> >>>> On 11/19/2017 08:21 PM, Lukasz Cwik wrote: >>>> >>>> The gradle build rules have been merged, I'm adding a >>>> precommit[1] to >>>> start >>>> collecting data about the build times. It currently only >>>> mirrors >>>> the Java >>>> mvn install precommit. I'll gather data over the next two >>>> weeks and >>>> provide >>>> a summary here. >>>> >>>> You can rerun the precommit by issuing "Run Java Gradle >>>> PreCommit" >>>> >>>> 1: https://github.com/apache/beam/pull/4146 >>>> <https://github.com/apache/beam/pull/4146> >>>> >>>> >>>> On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik >>>> <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> There has been plenty of time for comments on the PR and >>>> the >>>> approach. >>>> >>>> >>>> So far Ken Knowles has provided the most feedback on >>>> the PR, >>>> Ken would >>>> you >>>> like to finish the review? >>>> >>>> >>>> >>>> On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau < >>>> [email protected] <mailto:[email protected]> >>>> >>>> wrote: >>>> >>>> >>>> This is only a setup thing and better to not break the >>>> master history for >>>> >>>> poc/tests, in particular when no very localized. >>>> Alternative can be to >>>> ask >>>> another temp repo to infra and have a synchro >>>> between >>>> both but dont >>>> think >>>> it does worth it personally. >>>> >>>> >>>> >>>> Le 10 nov. 2017 18:57, "Lukasz Cwik" >>>> <[email protected]> a >>>> écrit : >>>> >>>> The reason to get it on master is because that is >>>> where >>>> all the PRs >>>> >>>> >>>> are. An >>>> >>>> upstream branch without any development means >>>> no data. >>>> Also, our Jenkins setup via job-dsl doesn't >>>> honor >>>> using the Jenkins >>>> configuration on the branch because the seed >>>> job >>>> always runs against >>>> master. >>>> >>>> On Thu, Nov 9, 2017 at 9:59 PM, Romain >>>> Manni-Bucau < >>>> >>>> [email protected] >>>> <mailto:[email protected]>> >>>> >>>> wrote: >>>> >>>> What about pushing it on a "upstream" branch >>>> and >>>> testing it for 1 >>>> >>>> >>>> week in >>>> >>>> >>>> parallel of the maven reference build? If >>>> gradle is >>>> always 50% faster >>>> >>>> >>>> on >>>> >>>> >>>> jenkins then it could become master setup >>>> without >>>> much discussion I >>>> >>>> >>>> guess. >>>> >>>> We can even have 2 jenkins jobs: one with >>>> the >>>> daemon etc and one >>>> >>>> without. >>>> >>>> >>>> >>>> Also noticed yesterday that gradle build >>>> is >>>> killing my machine (all 8 >>>> >>>> cores >>>> >>>> are 100%) during the first minutes vs >>>> maven >>>> build which let me do >>>> >>>> something >>>> >>>> else. Then all the consumed time which >>>> makes >>>> gradle not that fast is >>>> >>>> about >>>> >>>> python. Will try to send figures later >>>> today. >>>> >>>> Le 10 nov. 2017 00:10, "Lukasz Cwik" >>>> <[email protected]> a >>>> >>>> écrit >>>> >>>> >>>> : >>>> >>>> >>>> I wouldn't mind merging this change in so >>>> I >>>> could setup those Gradle >>>> >>>> Jenkins precommits. >>>> >>>> As per our contribution guidelines, >>>> any >>>> committer willing to sign >>>> >>>> off >>>> >>>> >>>> on >>>> >>>> the PR? >>>> >>>> >>>> On Thu, Nov 9, 2017 at 2:12 PM, Romain >>>> Manni-Bucau < >>>> >>>> [email protected] >>>> <mailto:[email protected]>> >>>> >>>> wrote: >>>> >>>> Le 9 nov. 2017 21:31, "Kenneth >>>> Knowles" >>>> <[email protected]> >>>> >>>> >>>> a >>>> >>>> >>>> écrit : >>>> >>>> >>>> >>>> Keep in mind that a clean build is >>>> unusual during development (it >>>> >>>> is >>>> >>>> >>>> common >>>> >>>> >>>> for mvn use and that is a bug) and >>>> also >>>> not necessary for >>>> >>>> precommits >>>> >>>> >>>> if >>>> >>>> the >>>> >>>> >>>> build tool is correct enough that >>>> caching is safe. So while this >>>> >>>> number >>>> >>>> >>>> matters, it is not the most important. >>>> >>>> >>>> >>>> Not sure, in dev you bypass the >>>> build >>>> tool most of the time >>>> >>>> anyway - >>>> >>>> >>>> thanks >>>> >>>> >>>> to IDE or other shortcuts - but >>>> not on >>>> PR and CI. Keep in mind >>>> >>>> that >>>> >>>> >>>> not >>>> >>>> doing a clean and killing gradle daemon >>>> makes >>>> the build not >>>> >>>> >>>> reproducible >>>> >>>> >>>> and therefore useful :(. Starting to >>>> build >>>> from a subpart of the >>>> >>>> >>>> reactor >>>> >>>> >>>> - >>>> >>>> with the mentionned mvn plugin for >>>> instance - can be nice on some >>>> >>>> CI >>>> >>>> >>>> like >>>> >>>> >>>> travis if the caching is well >>>> configured but >>>> still not a guarantee >>>> >>>> >>>> the >>>> >>>> >>>> build is "green". >>>> >>>> >>>> My trade off is to ensure an easy >>>> build >>>> and relevant result over >>>> >>>> the >>>> >>>> >>>> time >>>> >>>> >>>> criteria. Do you share it as well or >>>> prefer >>>> time over other >>>> >>>> >>>> criteria >>>> >>>> >>>> - >>>> >>>> which leads to other conclusions and >>>> options >>>> indeed and can make >>>> >>>> >>>> us >>>> >>>> >>>> not >>>> >>>> understanding each other? >>>> >>>> >>>> >>>> On Thu, Nov 9, 2017 at 11:30 AM, >>>> Romain >>>> Manni-Bucau < >>>> >>>> [email protected] >>>> <mailto:[email protected]> >>>> >>>> >>>> wrote: >>>> >>>> >>>> I will try next week yes but the 2 >>>> runs >>>> i did were 28mn vs 32mn >>>> >>>> >>>> from >>>> >>>> >>>> memory >>>> >>>> >>>> - after having downloaded all >>>> deps once. >>>> >>>> Le 9 nov. 2017 19:45, "Lukasz >>>> Cwik" >>>> <[email protected]> >>>> >>>> a >>>> >>>> >>>> écrit : >>>> >>>> >>>> >>>> If Gradle was slow, do you >>>> mind >>>> running the build with >>>> >>>> >>>> --profile >>>> >>>> >>>> and >>>> >>>> >>>> sharing that and also sharing the >>>> Maven >>>> build log? >>>> >>>> >>>> On Thu, Nov 9, 2017 at >>>> 10:43 AM, >>>> Lukasz Cwik < >>>> >>>> [email protected] >>>> <mailto:[email protected]>> >>>> >>>> >>>> wrote: >>>> >>>> >>>> >>>> Romain, I don't understand >>>> your >>>> last comment, were you >>>> >>>> >>>> trying >>>> >>>> >>>> to >>>> >>>> say >>>> >>>> >>>> that >>>> >>>> >>>> you had the same Gradle >>>> build >>>> times like I did and it >>>> was an >>>> >>>> >>>> improvement >>>> >>>> >>>> over Maven or that you did >>>> not >>>> and you experienced build >>>> >>>> >>>> times >>>> >>>> >>>> that >>>> >>>> >>>> were >>>> >>>> >>>> equivalent to Maven? >>>> >>>> >>>> On Thu, Nov 9, 2017 at >>>> 9:51 >>>> AM, Romain Manni-Bucau >>>> < >>>> >>>> [email protected] >>>> >>>> <mailto:[email protected]>> >>>> >>>> wrote: >>>> >>>> 2017-11-09 18:38 >>>> GMT+01:00 >>>> Kenneth Knowles >>>> >>>> >>>> >>>> <[email protected] >>>> >>>> >>>> : >>>> >>>> >>>> On Thu, Nov 9, 2017 at >>>> 9:11 AM, >>>> Romain Manni-Bucau < >>>> >>>> >>>> >>>> [email protected] >>>> >>>> <mailto:[email protected]>> >>>> >>>> wrote: >>>> >>>> (this is >>>> another >>>> topic so we >>>> can >>>> maybe open >>>> another >>>> >>>> >>>> thread) >>>> >>>> >>>> issue >>>> >>>> >>>> is >>>> >>>> >>>> not much about python but >>>> more >>>> about the fact the build >>>> >>>> >>>> is >>>> >>>> >>>> not >>>> >>>> >>>> self >>>> >>>> >>>> contained. it is a maven build >>>> and >>>> maven should be >>>> >>>> >>>> sufficient >>>> >>>> >>>> without >>>> >>>> >>>> having to install python + >>>> dependencies. >>>> >>>> >>>> >>>> >>>> Let's leave >>>> out the >>>> topic of >>>> whether our >>>> build should >>>> >>>> install >>>> >>>> >>>> things >>>> >>>> >>>> like >>>> >>>> >>>> JDKs, Python, >>>> Golang, >>>> Docker, >>>> protoc, >>>> findbugs, >>>> RAT, etc. >>>> >>>> That >>>> >>>> >>>> issue >>>> >>>> >>>> is >>>> >>>> >>>> somewhat independent >>>> of >>>> build tool, and the >>>> new build >>>> >>>> >>>> isn't >>>> >>>> >>>> worse >>>> >>>> >>>> than >>>> >>>> >>>> the >>>> >>>> >>>> old one as far >>>> as it >>>> goes. >>>> >>>> >>>> >>>> Yep, globally the >>>> same >>>> time with clean >>>> and >>>> killing the >>>> >>>> daemon. >>>> >>>> >>>> >>>> >>>> Kenn >>>> >>>> >>>> >>>> I don't see >>>> any >>>> technical >>>> >>>> blockers >>>> to do >>>> it (except >>>> time >>>> ;)) but it >>>> is >>>> always a >>>> >>>> bit >>>> >>>> >>>> annoying >>>> >>>> >>>> to >>>> >>>> git clone then not be able >>>> to build. >>>> >>>> >>>> Romain >>>> Manni-Bucau >>>> >>>> @rmannibucau | >>>> Blog | Old Blog >>>> | Github | >>>> LinkedIn >>>> >>>> >>>> 2017-11-09 >>>> 18:07 >>>> GMT+01:00 >>>> Lukasz >>>> Cwik >>>> >>>> >>>> <[email protected] >>>> >>>> >>>> : >>>> >>>> >>>> Hmm, I have had good luck >>>> when >>>> following the Python >>>> >>>> >>>> quick >>>> >>>> >>>> start >>>> >>>> >>>> setup >>>> >>>> >>>> >>>> <https://beam.apache.org/get-started/quickstart-py/ >>>> >>>> <https://beam.apache.org/get-started/quickstart-py/>> >>>> >>>> >>>> on >>>> >>>> >>>> multiple >>>> >>>> >>>> machines >>>> >>>> >>>> by >>>> ensuring >>>> the >>>> >>>> installed >>>> >>>> version of >>>> >>>> setuptools, >>>> >>>> virtualenv >>>> >>>> >>>> and >>>> >>>> >>>> pip >>>> >>>> >>>> are >>>> >>>> >>>> new >>>> >>>> >>>> enough >>>> versions. >>>> >>>> You >>>> can >>>> always >>>> skip >>>> the >>>> Python >>>> >>>> portion of >>>> the >>>> build by >>>> >>>> excluding >>>> >>>> >>>> the >>>> >>>> >>>> build >>>> >>>> >>>> task >>>> as so: >>>> >>>> ./gradlew >>>> build >>>> -x >>>> >>>> ":beam-sdks-parent:beam-sdks- >>>> >>>> >>>> python:build" >>>> >>>> >>>> >>>> On >>>> Thu, Nov >>>> 9, >>>> 2017 at >>>> 8:58 >>>> AM, >>>> Romain >>>> >>>> Manni-Bucau < >>>> >>>> >>>> [email protected] >>>> >>>> <mailto:[email protected]>> >>>> >>>> wrote: >>>> >>>> The >>>> 1.3.5 >>>> file >>>> is when >>>> i >>>> installed >>>> the >>>> python >>>> >>>> >>>> >>>> dependencies >>>> >>>> >>>> manually >>>> >>>> >>>> to make the build >>>> passing >>>> (the pip command never >>>> >>>> >>>> passed >>>> >>>> >>>> on >>>> >>>> my >>>> >>>> >>>> computer >>>> >>>> >>>> and therefore >>>> the >>>> build always >>>> has >>>> been broken >>>> until >>>> >>>> >>>> i >>>> >>>> >>>> installed >>>> >>>> >>>> it >>>> >>>> >>>> manually - >>>> independently >>>> from the build tool). >>>> >>>> >>>> >>>> Romain >>>> >>>> Manni-Bucau >>>> >>>> @rmannibucau >>>> | >>>> Blog >>>> | >>>> Old >>>> >>>> Blog | >>>> >>>> Github | >>>> >>>> LinkedIn >>>> >>>> >>>> >>>> 2017-11-09 >>>> >>>> 17:51 >>>> >>>> GMT+01:00 Lukasz >>>> >>>> Cwik >>>> >>>> >>>> <[email protected] >>>> >>>> >>>> : >>>> >>>> >>>> It turns out that >>>> the >>>> Apache Rat Ant >>>> task and the >>>> >>>> >>>> >>>> Apache >>>> >>>> >>>> Rat >>>> >>>> >>>> Maven >>>> >>>> >>>> plugin >>>> >>>> >>>> differ >>>> in >>>> that >>>> the >>>> plugin >>>> >>>> automatically excludes >>>> >>>> >>>> >>>> certain >>>> >>>> >>>> files >>>> >>>> >>>> by >>>> >>>> >>>> default >>>> >>>> >>>> while >>>> the >>>> Ant >>>> task >>>> does >>>> not. >>>> >>>> >>>> See: >>>> >>>> http://creadur.apache.org/rat/ >>>> >>>> >>>> apache-rat-plugin/check-mojo. >>>> >>>> >>>> html#useDefaultExcludes >>>> >>>> >>>> >>>> >>>> I >>>> >>>> fixed the >>>> >>>> list >>>> >>>> to >>>> >>>> exclude >>>> >>>> ".idea/" >>>> >>>> instead >>>> >>>> of >>>> >>>> >>>> "idea/" >>>> >>>> >>>> since >>>> >>>> >>>> there >>>> >>>> >>>> was a >>>> >>>> >>>> typo. >>>> >>>> >>>> >>>> I >>>> >>>> have >>>> >>>> no >>>> >>>> idea >>>> >>>> what >>>> >>>> the >>>> >>>> file >>>> >>>> "=1.3.5" >>>> >>>> is. >>>> >>>> Can you >>>> >>>> >>>> take a >>>> >>>> >>>> look >>>> >>>> >>>> at >>>> >>>> the >>>> >>>> >>>> contents? >>>> >>>> >>>> >>>> On >>>> >>>> Thu, >>>> >>>> Nov >>>> >>>> 9, >>>> >>>> 2017 >>>> >>>> at >>>> >>>> 12:03 AM, >>>> >>>> Romain >>>> >>>> >>>> Manni-Bucau >>>> < >>>> >>>> >>>> [email protected] >>>> <mailto:[email protected]>> >>>> >>>> >>>> >>>> wrote: >>>> >>>> >>>> Ok, >>>> >>>> the >>>> >>>> rat >>>> >>>> issues >>>> >>>> I >>>> >>>> got >>>> >>>> were: >>>> >>>> >>>> >>>> == >>>> >>>> File: >>>> >>>> /home/rmannibucau/1_dev/beam/.idea/* >>>> >>>> == >>>> >>>> File: >>>> >>>> /home/rmannibucau/1_dev/beam/ >>>> >>>> >>>> sdks/python/=1.3.5 >>>> >>>> >>>> >>>> >>>> The >>>> >>>> first >>>> >>>> one >>>> >>>> could >>>> >>>> be >>>> >>>> in >>>> >>>> my >>>> >>>> default >>>> >>>> exclude >>>> >>>> - >>>> >>>> >>>> even >>>> >>>> >>>> if >>>> >>>> eclipse/idea >>>> >>>> >>>> files should >>>> be in >>>> the default >>>> exclude >>>> set of beam >>>> >>>> >>>> >>>> rat >>>> >>>> >>>> config >>>> >>>> >>>> IMHO, >>>> >>>> >>>> the last one >>>> is more >>>> a "?" can >>>> probably be >>>> >>>> >>>> >>>> exclude >>>> >>>> as >>>> >>>> >>>> well >>>> >>>> >>>> if >>>> >>>> created >>>> >>>> >>>> by the build >>>> at some >>>> point. >>>> >>>> >>>> >>>> >>>> Romain >>>> >>>> Manni-Bucau >>>> >>>> @rmannibucau >>>> >>>> | >>>> Blog >>>> >>>> | Old >>>> >>>> Blog >>>> >>>> | Github >>>> >>>> | >>>> >>>> >>>> LinkedIn >>>> >>>> >>>> >>>> >>>> >>>> 2017-11-08 >>>> >>>> 19:17 >>>> >>>> GMT+01:00 >>>> >>>> Jean-Baptiste >>>> >>>> Onofré >>>> >>>> < >>>> >>>> >>>> [email protected] >>>> >>>> <mailto:[email protected]> >>>> >>>> >>>> : >>>> >>>> >>>> Thanks for >>>> the >>>> update. I >>>> was >>>> swamped on >>>> some >>>> >>>> >>>> >>>> meetings. >>>> >>>> >>>> I'm >>>> >>>> >>>> back to >>>> >>>> >>>> test >>>> >>>> >>>> >>>> the >>>> >>>> latest >>>> >>>> changes. >>>> >>>> >>>> >>>> >>>> Regards >>>> >>>> JB
