Romain, as mentioned earlier, I identified that Maven was slower because it needed to finish building the entire module before dependent modules could start which included running tests, performing checkstyle, etc... Gradle is able to increase the parallelism of the build process since it has task driven parallelism so as long as the files are compiled, the dependent projects can start.
Maven and Gradle are both heavily used since there are ~146k Maven projects on Github while there are ~122k Gradle project on Github. Do you have data which shows that Maven is significantly more "mainstream"? I believe we want a rich multi-language SDK and community and feel as though it would be unwise to treat non JDK based languages as second class. On Mon, Nov 27, 2017 at 11:00 AM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > Hi Lukasz, > > Did you manage to identify how maven was slower and test tesla stuff > and potentially a few other fixes? > > Side note: figures without python can be interesting cause locally - = > for me - python tends to flatten the figures whereas I get something > close to your conclusions without python part. > > My point is mainly that switching now on gradle and being back on > maven in a few months cause gradle ecosystem is far to support java 9 > - or any other volatile reason like this one - is probably not a good > choice for a community. Maven is way more mainstream than gradle so > helps to encourage people to contribute - vs gradle will increase the > step to do it. > > I'd like to be sure before a switch that it is a one way decision and > that the build tool was not just challenged by itself and its current > state but also in the way it could be improved (= its community and > potentially some local hacks). > > Romain Manni-Bucau > @rmannibucau | Blog | Old Blog | Github | LinkedIn > > > 2017-11-27 19:46 GMT+01:00 Lukasz Cwik <lc...@google.com.invalid>: > > I have collected data by running several builds against master using > Gradle > > and Maven without using Gradle's support for incremental builds. > > > > Gradle (mins) > > min: 25.04 > > max: 160.14 > > median: 45.78 > > average: 52.19 > > stdev: 30.80 > > > > Maven (mins) > > min: 56.86 > > max: 216.55 (actually > 240 mins because this data does not include > > timeouts) > > median: 87.93 > > average: 109.10 > > stdev: 48.01 > > > > I excluded a few timeouts (240 mins) that happened during the Maven build > > from its numbers but we can see conclusively that Gradle is twice as fast > > for the build when compared to Maven when run using Jenkins. > > On my desktop, I have enabled incremental builds and have seen a major > > improvement on the above numbers but it doesn't yet work correctly > because > > of incorrectly specified inputs/outputs for certain tasks. > > > > The data is available here > > https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_ > NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing > > > > With this data, I feel confident that we should swap and have opened the > > following issue https://issues.apache.org/jira/browse/BEAM-3249 and > related > > sub-tasks. > > > > On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré <j...@nanthrax.net> > > wrote: > > > >> Thanks for the update Luke. > >> > >> I'm updating my local working copy to do new tests. > >> > >> Regards > >> JB > >> > >> On 11/19/2017 08:21 PM, Lukasz Cwik wrote: > >> > >>> The gradle build rules have been merged, I'm adding a precommit[1] to > >>> start > >>> collecting data about the build times. It currently only mirrors the > Java > >>> mvn install precommit. I'll gather data over the next two weeks and > >>> provide > >>> a summary here. > >>> > >>> You can rerun the precommit by issuing "Run Java Gradle PreCommit" > >>> > >>> 1: https://github.com/apache/beam/pull/4146 > >>> > >>> > >>> On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik <lc...@google.com> wrote: > >>> > >>> There has been plenty of time for comments on the PR and the approach. > >>>> > >>>> So far Ken Knowles has provided the most feedback on the PR, Ken would > >>>> you > >>>> like to finish the review? > >>>> > >>>> > >>>> > >>>> On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau < > >>>> rmannibu...@gmail.com > >>>> > >>>>> wrote: > >>>>> > >>>> > >>>> This is only a setup thing and better to not break the master history > for > >>>>> poc/tests, in particular when no very localized. Alternative can be > to > >>>>> ask > >>>>> another temp repo to infra and have a synchro between both but dont > >>>>> think > >>>>> it does worth it personally. > >>>>> > >>>>> > >>>>> > >>>>> Le 10 nov. 2017 18:57, "Lukasz Cwik" <lc...@google.com.invalid> a > >>>>> écrit : > >>>>> > >>>>> The reason to get it on master is because that is where all the PRs > >>>>>> > >>>>> are. An > >>>>> > >>>>>> upstream branch without any development means no data. > >>>>>> Also, our Jenkins setup via job-dsl doesn't honor using the Jenkins > >>>>>> configuration on the branch because the seed job always runs against > >>>>>> master. > >>>>>> > >>>>>> On Thu, Nov 9, 2017 at 9:59 PM, Romain Manni-Bucau < > >>>>>> > >>>>> rmannibu...@gmail.com> > >>>>> > >>>>>> wrote: > >>>>>> > >>>>>> What about pushing it on a "upstream" branch and testing it for 1 > >>>>>>> > >>>>>> week in > >>>>> > >>>>>> parallel of the maven reference build? If gradle is always 50% > faster > >>>>>>> > >>>>>> on > >>>>> > >>>>>> jenkins then it could become master setup without much discussion I > >>>>>>> > >>>>>> guess. > >>>>>> > >>>>>>> We can even have 2 jenkins jobs: one with the daemon etc and one > >>>>>>> > >>>>>> without. > >>>>> > >>>>>> > >>>>>>> Also noticed yesterday that gradle build is killing my machine > (all 8 > >>>>>>> > >>>>>> cores > >>>>>> > >>>>>>> are 100%) during the first minutes vs maven build which let me do > >>>>>>> > >>>>>> something > >>>>>> > >>>>>>> else. Then all the consumed time which makes gradle not that fast > is > >>>>>>> > >>>>>> about > >>>>>> > >>>>>>> python. Will try to send figures later today. > >>>>>>> > >>>>>>> Le 10 nov. 2017 00:10, "Lukasz Cwik" <lc...@google.com.invalid> a > >>>>>>> > >>>>>> écrit > >>>>> > >>>>>> : > >>>>>> > >>>>>>> > >>>>>>> I wouldn't mind merging this change in so I could setup those > Gradle > >>>>>>>> Jenkins precommits. > >>>>>>>> > >>>>>>>> As per our contribution guidelines, any committer willing to sign > >>>>>>>> > >>>>>>> off > >>>>> > >>>>>> on > >>>>>> > >>>>>>> the PR? > >>>>>>>> > >>>>>>>> On Thu, Nov 9, 2017 at 2:12 PM, Romain Manni-Bucau < > >>>>>>>> > >>>>>>> rmannibu...@gmail.com> > >>>>>>> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>> Le 9 nov. 2017 21:31, "Kenneth Knowles" <k...@google.com.invalid> > >>>>>>>>> > >>>>>>>> a > >>>>> > >>>>>> écrit : > >>>>>>>> > >>>>>>>>> > >>>>>>>>> Keep in mind that a clean build is unusual during development (it > >>>>>>>>> > >>>>>>>> is > >>>>> > >>>>>> common > >>>>>>>> > >>>>>>>>> for mvn use and that is a bug) and also not necessary for > >>>>>>>>> > >>>>>>>> precommits > >>>>> > >>>>>> if > >>>>>> > >>>>>>> the > >>>>>>>> > >>>>>>>>> build tool is correct enough that caching is safe. So while this > >>>>>>>>> > >>>>>>>> number > >>>>>> > >>>>>>> matters, it is not the most important. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Not sure, in dev you bypass the build tool most of the time > >>>>>>>>> > >>>>>>>> anyway - > >>>>> > >>>>>> thanks > >>>>>>>> > >>>>>>>>> to IDE or other shortcuts - but not on PR and CI. Keep in mind > >>>>>>>>> > >>>>>>>> that > >>>>> > >>>>>> not > >>>>>> > >>>>>>> doing a clean and killing gradle daemon makes the build not > >>>>>>>>> > >>>>>>>> reproducible > >>>>>>> > >>>>>>>> and therefore useful :(. Starting to build from a subpart of the > >>>>>>>>> > >>>>>>>> reactor > >>>>>>> > >>>>>>>> - > >>>>>>>> > >>>>>>>>> with the mentionned mvn plugin for instance - can be nice on some > >>>>>>>>> > >>>>>>>> CI > >>>>> > >>>>>> like > >>>>>>> > >>>>>>>> travis if the caching is well configured but still not a guarantee > >>>>>>>>> > >>>>>>>> the > >>>>>> > >>>>>>> build is "green". > >>>>>>>>> > >>>>>>>>> My trade off is to ensure an easy build and relevant result over > >>>>>>>>> > >>>>>>>> the > >>>>> > >>>>>> time > >>>>>>> > >>>>>>>> criteria. Do you share it as well or prefer time over other > >>>>>>>>> > >>>>>>>> criteria > >>>>> > >>>>>> - > >>>>>> > >>>>>>> which leads to other conclusions and options indeed and can make > >>>>>>>>> > >>>>>>>> us > >>>>> > >>>>>> not > >>>>>> > >>>>>>> understanding each other? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Thu, Nov 9, 2017 at 11:30 AM, Romain Manni-Bucau < > >>>>>>>>> > >>>>>>>> rmannibu...@gmail.com > >>>>>>>> > >>>>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> I will try next week yes but the 2 runs i did were 28mn vs 32mn > >>>>>>>>>> > >>>>>>>>> from > >>>>>> > >>>>>>> memory > >>>>>>>>> > >>>>>>>>>> - after having downloaded all deps once. > >>>>>>>>>> > >>>>>>>>>> Le 9 nov. 2017 19:45, "Lukasz Cwik" <lc...@google.com.invalid> > >>>>>>>>>> > >>>>>>>>> a > >>>>> > >>>>>> écrit : > >>>>>>>> > >>>>>>>>> > >>>>>>>>>> If Gradle was slow, do you mind running the build with > >>>>>>>>>>> > >>>>>>>>>> --profile > >>>>> > >>>>>> and > >>>>>>> > >>>>>>>> sharing that and also sharing the Maven build log? > >>>>>>>>>>> > >>>>>>>>>>> On Thu, Nov 9, 2017 at 10:43 AM, Lukasz Cwik < > >>>>>>>>>>> > >>>>>>>>>> lc...@google.com> > >>>>> > >>>>>> wrote: > >>>>>>>> > >>>>>>>>> > >>>>>>>>>>> Romain, I don't understand your last comment, were you > >>>>>>>>>>>> > >>>>>>>>>>> trying > >>>>> > >>>>>> to > >>>>>> > >>>>>>> say > >>>>>>>> > >>>>>>>>> that > >>>>>>>>>> > >>>>>>>>>>> you had the same Gradle build times like I did and it was an > >>>>>>>>>>>> > >>>>>>>>>>> improvement > >>>>>>>>>> > >>>>>>>>>>> over Maven or that you did not and you experienced build > >>>>>>>>>>>> > >>>>>>>>>>> times > >>>>> > >>>>>> that > >>>>>>> > >>>>>>>> were > >>>>>>>>>> > >>>>>>>>>>> equivalent to Maven? > >>>>>>>>>>>> > >>>>>>>>>>>> On Thu, Nov 9, 2017 at 9:51 AM, Romain Manni-Bucau < > >>>>>>>>>>>> > >>>>>>>>>>> rmannibu...@gmail.com> > >>>>>>>>>>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> 2017-11-09 18:38 GMT+01:00 Kenneth Knowles > >>>>>>>>>>>>> > >>>>>>>>>>>> <k...@google.com.invalid > >>>>>>> > >>>>>>>> : > >>>>>>>>>> > >>>>>>>>>>> On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau < > >>>>>>>>>>>>>> > >>>>>>>>>>>>> rmannibu...@gmail.com> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> (this is another topic so we can maybe open another > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> thread) > >>>>> > >>>>>> issue > >>>>>>>> > >>>>>>>>> is > >>>>>>>>>> > >>>>>>>>>>> not much about python but more about the fact the build > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> is > >>>>> > >>>>>> not > >>>>>>> > >>>>>>>> self > >>>>>>>>> > >>>>>>>>>> contained. it is a maven build and maven should be > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> sufficient > >>>>>> > >>>>>>> without > >>>>>>>>>> > >>>>>>>>>>> having to install python + dependencies. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Let's leave out the topic of whether our build should > >>>>>>>>>>>>>> > >>>>>>>>>>>>> install > >>>>>> > >>>>>>> things > >>>>>>>>> > >>>>>>>>>> like > >>>>>>>>>>>>> > >>>>>>>>>>>>>> JDKs, Python, Golang, Docker, protoc, findbugs, RAT, etc. > >>>>>>>>>>>>>> > >>>>>>>>>>>>> That > >>>>>> > >>>>>>> issue > >>>>>>>>> > >>>>>>>>>> is > >>>>>>>>>>> > >>>>>>>>>>>> somewhat independent of build tool, and the new build > >>>>>>>>>>>>>> > >>>>>>>>>>>>> isn't > >>>>> > >>>>>> worse > >>>>>>>> > >>>>>>>>> than > >>>>>>>>>> > >>>>>>>>>>> the > >>>>>>>>>>>>> > >>>>>>>>>>>>>> old one as far as it goes. > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Yep, globally the same time with clean and killing the > >>>>>>>>>>>>> > >>>>>>>>>>>> daemon. > >>>>> > >>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Kenn > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I don't see any technical > >>>>>>>>>>>>>>> blockers to do it (except time ;)) but it is always a > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> bit > >>>>> > >>>>>> annoying > >>>>>>>>> > >>>>>>>>>> to > >>>>>>>>>> > >>>>>>>>>>> git clone then not be able to build. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Romain Manni-Bucau > >>>>>>>>>>>>>>> @rmannibucau | Blog | Old Blog | Github | LinkedIn > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> 2017-11-09 18:07 GMT+01:00 Lukasz Cwik > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> <lc...@google.com.invalid > >>>>>>>> > >>>>>>>>> : > >>>>>>>>>> > >>>>>>>>>>> Hmm, I have had good luck when following the Python > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> quick > >>>>> > >>>>>> start > >>>>>>>> > >>>>>>>>> setup > >>>>>>>>>>> > >>>>>>>>>>>> <https://beam.apache.org/get-started/quickstart-py/> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> on > >>>>> > >>>>>> multiple > >>>>>>>>> > >>>>>>>>>> machines > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> by ensuring the installed version of setuptools, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> virtualenv > >>>>>> > >>>>>>> and > >>>>>>>> > >>>>>>>>> pip > >>>>>>>>>> > >>>>>>>>>>> are > >>>>>>>>>>>>> > >>>>>>>>>>>>>> new > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> enough versions. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> You can always skip the Python portion of the build by > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> excluding > >>>>>>>>> > >>>>>>>>>> the > >>>>>>>>>>> > >>>>>>>>>>>> build > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> task as so: > >>>>>>>>>>>>>>>> ./gradlew build -x ":beam-sdks-parent:beam-sdks- > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> python:build" > >>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>>> On Thu, Nov 9, 2017 at 8:58 AM, Romain Manni-Bucau < > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> rmannibu...@gmail.com> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> The 1.3.5 file is when i installed the python > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> dependencies > >>>>>> > >>>>>>> manually > >>>>>>>>>>> > >>>>>>>>>>>> to make the build passing (the pip command never > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> passed > >>>>> > >>>>>> on > >>>>>> > >>>>>>> my > >>>>>>>> > >>>>>>>>> computer > >>>>>>>>>>>>> > >>>>>>>>>>>>>> and therefore the build always has been broken until > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> i > >>>>> > >>>>>> installed > >>>>>>>>> > >>>>>>>>>> it > >>>>>>>>>>> > >>>>>>>>>>>> manually - independently from the build tool). > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Romain Manni-Bucau > >>>>>>>>>>>>>>>>> @rmannibucau | Blog | Old Blog | Github | LinkedIn > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 2017-11-09 17:51 GMT+01:00 Lukasz Cwik > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> <lc...@google.com.invalid > >>>>>>>>> > >>>>>>>>>> : > >>>>>>>>>>>> > >>>>>>>>>>>>> It turns out that the Apache Rat Ant task and the > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Apache > >>>>>> > >>>>>>> Rat > >>>>>>>> > >>>>>>>>> Maven > >>>>>>>>>>> > >>>>>>>>>>>> plugin > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> differ in that the plugin automatically excludes > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> certain > >>>>>> > >>>>>>> files > >>>>>>>>> > >>>>>>>>>> by > >>>>>>>>>>> > >>>>>>>>>>>> default > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> while the Ant task does not. > >>>>>>>>>>>>>>>>>> See: > >>>>>>>>>>>>>>>>>> http://creadur.apache.org/rat/ > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> apache-rat-plugin/check-mojo. > >>>>>>>> > >>>>>>>>> html#useDefaultExcludes > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I fixed the list to exclude ".idea/" instead of > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> "idea/" > >>>>>> > >>>>>>> since > >>>>>>>>> > >>>>>>>>>> there > >>>>>>>>>>>>> > >>>>>>>>>>>>>> was a > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> typo. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I have no idea what the file "=1.3.5" is. Can you > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> take a > >>>>>> > >>>>>>> look > >>>>>>>>> > >>>>>>>>>> at > >>>>>>>>>> > >>>>>>>>>>> the > >>>>>>>>>>>>> > >>>>>>>>>>>>>> contents? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Thu, Nov 9, 2017 at 12:03 AM, Romain > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Manni-Bucau < > >>>>> > >>>>>> rmannibu...@gmail.com> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Ok, the rat issues I got were: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> == File: /home/rmannibucau/1_dev/beam/.idea/* > >>>>>>>>>>>>>>>>>>> == File: /home/rmannibucau/1_dev/beam/ > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> sdks/python/=1.3.5 > >>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>>>>>> The first one could be in my default exclude - > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> even > >>>>> > >>>>>> if > >>>>>> > >>>>>>> eclipse/idea > >>>>>>>>>>>>> > >>>>>>>>>>>>>> files should be in the default exclude set of beam > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> rat > >>>>>> > >>>>>>> config > >>>>>>>>> > >>>>>>>>>> IMHO, > >>>>>>>>>>>>> > >>>>>>>>>>>>>> the last one is more a "?" can probably be > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> exclude as > >>>>> > >>>>>> well > >>>>>>>> > >>>>>>>>> if > >>>>>>>>> > >>>>>>>>>> created > >>>>>>>>>>>>> > >>>>>>>>>>>>>> by the build at some point. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Romain Manni-Bucau > >>>>>>>>>>>>>>>>>>> @rmannibucau | Blog | Old Blog | Github | > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> LinkedIn > >>>>> > >>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 2017-11-08 19:17 GMT+01:00 Jean-Baptiste Onofré < > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> j...@nanthrax.net > >>>>>>>>>>> > >>>>>>>>>>>> : > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks for the update. I was swamped on some > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> meetings. > >>>>>>> > >>>>>>>> I'm > >>>>>>>>> > >>>>>>>>>> back to > >>>>>>>>>>>>> > >>>>>>>>>>>>>> test > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> the latest changes. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Regards > >>>>>>>>>>>>>>>>>>>> JB > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On Nov 8, 2017, 18:56, at 18:56, Lukasz Cwik > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> <lc...@google.com.INVALID > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Thanks everyone for trying this build out in > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> different > >>>>>>> > >>>>>>>> workspaces / > >>>>>>>>>>>>> > >>>>>>>>>>>>>> configurations. This will help make sure the > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> build > >>>>> > >>>>>> works > >>>>>>>> > >>>>>>>>> for > >>>>>>>>>> > >>>>>>>>>>> more > >>>>>>>>>>>>> > >>>>>>>>>>>>>> people > >>>>>>>>>>>>>>>>>>>>> and will get rid of any rough edges. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Performance (All): > >>>>>>>>>>>>>>>>>>>>> Maven performs parallelization at the module > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> level, > >>>>> > >>>>>> an > >>>>>>> > >>>>>>>> entire > >>>>>>>>>> > >>>>>>>>>>> module > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> needs > >>>>>>>>>>>>>>>>>>>>> to complete before any dependent modules can > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> start, > >>>>> > >>>>>> this > >>>>>>>> > >>>>>>>>> means > >>>>>>>>>>> > >>>>>>>>>>>> running > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> all > >>>>>>>>>>>>>>>>>>>>> the checks like findbugs, checkstyle, tests > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> need to > >>>>> > >>>>>> finish. > >>>>>>>>> > >>>>>>>>>> Gradle > >>>>>>>>>>>>> > >>>>>>>>>>>>>> has > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> task > >>>>>>>>>>>>>>>>>>>>> level parallelism between subprojects which > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> means > >>>>> > >>>>>> that > >>>>>>> > >>>>>>>> as > >>>>>>>> > >>>>>>>>> soon > >>>>>>>>>>> > >>>>>>>>>>>> as > >>>>>>>>>>>>> > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> compile and shade steps are done for a project, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> and > >>>>> > >>>>>> dependent > >>>>>>>>>> > >>>>>>>>>>> subprojects > >>>>>>>>>>>>>>>>>>>>> can typically start. This means that we get > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> increased > >>>>>> > >>>>>>> parallelism > >>>>>>>>>>>>> > >>>>>>>>>>>>>> due > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>> not needing to wait for findbugs, checkstyle, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> tests > >>>>> > >>>>>> to > >>>>>>> > >>>>>>>> run. I > >>>>>>>>>> > >>>>>>>>>>> typically > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> see > >>>>>>>>>>>>>>>>>>>>> ~20 tasks (at peak) running on my desktop in > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> parallel. > >>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Apache Rat (JB / Romain): > >>>>>>>>>>>>>>>>>>>>> What files are in the rat report that fail (its > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> likely > >>>>>>> > >>>>>>>> that > >>>>>>>>> > >>>>>>>>>> I'm > >>>>>>>>>>> > >>>>>>>>>>>> missing > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> some exclusion for a build time artifact)? Also, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> please > >>>>>>> > >>>>>>>> try > >>>>>>>>> > >>>>>>>>>> the > >>>>>>>>>>> > >>>>>>>>>>>> build > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> again > >>>>>>>>>>>>>>>>>>>>> after running `git clean -fdx` in your > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> workspace. > >>>>> > >>>>>> > >>>>>>>>>>>>>>>>>>>>> Python (JB): > >>>>>>>>>>>>>>>>>>>>> As for the Python SDK, you'll need to share more > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> details > >>>>>>>> > >>>>>>>>> about > >>>>>>>>>>> > >>>>>>>>>>>> the > >>>>>>>>>>>>> > >>>>>>>>>>>>>> failure. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Gradle 4.3: > >>>>>>>>>>>>>>>>>>>>> I would like to defer the swap to Gradle 4.3 > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> until > >>>>> > >>>>>> after > >>>>>>>> > >>>>>>>>> this > >>>>>>>>>> > >>>>>>>>>>> PR > >>>>>>>>>>>>> > >>>>>>>>>>>>>> since > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>> will be a much smaller set of changes. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> On Wed, Nov 8, 2017 at 12:54 AM, Jean-Baptiste > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Onofré < > >>>>>>> > >>>>>>>> j...@nanthrax.net> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Same for me for rat and python build too: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> FAILURE: Build completed with 2 failures. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 1: Task failed with an exception. > >>>>>>>>>>>>>>>>>>>>>> ----------- > >>>>>>>>>>>>>>>>>>>>>> * What went wrong: > >>>>>>>>>>>>>>>>>>>>>> Execution failed for task ':rat'. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Found 905 files with unapproved/unknown > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> licenses. > >>>>>> > >>>>>>> See > >>>>>>>> > >>>>>>>>> file:/home/jbonofre/Workspace/ > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> beam/build/reports/rat/rat- > >>>>>>>>>> > >>>>>>>>>>> report.txt > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> * Try: > >>>>>>>>>>>>>>>>>>>>>> Run with --stacktrace option to get the stack > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> trace. > >>>>>>> > >>>>>>>> Run > >>>>>>>>> > >>>>>>>>>> with > >>>>>>>>>>> > >>>>>>>>>>>> --info > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> or > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> --debug option to get more log output. > >>>>>>>>>>>>>>>>>>>>>> ============================== > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> ============================== > >>>>>>>>>>> > >>>>>>>>>>>> ================== > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 2: Task failed with an exception. > >>>>>>>>>>>>>>>>>>>>>> ----------- > >>>>>>>>>>>>>>>>>>>>>> * Where: > >>>>>>>>>>>>>>>>>>>>>> Build file '/home/jbonofre/Workspace/ > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> beam/sdks/python/build.gradle' > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> line: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 64 > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> * What went wrong: > >>>>>>>>>>>>>>>>>>>>>> Execution failed for task > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> ':beam-sdks-parent:beam-sdks- > >>>>>>>> > >>>>>>>>> python:lint'. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Process 'command 'tox'' finished with > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> >
