I have been a little bit out of the discussion on maven vs gradle because I was expecting the technical proof of concepts to evaluate the best approach. I deeply appreciate all the effort that Lukasz has put into the gradle version, and I also think that during the discussion Romain and others have bring some serious and important points that make the decision less simple than I expected (in the end sadly is not as simple as the fastest wins). In any case I don’t think that it is wise just to switch immediately to gradle, at least if switching means removing the maven files, we have to consider that the ‘full’ build/tests were introduced in the CI around one week ago, and I am not sure that this is sufficient time to evaluate any possible regression. Also I am particularly curious to know if the artifacts are correct and complete. Has somebody already simulated a release with the gradle build for example, this for me is a prerrequisite before we even start discussing about the switch.
On Mon, Nov 27, 2017 at 9:34 PM, Lukasz Cwik <[email protected]> wrote: > On Mon, Nov 27, 2017 at 11:51 AM, Romain Manni-Bucau <[email protected]> > wrote: > >> 2017-11-27 20:26 GMT+01:00 Lukasz Cwik <[email protected]>: >> > Romain, as mentioned earlier, I identified that Maven was slower because >> it >> > needed to finish building the entire module before dependent modules >> could >> > start which included running tests, performing checkstyle, etc... >> > Gradle is able to increase the parallelism of the build process since it >> > has task driven parallelism so as long as the files are compiled, the >> > dependent projects can start. >> >> This means we can implement a maven graph builder which is better than >> the default one - surely with a thread safe local repo - and >> contribute it back to solve it durably. >> >> If speed for a clean build was the only problem then maybe but lack of > incremental builds across tasks is a goal we can actually achieve using > Gradle and won't require rewriting almost all of the Maven plugins to > support incremental builds. > > >> > >> > Maven and Gradle are both heavily used since there are ~146k Maven >> projects >> > on Github while there are ~122k Gradle project on Github. Do you have >> data >> > which shows that Maven is significantly more "mainstream"? >> >> Yep, project i worked on in companies using gradle: 0, all were based >> on maven and maven was "tool-ed" versus gradle was "best effort" in >> term of plugins. >> Now - with my EE background - I can guarantee you gradle is not able >> to handle properly its build since it flatten the classpath and >> plugins conflicts very quickly (their plugin dependency feature never >> worked and almost no plugin impl it correctly). >> >> Wonder if it is easy to have the ASF stats, anyone knows? > >> >> > I believe we want a rich multi-language SDK and community and feel as >> > though it would be unwise to treat non JDK based languages as second >> class. >> >> Hmm, not sure how it is related to the build tool since Maven and >> Gradle have the same level of support - actually surprsingly maven is >> better for js and surely as bad as gradle for others - or here again >> we can create plugin like the frontend-maven-plugin if needed for >> other languages. >> >> That said it can be an interesting other thread since people consuming >> these languages will probably want their mainstream build tool and a >> "standard" repository layout rather than a java one. But this is >> harder to measure. >> >> >> >> > On Mon, Nov 27, 2017 at 11:00 AM, Romain Manni-Bucau < >> [email protected]> >> > wrote: >> > >> >> Hi Lukasz, >> >> >> >> Did you manage to identify how maven was slower and test tesla stuff >> >> and potentially a few other fixes? >> >> >> >> Side note: figures without python can be interesting cause locally - = >> >> for me - python tends to flatten the figures whereas I get something >> >> close to your conclusions without python part. >> >> >> >> My point is mainly that switching now on gradle and being back on >> >> maven in a few months cause gradle ecosystem is far to support java 9 >> >> - or any other volatile reason like this one - is probably not a good >> >> choice for a community. Maven is way more mainstream than gradle so >> >> helps to encourage people to contribute - vs gradle will increase the >> >> step to do it. >> >> >> >> I'd like to be sure before a switch that it is a one way decision and >> >> that the build tool was not just challenged by itself and its current >> >> state but also in the way it could be improved (= its community and >> >> potentially some local hacks). >> >> >> >> Romain Manni-Bucau >> >> @rmannibucau | Blog | Old Blog | Github | LinkedIn >> >> >> >> >> >> 2017-11-27 19:46 GMT+01:00 Lukasz Cwik <[email protected]>: >> >> > I have collected data by running several builds against master using >> >> Gradle >> >> > and Maven without using Gradle's support for incremental builds. >> >> > >> >> > Gradle (mins) >> >> > min: 25.04 >> >> > max: 160.14 >> >> > median: 45.78 >> >> > average: 52.19 >> >> > stdev: 30.80 >> >> > >> >> > Maven (mins) >> >> > min: 56.86 >> >> > max: 216.55 (actually > 240 mins because this data does not include >> >> > timeouts) >> >> > median: 87.93 >> >> > average: 109.10 >> >> > stdev: 48.01 >> >> > >> >> > I excluded a few timeouts (240 mins) that happened during the Maven >> build >> >> > from its numbers but we can see conclusively that Gradle is twice as >> fast >> >> > for the build when compared to Maven when run using Jenkins. >> >> > On my desktop, I have enabled incremental builds and have seen a major >> >> > improvement on the above numbers but it doesn't yet work correctly >> >> because >> >> > of incorrectly specified inputs/outputs for certain tasks. >> >> > >> >> > The data is available here >> >> > https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_ >> >> NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing >> >> > >> >> > With this data, I feel confident that we should swap and have opened >> the >> >> > following issue https://issues.apache.org/jira/browse/BEAM-3249 and >> >> related >> >> > sub-tasks. >> >> > >> >> > On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré < >> [email protected]> >> >> > wrote: >> >> > >> >> >> Thanks for the update Luke. >> >> >> >> >> >> I'm updating my local working copy to do new tests. >> >> >> >> >> >> Regards >> >> >> JB >> >> >> >> >> >> On 11/19/2017 08:21 PM, Lukasz Cwik wrote: >> >> >> >> >> >>> The gradle build rules have been merged, I'm adding a precommit[1] >> to >> >> >>> start >> >> >>> collecting data about the build times. It currently only mirrors the >> >> Java >> >> >>> mvn install precommit. I'll gather data over the next two weeks and >> >> >>> provide >> >> >>> a summary here. >> >> >>> >> >> >>> You can rerun the precommit by issuing "Run Java Gradle PreCommit" >> >> >>> >> >> >>> 1: https://github.com/apache/beam/pull/4146 >> >> >>> >> >> >>> >> >> >>> On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik <[email protected]> >> wrote: >> >> >>> >> >> >>> There has been plenty of time for comments on the PR and the >> approach. >> >> >>>> >> >> >>>> So far Ken Knowles has provided the most feedback on the PR, Ken >> would >> >> >>>> you >> >> >>>> like to finish the review? >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau < >> >> >>>> [email protected] >> >> >>>> >> >> >>>>> wrote: >> >> >>>>> >> >> >>>> >> >> >>>> This is only a setup thing and better to not break the master >> history >> >> for >> >> >>>>> poc/tests, in particular when no very localized. Alternative can >> be >> >> to >> >> >>>>> ask >> >> >>>>> another temp repo to infra and have a synchro between both but >> dont >> >> >>>>> think >> >> >>>>> it does worth it personally. >> >> >>>>> >> >> >>>>> >> >> >>>>> >> >> >>>>> Le 10 nov. 2017 18:57, "Lukasz Cwik" <[email protected]> a >> >> >>>>> écrit : >> >> >>>>> >> >> >>>>> The reason to get it on master is because that is where all the >> PRs >> >> >>>>>> >> >> >>>>> are. An >> >> >>>>> >> >> >>>>>> upstream branch without any development means no data. >> >> >>>>>> Also, our Jenkins setup via job-dsl doesn't honor using the >> Jenkins >> >> >>>>>> configuration on the branch because the seed job always runs >> against >> >> >>>>>> master. >> >> >>>>>> >> >> >>>>>> On Thu, Nov 9, 2017 at 9:59 PM, Romain Manni-Bucau < >> >> >>>>>> >> >> >>>>> [email protected]> >> >> >>>>> >> >> >>>>>> wrote: >> >> >>>>>> >> >> >>>>>> What about pushing it on a "upstream" branch and testing it for 1 >> >> >>>>>>> >> >> >>>>>> week in >> >> >>>>> >> >> >>>>>> parallel of the maven reference build? If gradle is always 50% >> >> faster >> >> >>>>>>> >> >> >>>>>> on >> >> >>>>> >> >> >>>>>> jenkins then it could become master setup without much >> discussion I >> >> >>>>>>> >> >> >>>>>> guess. >> >> >>>>>> >> >> >>>>>>> We can even have 2 jenkins jobs: one with the daemon etc and one >> >> >>>>>>> >> >> >>>>>> without. >> >> >>>>> >> >> >>>>>> >> >> >>>>>>> Also noticed yesterday that gradle build is killing my machine >> >> (all 8 >> >> >>>>>>> >> >> >>>>>> cores >> >> >>>>>> >> >> >>>>>>> are 100%) during the first minutes vs maven build which let me >> do >> >> >>>>>>> >> >> >>>>>> something >> >> >>>>>> >> >> >>>>>>> else. Then all the consumed time which makes gradle not that >> fast >> >> is >> >> >>>>>>> >> >> >>>>>> about >> >> >>>>>> >> >> >>>>>>> python. Will try to send figures later today. >> >> >>>>>>> >> >> >>>>>>> Le 10 nov. 2017 00:10, "Lukasz Cwik" <[email protected]> >> a >> >> >>>>>>> >> >> >>>>>> écrit >> >> >>>>> >> >> >>>>>> : >> >> >>>>>> >> >> >>>>>>> >> >> >>>>>>> I wouldn't mind merging this change in so I could setup those >> >> Gradle >> >> >>>>>>>> Jenkins precommits. >> >> >>>>>>>> >> >> >>>>>>>> As per our contribution guidelines, any committer willing to >> sign >> >> >>>>>>>> >> >> >>>>>>> off >> >> >>>>> >> >> >>>>>> on >> >> >>>>>> >> >> >>>>>>> the PR? >> >> >>>>>>>> >> >> >>>>>>>> On Thu, Nov 9, 2017 at 2:12 PM, Romain Manni-Bucau < >> >> >>>>>>>> >> >> >>>>>>> [email protected]> >> >> >>>>>>> >> >> >>>>>>>> wrote: >> >> >>>>>>>> >> >> >>>>>>>> Le 9 nov. 2017 21:31, "Kenneth Knowles" <[email protected] >> > >> >> >>>>>>>>> >> >> >>>>>>>> a >> >> >>>>> >> >> >>>>>> écrit : >> >> >>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> Keep in mind that a clean build is unusual during development >> (it >> >> >>>>>>>>> >> >> >>>>>>>> is >> >> >>>>> >> >> >>>>>> common >> >> >>>>>>>> >> >> >>>>>>>>> for mvn use and that is a bug) and also not necessary for >> >> >>>>>>>>> >> >> >>>>>>>> precommits >> >> >>>>> >> >> >>>>>> if >> >> >>>>>> >> >> >>>>>>> the >> >> >>>>>>>> >> >> >>>>>>>>> build tool is correct enough that caching is safe. So while >> this >> >> >>>>>>>>> >> >> >>>>>>>> number >> >> >>>>>> >> >> >>>>>>> matters, it is not the most important. >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> Not sure, in dev you bypass the build tool most of the time >> >> >>>>>>>>> >> >> >>>>>>>> anyway - >> >> >>>>> >> >> >>>>>> thanks >> >> >>>>>>>> >> >> >>>>>>>>> to IDE or other shortcuts - but not on PR and CI. Keep in mind >> >> >>>>>>>>> >> >> >>>>>>>> that >> >> >>>>> >> >> >>>>>> not >> >> >>>>>> >> >> >>>>>>> doing a clean and killing gradle daemon makes the build not >> >> >>>>>>>>> >> >> >>>>>>>> reproducible >> >> >>>>>>> >> >> >>>>>>>> and therefore useful :(. Starting to build from a subpart of >> the >> >> >>>>>>>>> >> >> >>>>>>>> reactor >> >> >>>>>>> >> >> >>>>>>>> - >> >> >>>>>>>> >> >> >>>>>>>>> with the mentionned mvn plugin for instance - can be nice on >> some >> >> >>>>>>>>> >> >> >>>>>>>> CI >> >> >>>>> >> >> >>>>>> like >> >> >>>>>>> >> >> >>>>>>>> travis if the caching is well configured but still not a >> guarantee >> >> >>>>>>>>> >> >> >>>>>>>> the >> >> >>>>>> >> >> >>>>>>> build is "green". >> >> >>>>>>>>> >> >> >>>>>>>>> My trade off is to ensure an easy build and relevant result >> over >> >> >>>>>>>>> >> >> >>>>>>>> the >> >> >>>>> >> >> >>>>>> time >> >> >>>>>>> >> >> >>>>>>>> criteria. Do you share it as well or prefer time over other >> >> >>>>>>>>> >> >> >>>>>>>> criteria >> >> >>>>> >> >> >>>>>> - >> >> >>>>>> >> >> >>>>>>> which leads to other conclusions and options indeed and can make >> >> >>>>>>>>> >> >> >>>>>>>> us >> >> >>>>> >> >> >>>>>> not >> >> >>>>>> >> >> >>>>>>> understanding each other? >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> On Thu, Nov 9, 2017 at 11:30 AM, Romain Manni-Bucau < >> >> >>>>>>>>> >> >> >>>>>>>> [email protected] >> >> >>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>>> wrote: >> >> >>>>>>>>> >> >> >>>>>>>>> I will try next week yes but the 2 runs i did were 28mn vs >> 32mn >> >> >>>>>>>>>> >> >> >>>>>>>>> from >> >> >>>>>> >> >> >>>>>>> memory >> >> >>>>>>>>> >> >> >>>>>>>>>> - after having downloaded all deps once. >> >> >>>>>>>>>> >> >> >>>>>>>>>> Le 9 nov. 2017 19:45, "Lukasz Cwik" <[email protected] >> > >> >> >>>>>>>>>> >> >> >>>>>>>>> a >> >> >>>>> >> >> >>>>>> écrit : >> >> >>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>>> If Gradle was slow, do you mind running the build with >> >> >>>>>>>>>>> >> >> >>>>>>>>>> --profile >> >> >>>>> >> >> >>>>>> and >> >> >>>>>>> >> >> >>>>>>>> sharing that and also sharing the Maven build log? >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> On Thu, Nov 9, 2017 at 10:43 AM, Lukasz Cwik < >> >> >>>>>>>>>>> >> >> >>>>>>>>>> [email protected]> >> >> >>>>> >> >> >>>>>> wrote: >> >> >>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>>>> Romain, I don't understand your last comment, were you >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>> trying >> >> >>>>> >> >> >>>>>> to >> >> >>>>>> >> >> >>>>>>> say >> >> >>>>>>>> >> >> >>>>>>>>> that >> >> >>>>>>>>>> >> >> >>>>>>>>>>> you had the same Gradle build times like I did and it was an >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>> improvement >> >> >>>>>>>>>> >> >> >>>>>>>>>>> over Maven or that you did not and you experienced build >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>> times >> >> >>>>> >> >> >>>>>> that >> >> >>>>>>> >> >> >>>>>>>> were >> >> >>>>>>>>>> >> >> >>>>>>>>>>> equivalent to Maven? >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> On Thu, Nov 9, 2017 at 9:51 AM, Romain Manni-Bucau < >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>> [email protected]> >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> wrote: >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>> 2017-11-09 18:38 GMT+01:00 Kenneth Knowles >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>> <[email protected] >> >> >>>>>>> >> >> >>>>>>>> : >> >> >>>>>>>>>> >> >> >>>>>>>>>>> On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau < >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>> [email protected]> >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> wrote: >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> (this is another topic so we can maybe open another >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> thread) >> >> >>>>> >> >> >>>>>> issue >> >> >>>>>>>> >> >> >>>>>>>>> is >> >> >>>>>>>>>> >> >> >>>>>>>>>>> not much about python but more about the fact the build >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> is >> >> >>>>> >> >> >>>>>> not >> >> >>>>>>> >> >> >>>>>>>> self >> >> >>>>>>>>> >> >> >>>>>>>>>> contained. it is a maven build and maven should be >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> sufficient >> >> >>>>>> >> >> >>>>>>> without >> >> >>>>>>>>>> >> >> >>>>>>>>>>> having to install python + dependencies. >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> Let's leave out the topic of whether our build should >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>> install >> >> >>>>>> >> >> >>>>>>> things >> >> >>>>>>>>> >> >> >>>>>>>>>> like >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> JDKs, Python, Golang, Docker, protoc, findbugs, RAT, etc. >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>> That >> >> >>>>>> >> >> >>>>>>> issue >> >> >>>>>>>>> >> >> >>>>>>>>>> is >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> somewhat independent of build tool, and the new build >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>> isn't >> >> >>>>> >> >> >>>>>> worse >> >> >>>>>>>> >> >> >>>>>>>>> than >> >> >>>>>>>>>> >> >> >>>>>>>>>>> the >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> old one as far as it goes. >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>> Yep, globally the same time with clean and killing the >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>> daemon. >> >> >>>>> >> >> >>>>>> >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> Kenn >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> I don't see any technical >> >> >>>>>>>>>>>>>>> blockers to do it (except time ;)) but it is always a >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> bit >> >> >>>>> >> >> >>>>>> annoying >> >> >>>>>>>>> >> >> >>>>>>>>>> to >> >> >>>>>>>>>> >> >> >>>>>>>>>>> git clone then not be able to build. >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> Romain Manni-Bucau >> >> >>>>>>>>>>>>>>> @rmannibucau | Blog | Old Blog | Github | LinkedIn >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> 2017-11-09 18:07 GMT+01:00 Lukasz Cwik >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> <[email protected] >> >> >>>>>>>> >> >> >>>>>>>>> : >> >> >>>>>>>>>> >> >> >>>>>>>>>>> Hmm, I have had good luck when following the Python >> >> >>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> quick >> >> >>>>> >> >> >>>>>> start >> >> >>>>>>>> >> >> >>>>>>>>> setup >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> <https://beam.apache.org/get-started/quickstart-py/> >> >> >>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> on >> >> >>>>> >> >> >>>>>> multiple >> >> >>>>>>>>> >> >> >>>>>>>>>> machines >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> by ensuring the installed version of setuptools, >> >> >>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> virtualenv >> >> >>>>>> >> >> >>>>>>> and >> >> >>>>>>>> >> >> >>>>>>>>> pip >> >> >>>>>>>>>> >> >> >>>>>>>>>>> are >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> new >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> enough versions. >> >> >>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> You can always skip the Python portion of the build by >> >> >>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> excluding >> >> >>>>>>>>> >> >> >>>>>>>>>> the >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> build >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> task as so: >> >> >>>>>>>>>>>>>>>> ./gradlew build -x ":beam-sdks-parent:beam-sdks- >> >> >>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> python:build" >> >> >>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>>>>>>>>> On Thu, Nov 9, 2017 at 8:58 AM, Romain Manni-Bucau < >> >> >>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> [email protected]> >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> wrote: >> >> >>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> The 1.3.5 file is when i installed the python >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> dependencies >> >> >>>>>> >> >> >>>>>>> manually >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> to make the build passing (the pip command never >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> passed >> >> >>>>> >> >> >>>>>> on >> >> >>>>>> >> >> >>>>>>> my >> >> >>>>>>>> >> >> >>>>>>>>> computer >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> and therefore the build always has been broken until >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> i >> >> >>>>> >> >> >>>>>> installed >> >> >>>>>>>>> >> >> >>>>>>>>>> it >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> manually - independently from the build tool). >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>> Romain Manni-Bucau >> >> >>>>>>>>>>>>>>>>> @rmannibucau | Blog | Old Blog | Github | LinkedIn >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>> 2017-11-09 17:51 GMT+01:00 Lukasz Cwik >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> <[email protected] >> >> >>>>>>>>> >> >> >>>>>>>>>> : >> >> >>>>>>>>>>>> >> >> >>>>>>>>>>>>> It turns out that the Apache Rat Ant task and the >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>> Apache >> >> >>>>>> >> >> >>>>>>> Rat >> >> >>>>>>>> >> >> >>>>>>>>> Maven >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> plugin >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> differ in that the plugin automatically excludes >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>> certain >> >> >>>>>> >> >> >>>>>>> files >> >> >>>>>>>>> >> >> >>>>>>>>>> by >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> default >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> while the Ant task does not. >> >> >>>>>>>>>>>>>>>>>> See: >> >> >>>>>>>>>>>>>>>>>> http://creadur.apache.org/rat/ >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>> apache-rat-plugin/check-mojo. >> >> >>>>>>>> >> >> >>>>>>>>> html#useDefaultExcludes >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> I fixed the list to exclude ".idea/" instead of >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>> "idea/" >> >> >>>>>> >> >> >>>>>>> since >> >> >>>>>>>>> >> >> >>>>>>>>>> there >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> was a >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> typo. >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> I have no idea what the file "=1.3.5" is. Can you >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>> take a >> >> >>>>>> >> >> >>>>>>> look >> >> >>>>>>>>> >> >> >>>>>>>>>> at >> >> >>>>>>>>>> >> >> >>>>>>>>>>> the >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> contents? >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> On Thu, Nov 9, 2017 at 12:03 AM, Romain >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>> Manni-Bucau < >> >> >>>>> >> >> >>>>>> [email protected]> >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> wrote: >> >> >>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> Ok, the rat issues I got were: >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>> == File: /home/rmannibucau/1_dev/beam/.idea/* >> >> >>>>>>>>>>>>>>>>>>> == File: /home/rmannibucau/1_dev/beam/ >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> sdks/python/=1.3.5 >> >> >>>>>>> >> >> >>>>>>>> >> >> >>>>>>>>>>>>>>>>>>> The first one could be in my default exclude - >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> even >> >> >>>>> >> >> >>>>>> if >> >> >>>>>> >> >> >>>>>>> eclipse/idea >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> files should be in the default exclude set of beam >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> rat >> >> >>>>>> >> >> >>>>>>> config >> >> >>>>>>>>> >> >> >>>>>>>>>> IMHO, >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> the last one is more a "?" can probably be >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> exclude as >> >> >>>>> >> >> >>>>>> well >> >> >>>>>>>> >> >> >>>>>>>>> if >> >> >>>>>>>>> >> >> >>>>>>>>>> created >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> by the build at some point. >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>> Romain Manni-Bucau >> >> >>>>>>>>>>>>>>>>>>> @rmannibucau | Blog | Old Blog | Github | >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> LinkedIn >> >> >>>>> >> >> >>>>>> >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>> 2017-11-08 19:17 GMT+01:00 Jean-Baptiste Onofré < >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> [email protected] >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> : >> >> >>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>> Thanks for the update. I was swamped on some >> >> >>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>> meetings. >> >> >>>>>>> >> >> >>>>>>>> I'm >> >> >>>>>>>>> >> >> >>>>>>>>>> back to >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> test >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> the latest changes. >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> Regards >> >> >>>>>>>>>>>>>>>>>>>> JB >> >> >>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> On Nov 8, 2017, 18:56, at 18:56, Lukasz Cwik >> >> >>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>> <[email protected] >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> wrote: >> >> >>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> Thanks everyone for trying this build out in >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> different >> >> >>>>>>> >> >> >>>>>>>> workspaces / >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> configurations. This will help make sure the >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> build >> >> >>>>> >> >> >>>>>> works >> >> >>>>>>>> >> >> >>>>>>>>> for >> >> >>>>>>>>>> >> >> >>>>>>>>>>> more >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> people >> >> >>>>>>>>>>>>>>>>>>>>> and will get rid of any rough edges. >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>> Performance (All): >> >> >>>>>>>>>>>>>>>>>>>>> Maven performs parallelization at the module >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> level, >> >> >>>>> >> >> >>>>>> an >> >> >>>>>>> >> >> >>>>>>>> entire >> >> >>>>>>>>>> >> >> >>>>>>>>>>> module >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> needs >> >> >>>>>>>>>>>>>>>>>>>>> to complete before any dependent modules can >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> start, >> >> >>>>> >> >> >>>>>> this >> >> >>>>>>>> >> >> >>>>>>>>> means >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> running >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> all >> >> >>>>>>>>>>>>>>>>>>>>> the checks like findbugs, checkstyle, tests >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> need to >> >> >>>>> >> >> >>>>>> finish. >> >> >>>>>>>>> >> >> >>>>>>>>>> Gradle >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> has >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> task >> >> >>>>>>>>>>>>>>>>>>>>> level parallelism between subprojects which >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> means >> >> >>>>> >> >> >>>>>> that >> >> >>>>>>> >> >> >>>>>>>> as >> >> >>>>>>>> >> >> >>>>>>>>> soon >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> as >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> the >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> compile and shade steps are done for a project, >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> and >> >> >>>>> >> >> >>>>>> dependent >> >> >>>>>>>>>> >> >> >>>>>>>>>>> subprojects >> >> >>>>>>>>>>>>>>>>>>>>> can typically start. This means that we get >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> increased >> >> >>>>>> >> >> >>>>>>> parallelism >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> due >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> to >> >> >>>>>>>>>>>>>>>>>>>>> not needing to wait for findbugs, checkstyle, >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> tests >> >> >>>>> >> >> >>>>>> to >> >> >>>>>>> >> >> >>>>>>>> run. I >> >> >>>>>>>>>> >> >> >>>>>>>>>>> typically >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> see >> >> >>>>>>>>>>>>>>>>>>>>> ~20 tasks (at peak) running on my desktop in >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> parallel. >> >> >>>>>>> >> >> >>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>> Apache Rat (JB / Romain): >> >> >>>>>>>>>>>>>>>>>>>>> What files are in the rat report that fail (its >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> likely >> >> >>>>>>> >> >> >>>>>>>> that >> >> >>>>>>>>> >> >> >>>>>>>>>> I'm >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> missing >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> some exclusion for a build time artifact)? Also, >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> please >> >> >>>>>>> >> >> >>>>>>>> try >> >> >>>>>>>>> >> >> >>>>>>>>>> the >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> build >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> again >> >> >>>>>>>>>>>>>>>>>>>>> after running `git clean -fdx` in your >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> workspace. >> >> >>>>> >> >> >>>>>> >> >> >>>>>>>>>>>>>>>>>>>>> Python (JB): >> >> >>>>>>>>>>>>>>>>>>>>> As for the Python SDK, you'll need to share more >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> details >> >> >>>>>>>> >> >> >>>>>>>>> about >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> the >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> failure. >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>> Gradle 4.3: >> >> >>>>>>>>>>>>>>>>>>>>> I would like to defer the swap to Gradle 4.3 >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> until >> >> >>>>> >> >> >>>>>> after >> >> >>>>>>>> >> >> >>>>>>>>> this >> >> >>>>>>>>>> >> >> >>>>>>>>>>> PR >> >> >>>>>>>>>>>>> >> >> >>>>>>>>>>>>>> since >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> it >> >> >>>>>>>>>>>>>>>>>>>>> will be a much smaller set of changes. >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>> On Wed, Nov 8, 2017 at 12:54 AM, Jean-Baptiste >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>> Onofré < >> >> >>>>>>> >> >> >>>>>>>> [email protected]> >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> wrote: >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>> Same for me for rat and python build too: >> >> >>>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>>> FAILURE: Build completed with 2 failures. >> >> >>>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>>> 1: Task failed with an exception. >> >> >>>>>>>>>>>>>>>>>>>>>> ----------- >> >> >>>>>>>>>>>>>>>>>>>>>> * What went wrong: >> >> >>>>>>>>>>>>>>>>>>>>>> Execution failed for task ':rat'. >> >> >>>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>>>> Found 905 files with unapproved/unknown >> >> >>>>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>>> licenses. >> >> >>>>>> >> >> >>>>>>> See >> >> >>>>>>>> >> >> >>>>>>>>> file:/home/jbonofre/Workspace/ >> >> >>>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>> beam/build/reports/rat/rat- >> >> >>>>>>>>>> >> >> >>>>>>>>>>> report.txt >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>>> * Try: >> >> >>>>>>>>>>>>>>>>>>>>>> Run with --stacktrace option to get the stack >> >> >>>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>> trace. >> >> >>>>>>> >> >> >>>>>>>> Run >> >> >>>>>>>>> >> >> >>>>>>>>>> with >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> --info >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> or >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>>> --debug option to get more log output. >> >> >>>>>>>>>>>>>>>>>>>>>> ============================== >> >> >>>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>> ============================== >> >> >>>>>>>>>>> >> >> >>>>>>>>>>>> ================== >> >> >>>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>>> 2: Task failed with an exception. >> >> >>>>>>>>>>>>>>>>>>>>>> ----------- >> >> >>>>>>>>>>>>>>>>>>>>>> * Where: >> >> >>>>>>>>>>>>>>>>>>>>>> Build file '/home/jbonofre/Workspace/ >> >> >>>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>> beam/sdks/python/build.gradle' >> >> >>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>> line: >> >> >>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>>> 64 >> >> >>>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>>> * What went wrong: >> >> >>>>>>>>>>>>>>>>>>>>>> Execution failed for task >> >> >>>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>> ':beam-sdks-parent:beam-sdks- >> >> >>>>>>>> >> >> >>>>>>>>> python:lint'. >> >> >>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>> Process 'command 'tox'' finished with >> >> >>>>>>>>>>>>>>>>>>>>>> >> >> >>>>>>>>>>>>>>>>>>>>>> >> >> >>
