Le 27 nov. 2017 23:07, "Eugene Kirpichov" <kirpic...@google.com.invalid> a écrit :
On Mon, Nov 27, 2017 at 11:52 AM Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > 2017-11-27 20:26 GMT+01:00 Lukasz Cwik <lc...@google.com.invalid>: > > Romain, as mentioned earlier, I identified that Maven was slower because > it > > needed to finish building the entire module before dependent modules > could > > start which included running tests, performing checkstyle, etc... > > Gradle is able to increase the parallelism of the build process since it > > has task driven parallelism so as long as the files are compiled, the > > dependent projects can start. > > This means we can implement a maven graph builder which is better than > the default one - surely with a thread safe local repo - and > contribute it back to solve it durably. > That would be indeed a wonderful contribution to the Java ecosystem as a whole, but I'd like to reiterate that Maven has existed for very long, I'm sure a lot of people have noticed its poor parallelization in the meanwhile, and still nobody has even tried doing this AFAIK. So it seems unlikely that this is doable in a reasonable timeframe, and I don't think that the possibility of this is a strong argument against switching to Gradle. Agree. Just to clarify: my strong argument against gradle is that it is not enterprise mainstream IMHO - dont forget github projects are a lot about personal tests, small projects, so hard to use all the repos as a valid stat - which leads to enhancing mvn rather than switching. Again, as an ASF project, not even contacting our colleagues is not very sane so I would go and check with them rather than just doing like a github project maintained by a single company. Side note on previous mail: mvn extensibility is "mature". Mojo let you do what you want and you can customize the lifecycle as well. The main difference is a standardized build (descriptive) vs a fully custom and "by developer" styled script. > > > > > Maven and Gradle are both heavily used since there are ~146k Maven > projects > > on Github while there are ~122k Gradle project on Github. Do you have > data > > which shows that Maven is significantly more "mainstream"? > > Yep, project i worked on in companies using gradle: 0, all were based > on maven and maven was "tool-ed" versus gradle was "best effort" in > term of plugins. > Now - with my EE background - I can guarantee you gradle is not able > to handle properly its build since it flatten the classpath and > plugins conflicts very quickly (their plugin dependency feature never > worked and almost no plugin impl it correctly). > > Wonder if it is easy to have the ASF stats, anyone knows? > > > > > I believe we want a rich multi-language SDK and community and feel as > > though it would be unwise to treat non JDK based languages as second > class. > > Hmm, not sure how it is related to the build tool since Maven and > Gradle have the same level of support - actually surprsingly maven is > better for js and surely as bad as gradle for others - or here again > we can create plugin like the frontend-maven-plugin if needed for > other languages. > > That said it can be an interesting other thread since people consuming > these languages will probably want their mainstream build tool and a > "standard" repository layout rather than a java one. But this is > harder to measure. > > > > > On Mon, Nov 27, 2017 at 11:00 AM, Romain Manni-Bucau < > rmannibu...@gmail.com> > > wrote: > > > >> Hi Lukasz, > >> > >> Did you manage to identify how maven was slower and test tesla stuff > >> and potentially a few other fixes? > >> > >> Side note: figures without python can be interesting cause locally - = > >> for me - python tends to flatten the figures whereas I get something > >> close to your conclusions without python part. > >> > >> My point is mainly that switching now on gradle and being back on > >> maven in a few months cause gradle ecosystem is far to support java 9 > >> - or any other volatile reason like this one - is probably not a good > >> choice for a community. Maven is way more mainstream than gradle so > >> helps to encourage people to contribute - vs gradle will increase the > >> step to do it. > >> > >> I'd like to be sure before a switch that it is a one way decision and > >> that the build tool was not just challenged by itself and its current > >> state but also in the way it could be improved (= its community and > >> potentially some local hacks). > >> > >> Romain Manni-Bucau > >> @rmannibucau | Blog | Old Blog | Github | LinkedIn > >> > >> > >> 2017-11-27 19:46 GMT+01:00 Lukasz Cwik <lc...@google.com.invalid>: > >> > I have collected data by running several builds against master using > >> Gradle > >> > and Maven without using Gradle's support for incremental builds. > >> > > >> > Gradle (mins) > >> > min: 25.04 > >> > max: 160.14 > >> > median: 45.78 > >> > average: 52.19 > >> > stdev: 30.80 > >> > > >> > Maven (mins) > >> > min: 56.86 > >> > max: 216.55 (actually > 240 mins because this data does not include > >> > timeouts) > >> > median: 87.93 > >> > average: 109.10 > >> > stdev: 48.01 > >> > > >> > I excluded a few timeouts (240 mins) that happened during the Maven > build > >> > from its numbers but we can see conclusively that Gradle is twice as > fast > >> > for the build when compared to Maven when run using Jenkins. > >> > On my desktop, I have enabled incremental builds and have seen a major > >> > improvement on the above numbers but it doesn't yet work correctly > >> because > >> > of incorrectly specified inputs/outputs for certain tasks. > >> > > >> > The data is available here > >> > https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_ > >> NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing > >> > > >> > With this data, I feel confident that we should swap and have opened > the > >> > following issue https://issues.apache.org/jira/browse/BEAM-3249 and > >> related > >> > sub-tasks. > >> > > >> > On Sun, Nov 19, 2017 at 11:23 AM, Jean-Baptiste Onofré < > j...@nanthrax.net> > >> > wrote: > >> > > >> >> Thanks for the update Luke. > >> >> > >> >> I'm updating my local working copy to do new tests. > >> >> > >> >> Regards > >> >> JB > >> >> > >> >> On 11/19/2017 08:21 PM, Lukasz Cwik wrote: > >> >> > >> >>> The gradle build rules have been merged, I'm adding a precommit[1] > to > >> >>> start > >> >>> collecting data about the build times. It currently only mirrors the > >> Java > >> >>> mvn install precommit. I'll gather data over the next two weeks and > >> >>> provide > >> >>> a summary here. > >> >>> > >> >>> You can rerun the precommit by issuing "Run Java Gradle PreCommit" > >> >>> > >> >>> 1: https://github.com/apache/beam/pull/4146 > >> >>> > >> >>> > >> >>> On Mon, Nov 13, 2017 at 9:08 AM, Lukasz Cwik <lc...@google.com> > wrote: > >> >>> > >> >>> There has been plenty of time for comments on the PR and the > approach. > >> >>>> > >> >>>> So far Ken Knowles has provided the most feedback on the PR, Ken > would > >> >>>> you > >> >>>> like to finish the review? > >> >>>> > >> >>>> > >> >>>> > >> >>>> On Fri, Nov 10, 2017 at 1:22 PM, Romain Manni-Bucau < > >> >>>> rmannibu...@gmail.com > >> >>>> > >> >>>>> wrote: > >> >>>>> > >> >>>> > >> >>>> This is only a setup thing and better to not break the master > history > >> for > >> >>>>> poc/tests, in particular when no very localized. Alternative can > be > >> to > >> >>>>> ask > >> >>>>> another temp repo to infra and have a synchro between both but > dont > >> >>>>> think > >> >>>>> it does worth it personally. > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> Le 10 nov. 2017 18:57, "Lukasz Cwik" <lc...@google.com.invalid> a > >> >>>>> écrit : > >> >>>>> > >> >>>>> The reason to get it on master is because that is where all the > PRs > >> >>>>>> > >> >>>>> are. An > >> >>>>> > >> >>>>>> upstream branch without any development means no data. > >> >>>>>> Also, our Jenkins setup via job-dsl doesn't honor using the > Jenkins > >> >>>>>> configuration on the branch because the seed job always runs > against > >> >>>>>> master. > >> >>>>>> > >> >>>>>> On Thu, Nov 9, 2017 at 9:59 PM, Romain Manni-Bucau < > >> >>>>>> > >> >>>>> rmannibu...@gmail.com> > >> >>>>> > >> >>>>>> wrote: > >> >>>>>> > >> >>>>>> What about pushing it on a "upstream" branch and testing it for 1 > >> >>>>>>> > >> >>>>>> week in > >> >>>>> > >> >>>>>> parallel of the maven reference build? If gradle is always 50% > >> faster > >> >>>>>>> > >> >>>>>> on > >> >>>>> > >> >>>>>> jenkins then it could become master setup without much > discussion I > >> >>>>>>> > >> >>>>>> guess. > >> >>>>>> > >> >>>>>>> We can even have 2 jenkins jobs: one with the daemon etc and one > >> >>>>>>> > >> >>>>>> without. > >> >>>>> > >> >>>>>> > >> >>>>>>> Also noticed yesterday that gradle build is killing my machine > >> (all 8 > >> >>>>>>> > >> >>>>>> cores > >> >>>>>> > >> >>>>>>> are 100%) during the first minutes vs maven build which let me > do > >> >>>>>>> > >> >>>>>> something > >> >>>>>> > >> >>>>>>> else. Then all the consumed time which makes gradle not that > fast > >> is > >> >>>>>>> > >> >>>>>> about > >> >>>>>> > >> >>>>>>> python. Will try to send figures later today. > >> >>>>>>> > >> >>>>>>> Le 10 nov. 2017 00:10, "Lukasz Cwik" <lc...@google.com.invalid> > a > >> >>>>>>> > >> >>>>>> écrit > >> >>>>> > >> >>>>>> : > >> >>>>>> > >> >>>>>>> > >> >>>>>>> I wouldn't mind merging this change in so I could setup those > >> Gradle > >> >>>>>>>> Jenkins precommits. > >> >>>>>>>> > >> >>>>>>>> As per our contribution guidelines, any committer willing to > sign > >> >>>>>>>> > >> >>>>>>> off > >> >>>>> > >> >>>>>> on > >> >>>>>> > >> >>>>>>> the PR? > >> >>>>>>>> > >> >>>>>>>> On Thu, Nov 9, 2017 at 2:12 PM, Romain Manni-Bucau < > >> >>>>>>>> > >> >>>>>>> rmannibu...@gmail.com> > >> >>>>>>> > >> >>>>>>>> wrote: > >> >>>>>>>> > >> >>>>>>>> Le 9 nov. 2017 21:31, "Kenneth Knowles" <k...@google.com.invalid > > > >> >>>>>>>>> > >> >>>>>>>> a > >> >>>>> > >> >>>>>> écrit : > >> >>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> Keep in mind that a clean build is unusual during development > (it > >> >>>>>>>>> > >> >>>>>>>> is > >> >>>>> > >> >>>>>> common > >> >>>>>>>> > >> >>>>>>>>> for mvn use and that is a bug) and also not necessary for > >> >>>>>>>>> > >> >>>>>>>> precommits > >> >>>>> > >> >>>>>> if > >> >>>>>> > >> >>>>>>> the > >> >>>>>>>> > >> >>>>>>>>> build tool is correct enough that caching is safe. So while > this > >> >>>>>>>>> > >> >>>>>>>> number > >> >>>>>> > >> >>>>>>> matters, it is not the most important. > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> Not sure, in dev you bypass the build tool most of the time > >> >>>>>>>>> > >> >>>>>>>> anyway - > >> >>>>> > >> >>>>>> thanks > >> >>>>>>>> > >> >>>>>>>>> to IDE or other shortcuts - but not on PR and CI. Keep in mind > >> >>>>>>>>> > >> >>>>>>>> that > >> >>>>> > >> >>>>>> not > >> >>>>>> > >> >>>>>>> doing a clean and killing gradle daemon makes the build not > >> >>>>>>>>> > >> >>>>>>>> reproducible > >> >>>>>>> > >> >>>>>>>> and therefore useful :(. Starting to build from a subpart of > the > >> >>>>>>>>> > >> >>>>>>>> reactor > >> >>>>>>> > >> >>>>>>>> - > >> >>>>>>>> > >> >>>>>>>>> with the mentionned mvn plugin for instance - can be nice on > some > >> >>>>>>>>> > >> >>>>>>>> CI > >> >>>>> > >> >>>>>> like > >> >>>>>>> > >> >>>>>>>> travis if the caching is well configured but still not a > guarantee > >> >>>>>>>>> > >> >>>>>>>> the > >> >>>>>> > >> >>>>>>> build is "green". > >> >>>>>>>>> > >> >>>>>>>>> My trade off is to ensure an easy build and relevant result > over > >> >>>>>>>>> > >> >>>>>>>> the > >> >>>>> > >> >>>>>> time > >> >>>>>>> > >> >>>>>>>> criteria. Do you share it as well or prefer time over other > >> >>>>>>>>> > >> >>>>>>>> criteria > >> >>>>> > >> >>>>>> - > >> >>>>>> > >> >>>>>>> which leads to other conclusions and options indeed and can make > >> >>>>>>>>> > >> >>>>>>>> us > >> >>>>> > >> >>>>>> not > >> >>>>>> > >> >>>>>>> understanding each other? > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> On Thu, Nov 9, 2017 at 11:30 AM, Romain Manni-Bucau < > >> >>>>>>>>> > >> >>>>>>>> rmannibu...@gmail.com > >> >>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>>> wrote: > >> >>>>>>>>> > >> >>>>>>>>> I will try next week yes but the 2 runs i did were 28mn vs > 32mn > >> >>>>>>>>>> > >> >>>>>>>>> from > >> >>>>>> > >> >>>>>>> memory > >> >>>>>>>>> > >> >>>>>>>>>> - after having downloaded all deps once. > >> >>>>>>>>>> > >> >>>>>>>>>> Le 9 nov. 2017 19:45, "Lukasz Cwik" <lc...@google.com.invalid > > > >> >>>>>>>>>> > >> >>>>>>>>> a > >> >>>>> > >> >>>>>> écrit : > >> >>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>>> If Gradle was slow, do you mind running the build with > >> >>>>>>>>>>> > >> >>>>>>>>>> --profile > >> >>>>> > >> >>>>>> and > >> >>>>>>> > >> >>>>>>>> sharing that and also sharing the Maven build log? > >> >>>>>>>>>>> > >> >>>>>>>>>>> On Thu, Nov 9, 2017 at 10:43 AM, Lukasz Cwik < > >> >>>>>>>>>>> > >> >>>>>>>>>> lc...@google.com> > >> >>>>> > >> >>>>>> wrote: > >> >>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>>>> Romain, I don't understand your last comment, were you > >> >>>>>>>>>>>> > >> >>>>>>>>>>> trying > >> >>>>> > >> >>>>>> to > >> >>>>>> > >> >>>>>>> say > >> >>>>>>>> > >> >>>>>>>>> that > >> >>>>>>>>>> > >> >>>>>>>>>>> you had the same Gradle build times like I did and it was an > >> >>>>>>>>>>>> > >> >>>>>>>>>>> improvement > >> >>>>>>>>>> > >> >>>>>>>>>>> over Maven or that you did not and you experienced build > >> >>>>>>>>>>>> > >> >>>>>>>>>>> times > >> >>>>> > >> >>>>>> that > >> >>>>>>> > >> >>>>>>>> were > >> >>>>>>>>>> > >> >>>>>>>>>>> equivalent to Maven? > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> On Thu, Nov 9, 2017 at 9:51 AM, Romain Manni-Bucau < > >> >>>>>>>>>>>> > >> >>>>>>>>>>> rmannibu...@gmail.com> > >> >>>>>>>>>>> > >> >>>>>>>>>>>> wrote: > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> 2017-11-09 18:38 GMT+01:00 Kenneth Knowles > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>> <k...@google.com.invalid > >> >>>>>>> > >> >>>>>>>> : > >> >>>>>>>>>> > >> >>>>>>>>>>> On Thu, Nov 9, 2017 at 9:11 AM, Romain Manni-Bucau < > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>> rmannibu...@gmail.com> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> (this is another topic so we can maybe open another > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> thread) > >> >>>>> > >> >>>>>> issue > >> >>>>>>>> > >> >>>>>>>>> is > >> >>>>>>>>>> > >> >>>>>>>>>>> not much about python but more about the fact the build > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> is > >> >>>>> > >> >>>>>> not > >> >>>>>>> > >> >>>>>>>> self > >> >>>>>>>>> > >> >>>>>>>>>> contained. it is a maven build and maven should be > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> sufficient > >> >>>>>> > >> >>>>>>> without > >> >>>>>>>>>> > >> >>>>>>>>>>> having to install python + dependencies. > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Let's leave out the topic of whether our build should > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>> install > >> >>>>>> > >> >>>>>>> things > >> >>>>>>>>> > >> >>>>>>>>>> like > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> JDKs, Python, Golang, Docker, protoc, findbugs, RAT, etc. > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>> That > >> >>>>>> > >> >>>>>>> issue > >> >>>>>>>>> > >> >>>>>>>>>> is > >> >>>>>>>>>>> > >> >>>>>>>>>>>> somewhat independent of build tool, and the new build > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>> isn't > >> >>>>> > >> >>>>>> worse > >> >>>>>>>> > >> >>>>>>>>> than > >> >>>>>>>>>> > >> >>>>>>>>>>> the > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> old one as far as it goes. > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> Yep, globally the same time with clean and killing the > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>> daemon. > >> >>>>> > >> >>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Kenn > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> I don't see any technical > >> >>>>>>>>>>>>>>> blockers to do it (except time ;)) but it is always a > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> bit > >> >>>>> > >> >>>>>> annoying > >> >>>>>>>>> > >> >>>>>>>>>> to > >> >>>>>>>>>> > >> >>>>>>>>>>> git clone then not be able to build. > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> Romain Manni-Bucau > >> >>>>>>>>>>>>>>> @rmannibucau | Blog | Old Blog | Github | LinkedIn > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> 2017-11-09 18:07 GMT+01:00 Lukasz Cwik > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> <lc...@google.com.invalid > >> >>>>>>>> > >> >>>>>>>>> : > >> >>>>>>>>>> > >> >>>>>>>>>>> Hmm, I have had good luck when following the Python > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> quick > >> >>>>> > >> >>>>>> start > >> >>>>>>>> > >> >>>>>>>>> setup > >> >>>>>>>>>>> > >> >>>>>>>>>>>> <https://beam.apache.org/get-started/quickstart-py/> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> on > >> >>>>> > >> >>>>>> multiple > >> >>>>>>>>> > >> >>>>>>>>>> machines > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> by ensuring the installed version of setuptools, > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> virtualenv > >> >>>>>> > >> >>>>>>> and > >> >>>>>>>> > >> >>>>>>>>> pip > >> >>>>>>>>>> > >> >>>>>>>>>>> are > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> new > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> enough versions. > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> You can always skip the Python portion of the build by > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> excluding > >> >>>>>>>>> > >> >>>>>>>>>> the > >> >>>>>>>>>>> > >> >>>>>>>>>>>> build > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> task as so: > >> >>>>>>>>>>>>>>>> ./gradlew build -x ":beam-sdks-parent:beam-sdks- > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> python:build" > >> >>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>>>>>>>>> On Thu, Nov 9, 2017 at 8:58 AM, Romain Manni-Bucau < > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> rmannibu...@gmail.com> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> The 1.3.5 file is when i installed the python > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> dependencies > >> >>>>>> > >> >>>>>>> manually > >> >>>>>>>>>>> > >> >>>>>>>>>>>> to make the build passing (the pip command never > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> passed > >> >>>>> > >> >>>>>> on > >> >>>>>> > >> >>>>>>> my > >> >>>>>>>> > >> >>>>>>>>> computer > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> and therefore the build always has been broken until > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> i > >> >>>>> > >> >>>>>> installed > >> >>>>>>>>> > >> >>>>>>>>>> it > >> >>>>>>>>>>> > >> >>>>>>>>>>>> manually - independently from the build tool). > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> Romain Manni-Bucau > >> >>>>>>>>>>>>>>>>> @rmannibucau | Blog | Old Blog | Github | LinkedIn > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> 2017-11-09 17:51 GMT+01:00 Lukasz Cwik > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> <lc...@google.com.invalid > >> >>>>>>>>> > >> >>>>>>>>>> : > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> It turns out that the Apache Rat Ant task and the > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> Apache > >> >>>>>> > >> >>>>>>> Rat > >> >>>>>>>> > >> >>>>>>>>> Maven > >> >>>>>>>>>>> > >> >>>>>>>>>>>> plugin > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> differ in that the plugin automatically excludes > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> certain > >> >>>>>> > >> >>>>>>> files > >> >>>>>>>>> > >> >>>>>>>>>> by > >> >>>>>>>>>>> > >> >>>>>>>>>>>> default > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> while the Ant task does not. > >> >>>>>>>>>>>>>>>>>> See: > >> >>>>>>>>>>>>>>>>>> http://creadur.apache.org/rat/ > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> apache-rat-plugin/check-mojo. > >> >>>>>>>> > >> >>>>>>>>> html#useDefaultExcludes > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> I fixed the list to exclude ".idea/" instead of > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> "idea/" > >> >>>>>> > >> >>>>>>> since > >> >>>>>>>>> > >> >>>>>>>>>> there > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> was a > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> typo. > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> I have no idea what the file "=1.3.5" is. Can you > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> take a > >> >>>>>> > >> >>>>>>> look > >> >>>>>>>>> > >> >>>>>>>>>> at > >> >>>>>>>>>> > >> >>>>>>>>>>> the > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> contents? > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> On Thu, Nov 9, 2017 at 12:03 AM, Romain > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> Manni-Bucau < > >> >>>>> > >> >>>>>> rmannibu...@gmail.com> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> Ok, the rat issues I got were: > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> == File: /home/rmannibucau/1_dev/beam/.idea/* > >> >>>>>>>>>>>>>>>>>>> == File: /home/rmannibucau/1_dev/beam/ > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> sdks/python/=1.3.5 > >> >>>>>>> > >> >>>>>>>> > >> >>>>>>>>>>>>>>>>>>> The first one could be in my default exclude - > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> even > >> >>>>> > >> >>>>>> if > >> >>>>>> > >> >>>>>>> eclipse/idea > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> files should be in the default exclude set of beam > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> rat > >> >>>>>> > >> >>>>>>> config > >> >>>>>>>>> > >> >>>>>>>>>> IMHO, > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> the last one is more a "?" can probably be > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> exclude as > >> >>>>> > >> >>>>>> well > >> >>>>>>>> > >> >>>>>>>>> if > >> >>>>>>>>> > >> >>>>>>>>>> created > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> by the build at some point. > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> Romain Manni-Bucau > >> >>>>>>>>>>>>>>>>>>> @rmannibucau | Blog | Old Blog | Github | > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> LinkedIn > >> >>>>> > >> >>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> 2017-11-08 19:17 GMT+01:00 Jean-Baptiste Onofré < > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> j...@nanthrax.net > >> >>>>>>>>>>> > >> >>>>>>>>>>>> : > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> Thanks for the update. I was swamped on some > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> meetings. > >> >>>>>>> > >> >>>>>>>> I'm > >> >>>>>>>>> > >> >>>>>>>>>> back to > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> test > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> the latest changes. > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> Regards > >> >>>>>>>>>>>>>>>>>>>> JB > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> On Nov 8, 2017, 18:56, at 18:56, Lukasz Cwik > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> <lc...@google.com.INVALID > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> Thanks everyone for trying this build out in > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> different > >> >>>>>>> > >> >>>>>>>> workspaces / > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> configurations. This will help make sure the > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> build > >> >>>>> > >> >>>>>> works > >> >>>>>>>> > >> >>>>>>>>> for > >> >>>>>>>>>> > >> >>>>>>>>>>> more > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> people > >> >>>>>>>>>>>>>>>>>>>>> and will get rid of any rough edges. > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> Performance (All): > >> >>>>>>>>>>>>>>>>>>>>> Maven performs parallelization at the module > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> level, > >> >>>>> > >> >>>>>> an > >> >>>>>>> > >> >>>>>>>> entire > >> >>>>>>>>>> > >> >>>>>>>>>>> module > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> needs > >> >>>>>>>>>>>>>>>>>>>>> to complete before any dependent modules can > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> start, > >> >>>>> > >> >>>>>> this > >> >>>>>>>> > >> >>>>>>>>> means > >> >>>>>>>>>>> > >> >>>>>>>>>>>> running > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> all > >> >>>>>>>>>>>>>>>>>>>>> the checks like findbugs, checkstyle, tests > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> need to > >> >>>>> > >> >>>>>> finish. > >> >>>>>>>>> > >> >>>>>>>>>> Gradle > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> has > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> task > >> >>>>>>>>>>>>>>>>>>>>> level parallelism between subprojects which > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> means > >> >>>>> > >> >>>>>> that > >> >>>>>>> > >> >>>>>>>> as > >> >>>>>>>> > >> >>>>>>>>> soon > >> >>>>>>>>>>> > >> >>>>>>>>>>>> as > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> the > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> compile and shade steps are done for a project, > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> and > >> >>>>> > >> >>>>>> dependent > >> >>>>>>>>>> > >> >>>>>>>>>>> subprojects > >> >>>>>>>>>>>>>>>>>>>>> can typically start. This means that we get > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> increased > >> >>>>>> > >> >>>>>>> parallelism > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> due > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> to > >> >>>>>>>>>>>>>>>>>>>>> not needing to wait for findbugs, checkstyle, > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> tests > >> >>>>> > >> >>>>>> to > >> >>>>>>> > >> >>>>>>>> run. I > >> >>>>>>>>>> > >> >>>>>>>>>>> typically > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> see > >> >>>>>>>>>>>>>>>>>>>>> ~20 tasks (at peak) running on my desktop in > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> parallel. > >> >>>>>>> > >> >>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> Apache Rat (JB / Romain): > >> >>>>>>>>>>>>>>>>>>>>> What files are in the rat report that fail (its > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> likely > >> >>>>>>> > >> >>>>>>>> that > >> >>>>>>>>> > >> >>>>>>>>>> I'm > >> >>>>>>>>>>> > >> >>>>>>>>>>>> missing > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> some exclusion for a build time artifact)? Also, > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> please > >> >>>>>>> > >> >>>>>>>> try > >> >>>>>>>>> > >> >>>>>>>>>> the > >> >>>>>>>>>>> > >> >>>>>>>>>>>> build > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> again > >> >>>>>>>>>>>>>>>>>>>>> after running `git clean -fdx` in your > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> workspace. > >> >>>>> > >> >>>>>> > >> >>>>>>>>>>>>>>>>>>>>> Python (JB): > >> >>>>>>>>>>>>>>>>>>>>> As for the Python SDK, you'll need to share more > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> details > >> >>>>>>>> > >> >>>>>>>>> about > >> >>>>>>>>>>> > >> >>>>>>>>>>>> the > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> failure. > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> Gradle 4.3: > >> >>>>>>>>>>>>>>>>>>>>> I would like to defer the swap to Gradle 4.3 > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> until > >> >>>>> > >> >>>>>> after > >> >>>>>>>> > >> >>>>>>>>> this > >> >>>>>>>>>> > >> >>>>>>>>>>> PR > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> since > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> it > >> >>>>>>>>>>>>>>>>>>>>> will be a much smaller set of changes. > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> On Wed, Nov 8, 2017 at 12:54 AM, Jean-Baptiste > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> Onofré < > >> >>>>>>> > >> >>>>>>>> j...@nanthrax.net> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> Same for me for rat and python build too: > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> FAILURE: Build completed with 2 failures. > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> 1: Task failed with an exception. > >> >>>>>>>>>>>>>>>>>>>>>> ----------- > >> >>>>>>>>>>>>>>>>>>>>>> * What went wrong: > >> >>>>>>>>>>>>>>>>>>>>>> Execution failed for task ':rat'. > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> Found 905 files with unapproved/unknown > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> licenses. > >> >>>>>> > >> >>>>>>> See > >> >>>>>>>> > >> >>>>>>>>> file:/home/jbonofre/Workspace/ > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> beam/build/reports/rat/rat- > >> >>>>>>>>>> > >> >>>>>>>>>>> report.txt > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> * Try: > >> >>>>>>>>>>>>>>>>>>>>>> Run with --stacktrace option to get the stack > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> trace. > >> >>>>>>> > >> >>>>>>>> Run > >> >>>>>>>>> > >> >>>>>>>>>> with > >> >>>>>>>>>>> > >> >>>>>>>>>>>> --info > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> or > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> --debug option to get more log output. > >> >>>>>>>>>>>>>>>>>>>>>> ============================== > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> ============================== > >> >>>>>>>>>>> > >> >>>>>>>>>>>> ================== > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> 2: Task failed with an exception. > >> >>>>>>>>>>>>>>>>>>>>>> ----------- > >> >>>>>>>>>>>>>>>>>>>>>> * Where: > >> >>>>>>>>>>>>>>>>>>>>>> Build file '/home/jbonofre/Workspace/ > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> beam/sdks/python/build.gradle' > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> line: > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> 64 > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> * What went wrong: > >> >>>>>>>>>>>>>>>>>>>>>> Execution failed for task > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> ':beam-sdks-parent:beam-sdks- > >> >>>>>>>> > >> >>>>>>>>> python:lint'. > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> Process 'command 'tox'' finished with > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >
