Great findings then! It’s not easy to catch and my experience is that such big projects are often in corporate environments which aren’t open at all and not really working with oss projects thus it’s not surprising to have this surfacing on Gradle support side.
Le mer. 24 avr. 2019 à 19:09, Karl Heinz Marbaise <[email protected]> a écrit : > Hi, > > On 24.04.19 17:52, Robert Scholte wrote: > > On Wed, 24 Apr 2019 12:45:42 +0200, Arnaud Héritier > > <[email protected]> wrote: > > > >> Just for my knowledge, is it a regression in a recent version or it is > >> like > >> this for ages ? > > > > It is like this for ages. AFAIK nobody of the team monitored memory > > consumption, focus was on correct + "readable" code. > > I had done some investigations a longer time ago...related to MNG-6030 > https://github.com/khmarbaise/maven-test-project-generator > > and yes it's not a regression... > > Kind regards > Karl Heinz > > > > > Robert > > > >> > >> > >> Le mer. 24 avr. 2019 à 10:56, Benedikt Ritter <[email protected]> a > >> écrit : > >> > >>> Am Mi., 24. Apr. 2019 um 10:50 Uhr schrieb Benedikt Ritter < > >>> [email protected]>: > >>> > >>> > Hello, > >>> > > >>> > this is a summary of a video conference call that happened yesterday > >>> > (April 24). > >>> > > >>> > >>> Sorry, actually yesterday was April 23... :o) > >>> > >>> > >>> > > >>> > Topic: > >>> > Discussion about performance improvements that have been proposed by > >>> > Stefan Oehme, namely: > >>> > > >>> > - [MNG-6638] - Prevent reparsing POMs in MavenMetadataSource ( > >>> > https://github.com/apache/maven/pull/244) > >>> > - [MNG-6633] - Reduce memory usage of excludes ( > >>> > https://github.com/apache/maven/pull/243) - Speed up project > >>> discovery ( > >>> > https://github.com/apache/maven/pull/242) - Make location handling > >>> more > >>> > memory efficient (https://github.com/codehaus-plexus/modello/pull/31 > ) > >>> > > >>> > The goal of this call was to give some more insights into how Stefan > >>> found > >>> > the improvements and to better understand what is missing before > these > >>> > changes be merged. > >>> > > >>> > Attendees of the call: > >>> > - Benedikt Ritter (Gradle Inc.) > >>> > - Stefan Oehme (Gradle Inc.) > >>> > - Robert Scholte (Apache Maven Team) > >>> > - Hervé Boutemy (Apache Maven Team; joined about half an hour after > >>> the > >>> > call started) > >>> > > >>> > Summary: > >>> > > >>> > Stefan gave some insights into how he discovered bottlenecks in > Maven: > >>> > > >>> > - > >>> > > >>> > One of our customers has a huge Maven build: > >>> > - > >>> > > >>> > Lots of sub projects (2000) > >>> > - > >>> > > >>> > Lots of entries in dependency management (4000) > >>> > - > >>> > > >>> > Results in a lot of garbage collection > >>> > - > >>> > > >>> > Problems discovered in that build: > >>> > - > >>> > > >>> > Re-parsing project POMs during dependency resolution > >>> > - > >>> > > >>> > Model objects are too large because of location tracking > >>> > - > >>> > > >>> > Low-level bottlenecks in project discovery (especially version > >>> > parsing) > >>> > - > >>> > > >>> > Customer now has a Maven fork with the proposed changes included: > >>> > - > >>> > > >>> > 1h 50min, 12GB RAM without changes > >>> > - > >>> > > >>> > 45min, 8GB RAM with changes > >>> > > >>> > > >>> > Robert: > >>> > > >>> > - > >>> > > >>> > How to ensure that improvements are not broken? > >>> > - > >>> > > >>> > No answer to how to test this > >>> > > >>> > > >>> > Stefan gave some insights into how performance testing works in the > >>> Gradle > >>> > project: > >>> > > >>> > - > >>> > > >>> > Build has a project generator > >>> > - > >>> > > >>> > Create different projects in different shapes (e.g. lots of > >>> > subprojects, deeply nested projects) during the build > >>> > - > >>> > > >>> > Download old Gradle version and run the build on generated > projects > >>> > - > >>> > > >>> > Run build again with current Gradle version > >>> > - > >>> > > >>> > Compare results > >>> > - > >>> > > >>> > use statistic methods to filter out variance > >>> > - > >>> > > >>> > Downside to this approach is that it requires a lot of computing > >>> > resources > >>> > > >>> > More information can be found on GitHub: > >>> > https://github.com/gradle/gradle/tree/master/subprojects/performance > >>> > The corresponding TeamCity build can be found here: > >>> > > >>> > https://builds.gradle.org/viewLog.html?buildId=22179604&buildTypeId=Gradle_Check_PerformanceExperimentCoordinator&tab=report_project941_Performance&branch_Gradle_Check_Stage_ReadyforRelease=master > >>> > >>> (use > >>> > "Login as guest" to view) > >>> > > >>> > Robert: > >>> > > >>> > - > >>> > > >>> > What about measuring performance using instruction calls? > >>> > > >>> > > >>> > Stefan: > >>> > > >>> > - > >>> > > >>> > The performance improvements we found were mostly about garbage > >>> being > >>> > created > >>> > - > >>> > > >>> > Measuring using instruction calls is interesting > >>> > - > >>> > > >>> > ... but it is also very machine dependent > >>> > > >>> > > >>> > Robert: > >>> > > >>> > - > >>> > > >>> > We need to find out who is interested in these kind improvements > >>> > inside the Maven community. > >>> > - > >>> > > >>> > Build a community of people who would like to work on these kind > of > >>> > things. > >>> > > >>> > > >>> > Stefan: > >>> > > >>> > - > >>> > > >>> > It's easy to get started. We just used open source tools: > >>> > - > >>> > > >>> > We used async-profiler for measuring things ( > >>> > https://github.com/jvm-profiling-tools/async-profiler) > >>> > - > >>> > > >>> > Heap dumps for analyzing memory usage > >>> > > >>> > To get started with performance tests in the maven project: > >>> > > >>> > - > >>> > > >>> > Start with only a few test projects > >>> > - > >>> > > >>> > The Gradle generator is Apache License v2 and can be used as a > >>> > starting point to generate a big maven project > >>> > > >>> > > >>> > Hervé: > >>> > > >>> > - > >>> > > >>> > PRs should be merged soon > >>> > - > >>> > > >>> > Discussion need to be resolved > >>> > - > >>> > > >>> > Why was the PR not merged after the discussion and resolving all > >>> > issues with the code? > >>> > - > >>> > > >>> > Hervé will take care that the changes are merged soon > >>> > > >>> > > >>> > Thank you! > >>> > Benedikt > >>> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- ----- Arnaud Héritier http://aheritier.net Mail/GTalk: aheritier AT gmail DOT com Twitter/Skype : aheritier
