Re: How Maven solves the problem of long builds on large projects?
Am Mon, 21 Dec 2015 21:23:56 +0100 schrieb Mirko Friedenhagen : > Hello Sergey, > > you may try to use > https://github.com/timgifford/maven-buildtime-extension to identify > which plugins contribute most to your long build times. I typically enable the timestamps in maven-home/conf/logging/simplelogger.properties org.slf4j.simpleLogger.showDateTime=true this can be done once for the maven home and all output has ms precision stamps on the console. Unfortunatelly this is not so easy to turn on on automatically installed mavens in Jenkins (but jenkins has its own extension to timestamp build output). (however I think the OP is more interested in incremetal build (plugins) or even Takari.io Gruss Bernd - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: How Maven solves the problem of long builds on large projects?
Hello Sergey, you may try to use https://github.com/timgifford/maven-buildtime-extension to identify which plugins contribute most to your long build times. Regards Mirko Regards Mirko -- http://illegalstateexception.blogspot.com/ https://github.com/mfriedenhagen/ (http://osrc.dfm.io/mfriedenhagen) https://bitbucket.org/mfriedenhagen/ On Mon, Dec 21, 2015 at 1:53 PM, Sergey Saraev wrote: > Hello! > > I am developing a project with 67 modules. > I use Apache Maven 3.0.4. > > Reassembly of the project take 1 hour and 50 minutes although usually commit > change only one module. > > The project is very large. It contains 5948 java classes (Basically, time > spent on their compilation.). > Build command: mvn clean install pmd:pmd checkstyle:checkstyle > cobertura:cobertura > > Plugins versions: > maven-compiler-plugin:2.3.2 > maven-antrun-plugin:1.6 (use wlappc task: > http://docs.oracle.com/cd/E21764_01/web./e13706/splitbuild.htm#WLPRG224) > maven-surefire-plugin:2.10 > maven-jar-plugin:2.3.2 > maven-install-plugin:2.3.1 > maven-pmd-plugin:2.7.1 > maven-checkstyle-plugin:2.6 > cobertura-maven-plugin:2.7 > > How to speed up the assembly? > (Maybe skip modules, which sources have not changed or something else) > > Regards, > > Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext. 2662 > | Mobile: +7 (917) 813-5604 | --www.NetCracker.com-- > Proven Partner to Communications Service Providers > > > > > > The information transmitted herein is intended only for the person or entity > to which it is addressed and may contain confidential, proprietary and/or > privileged material. Any review, retransmission, dissemination or other use > of, or taking of any action in reliance upon, this information by persons or > entities other than the intended recipient is prohibited. If you received > this in error, please contact the sender and delete the material from any > computer. - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: How Maven solves the problem of long builds on large projects?
Hi, we had a similar problem with our build. First step was to optimize our tests. By tracing the execution time of our tests we were able to reduce the time spent on integration tests from 1h to 15m. We use Team City as CI server. Therefore we were able to parallelize our build. How we run in parallel static code analysis, unit test, component test and similar stuff. Bye, Oliver Am 21.12.15 um 13:53 schrieb Sergey Saraev: Hello! I am developing a project with 67 modules. I use Apache Maven 3.0.4. Reassembly of the project take 1 hour and 50 minutes although usually commit change only one module. The project is very large. It contains 5948 java classes (Basically, time spent on their compilation.). Build command: mvn clean install pmd:pmd checkstyle:checkstyle cobertura:cobertura Plugins versions: maven-compiler-plugin:2.3.2 maven-antrun-plugin:1.6 (use wlappc task: http://docs.oracle.com/cd/E21764_01/web./e13706/splitbuild.htm#WLPRG224) maven-surefire-plugin:2.10 maven-jar-plugin:2.3.2 maven-install-plugin:2.3.1 maven-pmd-plugin:2.7.1 maven-checkstyle-plugin:2.6 cobertura-maven-plugin:2.7 How to speed up the assembly? (Maybe skip modules, which sources have not changed or something else) Regards, Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext. 2662 | Mobile: +7 (917) 813-5604 | --www.NetCracker.com-- Proven Partner to Communications Service Providers The information transmitted herein is intended only for the person or entity to which it is addressed and may contain confidential, proprietary and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. -- N Oliver B. Fischer A Schönhauser Allee 64, 10437 Berlin, Deutschland/Germany P +49 30 44793251 M +49 178 7903538 E o.b.fisc...@swe-blog.net S oliver.b.fischer J oliver.b.fisc...@jabber.org X http://xing.to/obf - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: How Maven solves the problem of long builds on large projects?
Ideas: - Break your project up into modules that build separately. If you have a well structured project that is loosely coupled, individual issues/bugs will only affect a few modules. We have a project with about 80 modules but never build them all at once. - Build separately deployable modules so that you are building small jars and wars that can be individually replaced in your run-time or test environments. - Build modules with 3rd party libraries so that you only have 1 copy of each library in your project rather than a copy in each module. The JVM can only use 1 copy of Apache commons-xxx; no point deploying 50 copies. - Make sure that you have a repo "near" your workstation so that downloads of maven components and third party libraries are fast. We use the community version of Nexus. - Use a RAMDrive to hold Maven and your compiler and your build temporary files. I have dedicated 2 Gb of RAM on my 6 Gb laptop. makes a big difference. I also have Eclipse/STS on that drive which makes a big difference in working with the IDE. If I had more memory, I would put my Maven cache in RAM but it currently has 14,799 files and takes 1.81GB. I probably should delete it and let Maven rebuild it to get rid of old crap(9 versions of junit). - Use the latest Maven (including plug-ins) and Java. - Only clean when you need to. Maven does a pretty good job of figuring out what needs to be re-compiled. The first 3 ideas will give the biggest improvement. If you have the RAM, the RAMDrive is easiest to do since it does not require much thinking. Ron On 21/12/2015 8:17 AM, Stephen Connolly wrote: I doubt it is the compilation that sucks up the time. Try running with test execution skipped to see the time difference, eg `mvn clean install` vs `mvn clean install -DskipTests` If you are using `install` then likely there is no need to rebuild everything each time and you can use the -pl -am and -and cli options to tune how much of the reactor to rebuild On Monday 21 December 2015, Sergey Saraev wrote: Hello! I am developing a project with 67 modules. I use Apache Maven 3.0.4. Reassembly of the project take 1 hour and 50 minutes although usually commit change only one module. The project is very large. It contains 5948 java classes (Basically, time spent on their compilation.). Build command: mvn clean install pmd:pmd checkstyle:checkstyle cobertura:cobertura Plugins versions: maven-compiler-plugin:2.3.2 maven-antrun-plugin:1.6 (use wlappc task: http://docs.oracle.com/cd/E21764_01/web./e13706/splitbuild.htm#WLPRG224 ) maven-surefire-plugin:2.10 maven-jar-plugin:2.3.2 maven-install-plugin:2.3.1 maven-pmd-plugin:2.7.1 maven-checkstyle-plugin:2.6 cobertura-maven-plugin:2.7 How to speed up the assembly? (Maybe skip modules, which sources have not changed or something else) Regards, Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext. 2662 | Mobile: +7 (917) 813-5604 | --www.NetCracker.com-- Proven Partner to Communications Service Providers The information transmitted herein is intended only for the person or entity to which it is addressed and may contain confidential, proprietary and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. -- Ron Wheeler President Artifact Software Inc email: rwhee...@artifact-software.com skype: ronaldmwheeler phone: 866-970-2435, ext 102 - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: How Maven solves the problem of long builds on large projects?
I doubt it is the compilation that sucks up the time. Try running with test execution skipped to see the time difference, eg `mvn clean install` vs `mvn clean install -DskipTests` If you are using `install` then likely there is no need to rebuild everything each time and you can use the -pl -am and -and cli options to tune how much of the reactor to rebuild On Monday 21 December 2015, Sergey Saraev wrote: > Hello! > > I am developing a project with 67 modules. > I use Apache Maven 3.0.4. > > Reassembly of the project take 1 hour and 50 minutes although usually > commit change only one module. > > The project is very large. It contains 5948 java classes (Basically, time > spent on their compilation.). > Build command: mvn clean install pmd:pmd checkstyle:checkstyle > cobertura:cobertura > > Plugins versions: > maven-compiler-plugin:2.3.2 > maven-antrun-plugin:1.6 (use wlappc task: > http://docs.oracle.com/cd/E21764_01/web./e13706/splitbuild.htm#WLPRG224 > ) > maven-surefire-plugin:2.10 > maven-jar-plugin:2.3.2 > maven-install-plugin:2.3.1 > maven-pmd-plugin:2.7.1 > maven-checkstyle-plugin:2.6 > cobertura-maven-plugin:2.7 > > How to speed up the assembly? > (Maybe skip modules, which sources have not changed or something else) > > Regards, > > Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext. > 2662 | Mobile: +7 (917) 813-5604 | --www.NetCracker.com-- > Proven Partner to Communications Service Providers > > > > > > The information transmitted herein is intended only for the person or > entity to which it is addressed and may contain confidential, proprietary > and/or privileged material. Any review, retransmission, dissemination or > other use of, or taking of any action in reliance upon, this information by > persons or entities other than the intended recipient is prohibited. If you > received this in error, please contact the sender and delete the material > from any computer. > -- Sent from my phone
Re: How Maven solves the problem of long builds on large projects?
Hi, On 12/21/15 1:53 PM, Sergey Saraev wrote: Hello! I am developing a project with 67 modules. I use Apache Maven 3.0.4. Reassembly of the project take 1 hour and 50 minutes although usually commit change only one module. You should use only: mvn clean install ... any other things like pmd, checkstyle etc. don't make sense in a usual build.. Furthermore have you taken a look how long the build time of the different modules is? What is the module with the longest time? How many tests do you run? How long do the tests take? I can give an impression of a large build (420 module) about 6500 tests, can be built in ca. 35 minutes ca. 580,000 lines of code...running (mvn clean deploy)... BTW: Why are you using such old plugin versions? (http://maven.apache.org/plugins/)... And of course on what kind of machine do you do the build? dedicated build machine? Kind regards Karl Heinz Marbaise The project is very large. It contains 5948 java classes (Basically, time spent on their compilation.). Build command: mvn clean install pmd:pmd checkstyle:checkstyle cobertura:cobertura Plugins versions: maven-compiler-plugin:2.3.2 maven-antrun-plugin:1.6 (use wlappc task: http://docs.oracle.com/cd/E21764_01/web./e13706/splitbuild.htm#WLPRG224) maven-surefire-plugin:2.10 maven-jar-plugin:2.3.2 maven-install-plugin:2.3.1 maven-pmd-plugin:2.7.1 maven-checkstyle-plugin:2.6 cobertura-maven-plugin:2.7 How to speed up the assembly? (Maybe skip modules, which sources have not changed or something else) Regards, Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext. 2662 | Mobile: +7 (917) 813-5604 | --www.NetCracker.com-- Proven Partner to Communications Service Providers - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: How Maven solves the problem of long builds on large projects?
On 21 Dec 2015, at 12:53, Sergey Saraev wrote: > I am developing a project with 67 modules. > I use Apache Maven 3.0.4. > > Reassembly of the project take 1 hour and 50 minutes although usually commit > change only one module. > > The project is very large. It contains 5948 java classes (Basically, time > spent on their compilation.). > Build command: mvn clean install pmd:pmd checkstyle:checkstyle > cobertura:cobertura Get rid of "clean" - that means completely delete all existing code and recompile from scratch, which you don't want. Get rid of pmd, checkstyle and cobertura until you need it. Regards, Graham -- - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
How Maven solves the problem of long builds on large projects?
Hello! I am developing a project with 67 modules. I use Apache Maven 3.0.4. Reassembly of the project take 1 hour and 50 minutes although usually commit change only one module. The project is very large. It contains 5948 java classes (Basically, time spent on their compilation.). Build command: mvn clean install pmd:pmd checkstyle:checkstyle cobertura:cobertura Plugins versions: maven-compiler-plugin:2.3.2 maven-antrun-plugin:1.6 (use wlappc task: http://docs.oracle.com/cd/E21764_01/web./e13706/splitbuild.htm#WLPRG224) maven-surefire-plugin:2.10 maven-jar-plugin:2.3.2 maven-install-plugin:2.3.1 maven-pmd-plugin:2.7.1 maven-checkstyle-plugin:2.6 cobertura-maven-plugin:2.7 How to speed up the assembly? (Maybe skip modules, which sources have not changed or something else) Regards, Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext. 2662 | Mobile: +7 (917) 813-5604 | --www.NetCracker.com-- Proven Partner to Communications Service Providers The information transmitted herein is intended only for the person or entity to which it is addressed and may contain confidential, proprietary and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
