Re: How Maven solves the problem of long builds on large projects?

[Mirko Friedenhagen]

Hello Sergey,

you may try to use
https://github.com/timgifford/maven-buildtime-extension to identify
which plugins contribute most to your long build times.

Regards
Mirko
Regards Mirko
--
http://illegalstateexception.blogspot.com/
https://github.com/mfriedenhagen/ (http://osrc.dfm.io/mfriedenhagen)
https://bitbucket.org/mfriedenhagen/


On Mon, Dec 21, 2015 at 1:53 PM, Sergey Saraev  wrote:
> Hello!
>
> I am developing a project with 67 modules.
> I use Apache Maven 3.0.4.
>
> Reassembly of the project take 1 hour and 50 minutes although usually commit 
> change only one module.
>
> The project is very large. It contains 5948 java classes (Basically, time 
> spent on their compilation.).
> Build command: mvn clean install pmd:pmd checkstyle:checkstyle 
> cobertura:cobertura
>
> Plugins versions:
> maven-compiler-plugin:2.3.2
> maven-antrun-plugin:1.6 (use wlappc task: 
> http://docs.oracle.com/cd/E21764_01/web./e13706/splitbuild.htm#WLPRG224)
> maven-surefire-plugin:2.10
> maven-jar-plugin:2.3.2
> maven-install-plugin:2.3.1
> maven-pmd-plugin:2.7.1
> maven-checkstyle-plugin:2.6
> cobertura-maven-plugin:2.7
>
> How to speed up the assembly?
> (Maybe skip modules, which sources have not changed or something else)
>
> Regards,
>
> Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext. 2662 
> | Mobile: +7 (917) 813-5604 | --www.NetCracker.com--
> Proven Partner to Communications Service Providers
>
>
>
>
> 
> The information transmitted herein is intended only for the person or entity 
> to which it is addressed and may contain confidential, proprietary and/or 
> privileged material. Any review, retransmission, dissemination or other use 
> of, or taking of any action in reliance upon, this information by persons or 
> entities other than the intended recipient is prohibited. If you received 
> this in error, please contact the sender and delete the material from any 
> computer.

-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: How Maven solves the problem of long builds on large projects?

[Bernd Eckenfels]

Am Mon, 21 Dec 2015 21:23:56 +0100
schrieb Mirko Friedenhagen :

> Hello Sergey,
> 
> you may try to use
> https://github.com/timgifford/maven-buildtime-extension to identify
> which plugins contribute most to your long build times.

I typically enable the timestamps in

maven-home/conf/logging/simplelogger.properties
  org.slf4j.simpleLogger.showDateTime=true

this can be done once for the maven home and all output has ms
precision stamps on the console.

Unfortunatelly this is not so easy to
turn on on automatically installed mavens in Jenkins (but jenkins has
its own extension to timestamp build output).

(however I think the OP is more interested in incremetal build
(plugins) or even Takari.io

Gruss
Bernd

-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: How Maven solves the problem of long builds on large projects?

[Ron Wheeler]

Ideas:
- Break your project up into modules that build separately. If you have 
a well structured project that is loosely coupled, individual 
issues/bugs will only affect a few modules.

 We have a project with about 80 modules but never build them all at once.
- Build separately deployable modules so that you are building small 
jars and wars that can be individually replaced in your run-time or test 
environments.
- Build modules with 3rd party libraries so that you only have 1 copy of 
each library in your project rather than a copy in each module. The JVM 
can only use 1 copy of Apache commons-xxx; no point deploying 50 copies.
- Make sure that you have a repo "near" your workstation so that 
downloads of maven components and third party libraries are fast. We use 
the community version of Nexus.
- Use a RAMDrive to hold Maven and your compiler and your build 
temporary files. I have dedicated 2 Gb of RAM on my 6 Gb laptop. makes a 
big difference.
 I also have Eclipse/STS on that drive which makes a big difference in 
working with the IDE. If I had more memory,
 I would put my Maven cache in RAM but it currently has 14,799 files 
and takes 1.81GB. I probably should delete it and let Maven rebuild it 
to get rid of old crap(9 versions of junit).

- Use the latest Maven (including plug-ins) and Java.
- Only clean when you need to. Maven does a pretty good job of figuring 
out what needs to be re-compiled.



The first 3 ideas will give the biggest improvement.
If you have the RAM, the RAMDrive is easiest to do since it does not 
require much thinking.


Ron

On 21/12/2015 8:17 AM, Stephen Connolly wrote:

I doubt it is the compilation that sucks up the time. Try running with test
execution skipped to see the time difference, eg `mvn clean install` vs
`mvn clean install -DskipTests`

If you are using `install` then likely there is no need to rebuild
everything each time and you can use the -pl -am and -and cli options to
tune how much of the reactor to rebuild

On Monday 21 December 2015, Sergey Saraev  wrote:


Hello!

I am developing a project with 67 modules.
I use Apache Maven 3.0.4.

Reassembly of the project take 1 hour and 50 minutes although usually
commit change only one module.

The project is very large. It contains 5948 java classes (Basically, time
spent on their compilation.).
Build command: mvn clean install pmd:pmd checkstyle:checkstyle
cobertura:cobertura

Plugins versions:
maven-compiler-plugin:2.3.2
maven-antrun-plugin:1.6 (use wlappc task:
http://docs.oracle.com/cd/E21764_01/web./e13706/splitbuild.htm#WLPRG224
)
maven-surefire-plugin:2.10
maven-jar-plugin:2.3.2
maven-install-plugin:2.3.1
maven-pmd-plugin:2.7.1
maven-checkstyle-plugin:2.6
cobertura-maven-plugin:2.7

How to speed up the assembly?
(Maybe skip modules, which sources have not changed or something else)

Regards,

Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext.
2662 | Mobile: +7 (917) 813-5604 | --www.NetCracker.com--
Proven Partner to Communications Service Providers





The information transmitted herein is intended only for the person or
entity to which it is addressed and may contain confidential, proprietary
and/or privileged material. Any review, retransmission, dissemination or
other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient is prohibited. If you
received this in error, please contact the sender and delete the material
from any computer.






--
Ron Wheeler
President
Artifact Software Inc
email: rwhee...@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102


-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

How Maven solves the problem of long builds on large projects?

[Sergey Saraev]

Hello!

I am developing a project with 67 modules.
I use Apache Maven 3.0.4.

Reassembly of the project take 1 hour and 50 minutes although usually commit 
change only one module.

The project is very large. It contains 5948 java classes (Basically, time spent 
on their compilation.).
Build command: mvn clean install pmd:pmd checkstyle:checkstyle 
cobertura:cobertura

Plugins versions:
maven-compiler-plugin:2.3.2
maven-antrun-plugin:1.6 (use wlappc task: 
http://docs.oracle.com/cd/E21764_01/web./e13706/splitbuild.htm#WLPRG224)
maven-surefire-plugin:2.10
maven-jar-plugin:2.3.2
maven-install-plugin:2.3.1
maven-pmd-plugin:2.7.1
maven-checkstyle-plugin:2.6
cobertura-maven-plugin:2.7

How to speed up the assembly?
(Maybe skip modules, which sources have not changed or something else)

Regards,

Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext. 2662 | 
Mobile: +7 (917) 813-5604 | --www.NetCracker.com--
Proven Partner to Communications Service Providers





The information transmitted herein is intended only for the person or entity to 
which it is addressed and may contain confidential, proprietary and/or 
privileged material. Any review, retransmission, dissemination or other use of, 
or taking of any action in reliance upon, this information by persons or 
entities other than the intended recipient is prohibited. If you received this 
in error, please contact the sender and delete the material from any computer.

Re: How Maven solves the problem of long builds on large projects?

[Graham Leggett]

On 21 Dec 2015, at 12:53, Sergey Saraev  wrote:

> I am developing a project with 67 modules.
> I use Apache Maven 3.0.4.
> 
> Reassembly of the project take 1 hour and 50 minutes although usually commit 
> change only one module.
> 
> The project is very large. It contains 5948 java classes (Basically, time 
> spent on their compilation.).
> Build command: mvn clean install pmd:pmd checkstyle:checkstyle 
> cobertura:cobertura

Get rid of "clean" - that means completely delete all existing code and 
recompile from scratch, which you don't want.

Get rid of pmd, checkstyle and cobertura until you need it.

Regards,
Graham
--


-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: How Maven solves the problem of long builds on large projects?

[Karl Heinz Marbaise]

Hi,

On 12/21/15 1:53 PM, Sergey Saraev wrote:

Hello!

I am developing a project with 67 modules.
I use Apache Maven 3.0.4.

Reassembly of the project take 1 hour and 50 minutes although usually commit 
change only one module.


You should use only: mvn clean install ...

any other things like pmd, checkstyle etc. don't make sense in a usual 
build..


Furthermore have you taken a look how long the build time of the 
different modules is? What is the module with the longest time? How many 
tests do you run? How long do the tests take?


I can give an impression of a large build (420 module) about 6500 tests, 
can be built in ca. 35 minutes ca. 580,000 lines of code...running (mvn 
clean deploy)...


BTW: Why are you using such old plugin versions? 
(http://maven.apache.org/plugins/)...


And of course on what kind of machine do you do the build? dedicated 
build machine?


Kind regards
Karl Heinz Marbaise



The project is very large. It contains 5948 java classes (Basically, time spent 
on their compilation.).
Build command: mvn clean install pmd:pmd checkstyle:checkstyle 
cobertura:cobertura

Plugins versions:
maven-compiler-plugin:2.3.2
maven-antrun-plugin:1.6 (use wlappc task: 
http://docs.oracle.com/cd/E21764_01/web./e13706/splitbuild.htm#WLPRG224)
maven-surefire-plugin:2.10
maven-jar-plugin:2.3.2
maven-install-plugin:2.3.1
maven-pmd-plugin:2.7.1
maven-checkstyle-plugin:2.6
cobertura-maven-plugin:2.7

How to speed up the assembly?
(Maybe skip modules, which sources have not changed or something else)

Regards,

Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext. 2662 | 
Mobile: +7 (917) 813-5604 | --www.NetCracker.com--
Proven Partner to Communications Service Providers




-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: How Maven solves the problem of long builds on large projects?

[Stephen Connolly]

I doubt it is the compilation that sucks up the time. Try running with test
execution skipped to see the time difference, eg `mvn clean install` vs
`mvn clean install -DskipTests`

If you are using `install` then likely there is no need to rebuild
everything each time and you can use the -pl -am and -and cli options to
tune how much of the reactor to rebuild

On Monday 21 December 2015, Sergey Saraev  wrote:

> Hello!
>
> I am developing a project with 67 modules.
> I use Apache Maven 3.0.4.
>
> Reassembly of the project take 1 hour and 50 minutes although usually
> commit change only one module.
>
> The project is very large. It contains 5948 java classes (Basically, time
> spent on their compilation.).
> Build command: mvn clean install pmd:pmd checkstyle:checkstyle
> cobertura:cobertura
>
> Plugins versions:
> maven-compiler-plugin:2.3.2
> maven-antrun-plugin:1.6 (use wlappc task:
> http://docs.oracle.com/cd/E21764_01/web./e13706/splitbuild.htm#WLPRG224
> )
> maven-surefire-plugin:2.10
> maven-jar-plugin:2.3.2
> maven-install-plugin:2.3.1
> maven-pmd-plugin:2.7.1
> maven-checkstyle-plugin:2.6
> cobertura-maven-plugin:2.7
>
> How to speed up the assembly?
> (Maybe skip modules, which sources have not changed or something else)
>
> Regards,
>
> Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext.
> 2662 | Mobile: +7 (917) 813-5604 | --www.NetCracker.com--
> Proven Partner to Communications Service Providers
>
>
>
>
> 
> The information transmitted herein is intended only for the person or
> entity to which it is addressed and may contain confidential, proprietary
> and/or privileged material. Any review, retransmission, dissemination or
> other use of, or taking of any action in reliance upon, this information by
> persons or entities other than the intended recipient is prohibited. If you
> received this in error, please contact the sender and delete the material
> from any computer.
>


-- 
Sent from my phone

Re: How Maven solves the problem of long builds on large projects?

[Oliver B. Fischer]

Hi,

we had a similar problem with our build. First step was to optimize our 
tests. By tracing the execution time of our tests we were able to reduce 
the time spent on integration tests from 1h to 15m.


We use Team City as CI server. Therefore we were able to parallelize our 
build. How we run in parallel static code analysis, unit test, component 
test and similar stuff.


Bye,

Oliver

Am 21.12.15 um 13:53 schrieb Sergey Saraev:

Hello!

I am developing a project with 67 modules.
I use Apache Maven 3.0.4.

Reassembly of the project take 1 hour and 50 minutes although usually commit 
change only one module.

The project is very large. It contains 5948 java classes (Basically, time spent 
on their compilation.).
Build command: mvn clean install pmd:pmd checkstyle:checkstyle 
cobertura:cobertura

Plugins versions:
maven-compiler-plugin:2.3.2
maven-antrun-plugin:1.6 (use wlappc task: 
http://docs.oracle.com/cd/E21764_01/web./e13706/splitbuild.htm#WLPRG224)
maven-surefire-plugin:2.10
maven-jar-plugin:2.3.2
maven-install-plugin:2.3.1
maven-pmd-plugin:2.7.1
maven-checkstyle-plugin:2.6
cobertura-maven-plugin:2.7

How to speed up the assembly?
(Maybe skip modules, which sources have not changed or something else)

Regards,

Sergey Saraev | Research & Development | Office: +7 (846) 270-7800 ext. 2662 | 
Mobile: +7 (917) 813-5604 | --www.NetCracker.com--
Proven Partner to Communications Service Providers





The information transmitted herein is intended only for the person or entity to 
which it is addressed and may contain confidential, proprietary and/or 
privileged material. Any review, retransmission, dissemination or other use of, 
or taking of any action in reliance upon, this information by persons or 
entities other than the intended recipient is prohibited. If you received this 
in error, please contact the sender and delete the material from any computer.



--
N Oliver B. Fischer
A Schönhauser Allee 64, 10437 Berlin, Deutschland/Germany
P +49 30 44793251
M +49 178 7903538
E o.b.fisc...@swe-blog.net
S oliver.b.fischer
J oliver.b.fisc...@jabber.org
X http://xing.to/obf


-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org