Parallel vs weave
I am picking up some positive tweets here and there, but I'd be really happy if those of you who test parallel building could report back with some numbers in response to this message. I am particularly interested in the difference between parallel and weave (and linear too), mostly to assess that weave is worth the effort. So after tweaking around with threads a bit to find out what runs fastest I'd really enjoy reports like this one (real data): Linear: 50 seconds Parallel: 40 seconds Weave: 29 seconds For a 10 module project using -T 1C on a 6 core box. For those wondering: * Both parallel and weave favorize projects with lots of modules and wide dependency graphs. * Weave is /probably/ quite a lot faster than parallel for projects with lots of tests fairly evenly distributed among the modules. * Putting all your tests in one module is a bad idea in this context. * Some mojos (like war/ear) can dwarf all the others, effectively limiting the overall potential somewhat. Kristian - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Parallel vs weave
Well, weave mode doesn't work for CXF. Ends up with a NullPointerException down in antrun-plugin someplace. Thus, I cannot help you there. For CXF on my i7 820QM (4 cores, 8 threads) turning off checkstyle/pmd and having everything already code generated/compiled (so mostly just running tests) Linear: 26 minutes Parallel: 9 minutes (-T 12 is what I'm using) Dan On Friday 11 June 2010 12:24:13 pm Kristian Rosenvold wrote: I am picking up some positive tweets here and there, but I'd be really happy if those of you who test parallel building could report back with some numbers in response to this message. I am particularly interested in the difference between parallel and weave (and linear too), mostly to assess that weave is worth the effort. So after tweaking around with threads a bit to find out what runs fastest I'd really enjoy reports like this one (real data): Linear: 50 seconds Parallel: 40 seconds Weave: 29 seconds For a 10 module project using -T 1C on a 6 core box. For those wondering: * Both parallel and weave favorize projects with lots of modules and wide dependency graphs. * Weave is /probably/ quite a lot faster than parallel for projects with lots of tests fairly evenly distributed among the modules. * Putting all your tests in one module is a bad idea in this context. * Some mojos (like war/ear) can dwarf all the others, effectively limiting the overall potential somewhat. Kristian - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org -- Daniel Kulp dk...@apache.org http://dankulp.com/blog - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Parallel vs weave
Interesting info, and good numbers! Weave is a double edged sword; while it may be promising even faster builds, by design it has this irritating tendency to run mojos (of the same execution in different modules) *exactly* in parallel, more or less simulating how I'd write a unit test to test thread safety. So weave has proved to find almost *all* of the concurrency bugs I've located (where parallel doesn't have any problems). There are also a couple of minor details regarding the actual execution planning in weave mode that are still somewhat open - it's a much more complex execution model than parallel; but I'm concentrating on parallel and the ecosystem in general ATM and just making sure weave works for every project I test. Antrun itself is threadsafe, but whatever is being executed there is another matter. If this is committed to cxf trunk I can/will look at it once I get through the known issues. Kristian On Fri, Jun 11, 2010 at 6:57 PM, Daniel Kulp dk...@apache.org wrote: Well, weave mode doesn't work for CXF. Ends up with a NullPointerException down in antrun-plugin someplace. Thus, I cannot help you there. For CXF on my i7 820QM (4 cores, 8 threads) turning off checkstyle/pmd and having everything already code generated/compiled (so mostly just running tests) Linear: 26 minutes Parallel: 9 minutes (-T 12 is what I'm using) Dan On Friday 11 June 2010 12:24:13 pm Kristian Rosenvold wrote: I am picking up some positive tweets here and there, but I'd be really happy if those of you who test parallel building could report back with some numbers in response to this message. I am particularly interested in the difference between parallel and weave (and linear too), mostly to assess that weave is worth the effort. So after tweaking around with threads a bit to find out what runs fastest I'd really enjoy reports like this one (real data): Linear: 50 seconds Parallel: 40 seconds Weave: 29 seconds For a 10 module project using -T 1C on a 6 core box. For those wondering: * Both parallel and weave favorize projects with lots of modules and wide dependency graphs. * Weave is /probably/ quite a lot faster than parallel for projects with lots of tests fairly evenly distributed among the modules. * Putting all your tests in one module is a bad idea in this context. * Some mojos (like war/ear) can dwarf all the others, effectively limiting the overall potential somewhat. Kristian - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org -- Daniel Kulp dk...@apache.org http://dankulp.com/blog - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Parallel vs weave
On Friday 11 June 2010 1:40:07 pm Kristian Rosenvold wrote: Interesting info, and good numbers! Weave is a double edged sword; while it may be promising even faster builds, by design it has this irritating tendency to run mojos (of the same execution in different modules) *exactly* in parallel, more or less simulating how I'd write a unit test to test thread safety. So weave has proved to find almost *all* of the concurrency bugs I've located (where parallel doesn't have any problems). There are also a couple of minor details regarding the actual execution planning in weave mode that are still somewhat open - it's a much more complex execution model than parallel; but I'm concentrating on parallel and the ecosystem in general ATM and just making sure weave works for every project I test. Antrun itself is threadsafe, but whatever is being executed there is another matter. If this is committed to cxf trunk I can/will look at it once I get through the known issues. Yep. To reproduce 1) checkout CXF trunk from: http://svn.apache.org/repos/asf/cxf/trunk/ 2) Build everything (non-parallel right now) mvn -Pnochecks -Dmaven.test.skip.exec=true install That gets everything set and all the code generation done and such. (I'm pretty sure not all the code gen plugins are threadsafe yet, but they all do nothing if they don't have to do anything) Then just run: mvn -Pnochecks -T 12W or similar. For me, 100% of the time it barfs in antrun in the testutils module. You need the -Pnochecks right now when using -T until we get a release of checkstyle that is threadsafe. Alternatively, update the parent/pom.xml to use the snapshots of checkstyle and pmd. Dan Kristian On Fri, Jun 11, 2010 at 6:57 PM, Daniel Kulp dk...@apache.org wrote: Well, weave mode doesn't work for CXF. Ends up with a NullPointerException down in antrun-plugin someplace. Thus, I cannot help you there. For CXF on my i7 820QM (4 cores, 8 threads) turning off checkstyle/pmd and having everything already code generated/compiled (so mostly just running tests) Linear: 26 minutes Parallel: 9 minutes (-T 12 is what I'm using) Dan On Friday 11 June 2010 12:24:13 pm Kristian Rosenvold wrote: I am picking up some positive tweets here and there, but I'd be really happy if those of you who test parallel building could report back with some numbers in response to this message. I am particularly interested in the difference between parallel and weave (and linear too), mostly to assess that weave is worth the effort. So after tweaking around with threads a bit to find out what runs fastest I'd really enjoy reports like this one (real data): Linear: 50 seconds Parallel: 40 seconds Weave: 29 seconds For a 10 module project using -T 1C on a 6 core box. For those wondering: * Both parallel and weave favorize projects with lots of modules and wide dependency graphs. * Weave is /probably/ quite a lot faster than parallel for projects with lots of tests fairly evenly distributed among the modules. * Putting all your tests in one module is a bad idea in this context. * Some mojos (like war/ear) can dwarf all the others, effectively limiting the overall potential somewhat. Kristian - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org -- Daniel Kulp dk...@apache.org http://dankulp.com/blog - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org -- Daniel Kulp dk...@apache.org http://dankulp.com/blog - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
