I'm looking a bit at aggregator goals. At first glance, it seems most of the problem comes from the fact that the build ordering is done mostly per-project. This causes obvious problems for aggregators when a mojo needs a given phase to be executed for all children before it can run (for example the compilation phase before creating the javadoc). Another problem comes from the clean task which has obvious side effects. Calling clean with an aggregator goal involved in the build seems like a recipe for problems...
Would you see bad side effects to split the build in smaller chunks (basically down to a mojo execution) so that ordering can be changed in a more meaningful way ? For example when running an aggregate javadoc, the top level project would be built first until before the javadoc aggregate goal. The build would then go to all other projects (like the forked lifecycle) until all required goals have been run, then resume from the javadoc goal. Note that with the current behavior, when you run "clean verify", the execution is the following [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] forked [pom] [INFO] forked-mod1 [jar] [INFO] forked-mod2 [jar] [INFO] [INFO] ---------------< org.mvndaemon.mvnd.test.forked:forked >---------------- [INFO] Building forked 0.0.1-SNAPSHOT [1/3] [INFO] --------------------------------[ pom ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked --- [INFO] >>> maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) > compile @ forked >>> [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] Forking forked-mod1 0.0.1-SNAPSHOT [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ forked-mod1 --- [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ forked-mod1 --- [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] Forking forked-mod2 0.0.1-SNAPSHOT [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ forked-mod2 --- [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ forked-mod2 --- [INFO] <<< maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) < compile @ forked <<< [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) @ forked --- [INFO] [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod1 >------------- [INFO] Building forked-mod1 0.0.1-SNAPSHOT [2/3] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod1 --- [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ forked-mod1 --- [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ forked-mod1 --- [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ forked-mod1 --- [INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ forked-mod1 --- [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ forked-mod1 --- [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod1 --- [INFO] [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod2 >------------- [INFO] Building forked-mod2 0.0.1-SNAPSHOT [3/3] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod2 --- [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ forked-mod2 --- [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ forked-mod2 --- [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ forked-mod2 --- [INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ forked-mod2 --- [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ forked-mod2 --- [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod2 --- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for forked 0.0.1-SNAPSHOT: [INFO] [INFO] forked ............................................. SUCCESS [ 1.921 s] [INFO] forked-mod1 ........................................ SUCCESS [ 0.498 s] [INFO] forked-mod2 ........................................ SUCCESS [ 0.046 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2.535 s [INFO] Finished at: 2021-05-18T14:32:09+02:00 [INFO] ------------------------------------------------------------------------ As you can see, the resources and compile goals are run twice for each project. Furthermore, given the clean goal is in the loop, the compiler actually compiles things twice. The build plan could be rewritten as [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] forked [pom] [INFO] forked-mod1 [jar] [INFO] forked-mod2 [jar] [INFO] [INFO] ---------------< org.mvndaemon.mvnd.test.forked:forked >---------------- [INFO] Building forked 0.0.1-SNAPSHOT [1/3] [INFO] --------------------------------[ pom ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked --- ... normal lifecycle interrupted because the next goal is the aggregator goal ... [INFO] [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod1 >------------- [INFO] Building forked-mod1 0.0.1-SNAPSHOT [2/3] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod1 --- [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ forked-mod1 --- [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ forked-mod1 --- ... interrupting forked-mod1 build as it reached the required point execution ... [INFO] [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod2 >------------- [INFO] Building forked-mod2 0.0.1-SNAPSHOT [3/3] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod2 --- [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ forked-mod2 --- [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ forked-mod2 --- ... interrupting forked-mod2 build as it reached the required point execution ... ... resuming the top level project... [INFO] [INFO] ---------------< org.mvndaemon.mvnd.test.forked:forked >---------------- [INFO] Building forked 0.0.1-SNAPSHOT [1/3] [INFO] --------------------------------[ pom ]--------------------------------- [INFO] [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) @ forked --- ... then resuming the forked-mod1 project... [INFO] [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod1 >------------- [INFO] Building forked-mod1 0.0.1-SNAPSHOT [2/3] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ forked-mod1 --- [INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ forked-mod1 --- [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) @ forked --- [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ forked-mod1 --- [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod1 --- ... and resuming the forked-mod2 project... [INFO] [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod2 >------------- [INFO] Building forked-mod2 0.0.1-SNAPSHOT [3/3] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ forked-mod2 --- [INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ forked-mod2 --- [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ forked-mod2 --- [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod2 --- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for forked 0.0.1-SNAPSHOT: [INFO] [INFO] forked ............................................. SUCCESS [ 1.921 s] [INFO] forked-mod1 ........................................ SUCCESS [ 0.498 s] [INFO] forked-mod2 ........................................ SUCCESS [ 0.046 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2.535 s [INFO] Finished at: 2021-05-18T14:32:09+02:00 [INFO] ------------------------------------------------------------------------ This would also solve the parallel build problem because each project would not be able to be both "cleaned" and "compiled" concurrently, which is currently possible in the current state. Le lun. 17 mai 2021 à 14:06, Guillaume Nodet <gno...@apache.org> a écrit : > I've raised https://issues.apache.org/jira/browse/MNG-7156 > > Le mer. 12 mai 2021 à 17:57, Falko Modler <f.mod...@gmx.net> a écrit : > >> Hi Guillaume, >> >> aggregation goals and parallel builds in combination are a bit of a >> mess, e.g.: >> >> - https://issues.apache.org/jira/browse/MNG-6843 >> - https://github.com/apache/maven/pull/413 >> - https://www.mail-archive.com/dev@maven.apache.org/msg123439.html >> >> Cheers, >> >> Falko >> >> Am 12.05.2021 um 17:25 schrieb Guillaume Nodet: >> > Hi >> > >> > I've analyzed a bug reported on mvnd this afternoon ( >> > https://github.com/mvndaemon/mvnd/issues/408). It appears that the >> parent >> > pom executes the javadoc aggregate goal, which forks the lifecycle of >> the >> > children modules in order to compile the sources. In a traditional >> build, >> > this does not cause any real problem, but in a parallel build, a clean >> > verify can definitely cause issues if the forked lifecycle and the >> normal >> > project build (and especially the clean) are run concurrently. >> > This definitely looks like an issue to me. Any idea where I should >> look at >> > how to solve the problem ? I wonder if the MojoExecutor should somehow >> > delegate to the Builder which is responsible for synchronizing the >> > executions in the case of a multithreaded build... >> > Thoughts ? >> > >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org >> For additional commands, e-mail: dev-h...@maven.apache.org >> >> > > -- > ------------------------ > Guillaume Nodet > > -- ------------------------ Guillaume Nodet
