Le sam. 29 mai 2021 à 10:13, Martin Kanters <martinkant...@apache.org> a écrit :
> So we took a look at this. First we wanted to know why the forked > goals were executed multiple times, while MNG-6566 [1] should have taken > care of that. But it seems that that only optimizes duplicate goals in the > context of one project. I attempted to explain that [2]. We created a > ticket for optimizing this for multi module projects [3], but we think that > it's not straightforward to fix this. > I still think [3] is slightly different from my original problem. You seem to indicate that the forked lifecycle is executed after, while in my case, they were executed at the very beginning of the build, so that the forked goals were executed before the usual execution from their respective project. The effect is that for a given project, the following goals were executed: compile -> clean -> compile The second problem is that the concurrent execution of the clean and compile goals lead to unpredictable results. It seems the issue only stresses a re-execution of the goal as a possible optimization... That may be just me not reading correctly though... > Back to the problem at hand. IIUC your suggestion is to pause > project builds halfway, resume them after the required child projects goals > have been executed (instead of forking), and finally finish the child > project builds afterwards. I think it's elegant that in this solution, no > duplicate goals will be executed, since no real forking has to be done. I > do have the feeling that it would become quite a drastic change. I am also > not sure whether we can assume that switching halfway through project > builds works fine in every scenario. For the user it would also become hard > to follow the build, I guess... > > Perhaps it would be easier to ensure projects can only be built once at > a time (e.g. using locks or something). It wouldn't be the most > efficient, but I guess it does not involve big changes to both the inner > workings and user perspective. I could try the second solution, though this could still leave an execution order: compile -> clean -> compile which I think is really wrong. I think the first solution brings a few benefits : during parallel builds, more things could be executed in parallel. For example the ordering could be improved at the mojo execution level (or at least per group of mojos execution in the same phase) instead of a global per-project ordering. In theory, we could even make the project resolution happen in parallel with some goals execution: compile time dependencies could be resolved during the clean / resource generation phases, and test dependencies during compilation, etc... Also, dependencies between projects only really happen for the compilation (or test) phases, so the clean / resource generation phases can happen concurrently. For the output, I'm not so worried. In the case of parallel builds, the output has to be reordered anyway to be understandable, as done in mvnd. I'll have a look at a simple per-project locking mechanism first though... > Martin > > [1] https://issues.apache.org/jira/browse/MNG-6566 > [2] > > https://issues.apache.org/jira/browse/MNG-6566?focusedCommentId=17353384&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17353384 > [3] https://issues.apache.org/jira/browse/MNG-7163 > > Op di 18 mei 2021 om 21:45 schreef Martin Kanters < > martinkant...@apache.org > >: > > > Ah, too bad. I thought it was relevant. Perhaps I can take a look at this > > with Maarten Mulders, but that will not be in the coming week at least. > > > > Op di 18 mei 2021 om 21:34 schreef Guillaume Nodet <gno...@apache.org>: > > > >> I just tried with master and the problem is the same. > >> The reason is that in my use case, the forked goals are executed before > >> the > >> standard executions, and not even in the same project. > >> > >> Le mar. 18 mai 2021 à 20:41, Martin Kanters <martinkant...@apache.org> > a > >> écrit : > >> > >> > Not sure how I've missed this post. Have you tried this build with the > >> > master build of Maven? > >> > MNG-6566 [1] should prevent any unnecessary double executions, thus > >> > optimizing the buildplan. > >> > I'm interested if this will solve the problem at hand. > >> > > >> > Martin > >> > > >> > [1] https://issues.apache.org/jira/browse/MNG-6566 > >> > > >> > Op di 18 mei 2021 om 14:57 schreef Guillaume Nodet <gno...@apache.org > >: > >> > > >> > > I'm looking a bit at aggregator goals. > >> > > At first glance, it seems most of the problem comes from the fact > that > >> > the > >> > > build ordering is done mostly per-project. This causes obvious > >> problems > >> > > for aggregators when a mojo needs a given phase to be executed for > all > >> > > children before it can run (for example the compilation phase before > >> > > creating the javadoc). Another problem comes from the clean task > >> which > >> > has > >> > > obvious side effects. Calling clean with an aggregator goal > involved > >> in > >> > > the build seems like a recipe for problems... > >> > > > >> > > Would you see bad side effects to split the build in smaller chunks > >> > > (basically down to a mojo execution) so that ordering can be changed > >> in a > >> > > more meaningful way ? For example when running an aggregate javadoc, > >> the > >> > > top level project would be built first until before the javadoc > >> aggregate > >> > > goal. The build would then go to all other projects (like the > forked > >> > > lifecycle) until all required goals have been run, then resume from > >> the > >> > > javadoc goal. > >> > > Note that with the current behavior, when you run "clean verify", > the > >> > > execution is the following > >> > > > >> > > [INFO] Scanning for projects... > >> > > [INFO] > >> > > > >> ------------------------------------------------------------------------ > >> > > [INFO] Reactor Build Order: > >> > > [INFO] > >> > > [INFO] forked > >> > > [pom] > >> > > [INFO] forked-mod1 > >> > > [jar] > >> > > [INFO] forked-mod2 > >> > > [jar] > >> > > [INFO] > >> > > [INFO] ---------------< org.mvndaemon.mvnd.test.forked:forked > >> > > >---------------- > >> > > [INFO] Building forked 0.0.1-SNAPSHOT > >> > > [1/3] > >> > > [INFO] --------------------------------[ pom > >> > > ]--------------------------------- > >> > > [INFO] > >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked --- > >> > > [INFO] >>> maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) > > > >> > > compile @ forked >>> > >> > > [INFO] > >> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >> > > [INFO] Forking forked-mod1 0.0.1-SNAPSHOT > >> > > [INFO] > >> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >> > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) > @ > >> > > forked-mod1 --- > >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > >> > > forked-mod1 --- > >> > > [INFO] > >> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >> > > [INFO] Forking forked-mod2 0.0.1-SNAPSHOT > >> > > [INFO] > >> > > > >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >> > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) > @ > >> > > forked-mod2 --- > >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > >> > > forked-mod2 --- > >> > > [INFO] <<< maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) > < > >> > > compile @ forked <<< > >> > > [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) > @ > >> > > forked --- > >> > > [INFO] > >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod1 > >> > > >------------- > >> > > [INFO] Building forked-mod1 0.0.1-SNAPSHOT > >> > > [2/3] > >> > > [INFO] --------------------------------[ jar > >> > > ]--------------------------------- > >> > > [INFO] > >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ > forked-mod1 > >> --- > >> > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) > @ > >> > > forked-mod1 --- > >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > >> > > forked-mod1 --- > >> > > [INFO] --- maven-resources-plugin:2.6:testResources > >> > (default-testResources) > >> > > @ forked-mod1 --- > >> > > [INFO] --- maven-compiler-plugin:3.8.0:testCompile > >> (default-testCompile) > >> > @ > >> > > forked-mod1 --- > >> > > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ > >> forked-mod1 > >> > > --- > >> > > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod1 --- > >> > > [INFO] > >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod2 > >> > > >------------- > >> > > [INFO] Building forked-mod2 0.0.1-SNAPSHOT > >> > > [3/3] > >> > > [INFO] --------------------------------[ jar > >> > > ]--------------------------------- > >> > > [INFO] > >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ > forked-mod2 > >> --- > >> > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) > @ > >> > > forked-mod2 --- > >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > >> > > forked-mod2 --- > >> > > [INFO] --- maven-resources-plugin:2.6:testResources > >> > (default-testResources) > >> > > @ forked-mod2 --- > >> > > [INFO] --- maven-compiler-plugin:3.8.0:testCompile > >> (default-testCompile) > >> > @ > >> > > forked-mod2 --- > >> > > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ > >> forked-mod2 > >> > > --- > >> > > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod2 --- > >> > > [INFO] > >> > > > >> ------------------------------------------------------------------------ > >> > > [INFO] Reactor Summary for forked 0.0.1-SNAPSHOT: > >> > > [INFO] > >> > > [INFO] forked ............................................. SUCCESS > [ > >> > > 1.921 s] > >> > > [INFO] forked-mod1 ........................................ SUCCESS > [ > >> > > 0.498 s] > >> > > [INFO] forked-mod2 ........................................ SUCCESS > [ > >> > > 0.046 s] > >> > > [INFO] > >> > > > >> ------------------------------------------------------------------------ > >> > > [INFO] BUILD SUCCESS > >> > > [INFO] > >> > > > >> ------------------------------------------------------------------------ > >> > > [INFO] Total time: 2.535 s > >> > > [INFO] Finished at: 2021-05-18T14:32:09+02:00 > >> > > [INFO] > >> > > > >> ------------------------------------------------------------------------ > >> > > > >> > > As you can see, the resources and compile goals are run twice for > >> each > >> > > project. Furthermore, given the clean goal is in the loop, the > >> compiler > >> > > actually compiles things twice. > >> > > > >> > > The build plan could be rewritten as > >> > > > >> > > [INFO] Scanning for projects... > >> > > [INFO] > >> > > > >> ------------------------------------------------------------------------ > >> > > [INFO] Reactor Build Order: > >> > > [INFO] > >> > > [INFO] forked > >> > > [pom] > >> > > [INFO] forked-mod1 > >> > > [jar] > >> > > [INFO] forked-mod2 > >> > > [jar] > >> > > [INFO] > >> > > [INFO] ---------------< org.mvndaemon.mvnd.test.forked:forked > >> > > >---------------- > >> > > [INFO] Building forked 0.0.1-SNAPSHOT > >> > > [1/3] > >> > > [INFO] --------------------------------[ pom > >> > > ]--------------------------------- > >> > > [INFO] > >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked --- > >> > > ... normal lifecycle interrupted because the next goal is the > >> aggregator > >> > > goal ... > >> > > [INFO] > >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod1 > >> > > >------------- > >> > > [INFO] Building forked-mod1 0.0.1-SNAPSHOT > >> > > [2/3] > >> > > [INFO] --------------------------------[ jar > >> > > ]--------------------------------- > >> > > [INFO] > >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ > forked-mod1 > >> --- > >> > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) > @ > >> > > forked-mod1 --- > >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > >> > > forked-mod1 --- > >> > > ... interrupting forked-mod1 build as it reached the required point > >> > > execution ... > >> > > [INFO] > >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod2 > >> > > >------------- > >> > > [INFO] Building forked-mod2 0.0.1-SNAPSHOT > >> > > [3/3] > >> > > [INFO] --------------------------------[ jar > >> > > ]--------------------------------- > >> > > [INFO] > >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ > forked-mod2 > >> --- > >> > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) > @ > >> > > forked-mod2 --- > >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > >> > > forked-mod2 --- > >> > > ... interrupting forked-mod2 build as it reached the required point > >> > > execution ... > >> > > ... resuming the top level project... > >> > > [INFO] > >> > > [INFO] ---------------< org.mvndaemon.mvnd.test.forked:forked > >> > > >---------------- > >> > > [INFO] Building forked 0.0.1-SNAPSHOT > >> > > [1/3] > >> > > [INFO] --------------------------------[ pom > >> > > ]--------------------------------- > >> > > [INFO] > >> > > [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) > @ > >> > > forked --- > >> > > ... then resuming the forked-mod1 project... > >> > > [INFO] > >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod1 > >> > > >------------- > >> > > [INFO] Building forked-mod1 0.0.1-SNAPSHOT > >> > > [2/3] > >> > > [INFO] --------------------------------[ jar > >> > > ]--------------------------------- > >> > > [INFO] > >> > > [INFO] --- maven-resources-plugin:2.6:testResources > >> > (default-testResources) > >> > > @ forked-mod1 --- > >> > > [INFO] --- maven-compiler-plugin:3.8.0:testCompile > >> (default-testCompile) > >> > @ > >> > > forked-mod1 --- > >> > > [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) > @ > >> > > forked --- > >> > > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ > >> forked-mod1 > >> > > --- > >> > > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod1 --- > >> > > ... and resuming the forked-mod2 project... > >> > > [INFO] > >> > > [INFO] -------------< org.mvndaemon.mvnd.test.forked:forked-mod2 > >> > > >------------- > >> > > [INFO] Building forked-mod2 0.0.1-SNAPSHOT > >> > > [3/3] > >> > > [INFO] --------------------------------[ jar > >> > > ]--------------------------------- > >> > > [INFO] > >> > > [INFO] --- maven-resources-plugin:2.6:testResources > >> > (default-testResources) > >> > > @ forked-mod2 --- > >> > > [INFO] --- maven-compiler-plugin:3.8.0:testCompile > >> (default-testCompile) > >> > @ > >> > > forked-mod2 --- > >> > > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ > >> forked-mod2 > >> > > --- > >> > > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod2 --- > >> > > [INFO] > >> > > > >> ------------------------------------------------------------------------ > >> > > [INFO] Reactor Summary for forked 0.0.1-SNAPSHOT: > >> > > [INFO] > >> > > [INFO] forked ............................................. SUCCESS > [ > >> > > 1.921 s] > >> > > [INFO] forked-mod1 ........................................ SUCCESS > [ > >> > > 0.498 s] > >> > > [INFO] forked-mod2 ........................................ SUCCESS > [ > >> > > 0.046 s] > >> > > [INFO] > >> > > > >> ------------------------------------------------------------------------ > >> > > [INFO] BUILD SUCCESS > >> > > [INFO] > >> > > > >> ------------------------------------------------------------------------ > >> > > [INFO] Total time: 2.535 s > >> > > [INFO] Finished at: 2021-05-18T14:32:09+02:00 > >> > > [INFO] > >> > > > >> ------------------------------------------------------------------------ > >> > > > >> > > This would also solve the parallel build problem because each > project > >> > would > >> > > not be able to be both "cleaned" and "compiled" concurrently, which > is > >> > > currently possible in the current state. > >> > > > >> > > > >> > > Le lun. 17 mai 2021 à 14:06, Guillaume Nodet <gno...@apache.org> a > >> > écrit : > >> > > > >> > > > I've raised https://issues.apache.org/jira/browse/MNG-7156 > >> > > > > >> > > > Le mer. 12 mai 2021 à 17:57, Falko Modler <f.mod...@gmx.net> a > >> écrit : > >> > > > > >> > > >> Hi Guillaume, > >> > > >> > >> > > >> aggregation goals and parallel builds in combination are a bit > of a > >> > > >> mess, e.g.: > >> > > >> > >> > > >> - https://issues.apache.org/jira/browse/MNG-6843 > >> > > >> - https://github.com/apache/maven/pull/413 > >> > > >> - > https://www.mail-archive.com/dev@maven.apache.org/msg123439.html > >> > > >> > >> > > >> Cheers, > >> > > >> > >> > > >> Falko > >> > > >> > >> > > >> Am 12.05.2021 um 17:25 schrieb Guillaume Nodet: > >> > > >> > Hi > >> > > >> > > >> > > >> > I've analyzed a bug reported on mvnd this afternoon ( > >> > > >> > https://github.com/mvndaemon/mvnd/issues/408). It appears > that > >> the > >> > > >> parent > >> > > >> > pom executes the javadoc aggregate goal, which forks the > >> lifecycle > >> > of > >> > > >> the > >> > > >> > children modules in order to compile the sources. In a > >> traditional > >> > > >> build, > >> > > >> > this does not cause any real problem, but in a parallel build, > a > >> > > clean > >> > > >> > verify can definitely cause issues if the forked lifecycle and > >> the > >> > > >> normal > >> > > >> > project build (and especially the clean) are run concurrently. > >> > > >> > This definitely looks like an issue to me. Any idea where I > >> should > >> > > >> look at > >> > > >> > how to solve the problem ? I wonder if the MojoExecutor should > >> > > somehow > >> > > >> > delegate to the Builder which is responsible for synchronizing > >> the > >> > > >> > executions in the case of a multithreaded build... > >> > > >> > Thoughts ? > >> > > >> > > >> > > >> > >> > > >> > >> > > >> > >> --------------------------------------------------------------------- > >> > > >> To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > >> > > >> For additional commands, e-mail: dev-h...@maven.apache.org > >> > > >> > >> > > >> > >> > > > > >> > > > -- > >> > > > ------------------------ > >> > > > Guillaume Nodet > >> > > > > >> > > > > >> > > > >> > > -- > >> > > ------------------------ > >> > > Guillaume Nodet > >> > > > >> > > >> > >> > >> -- > >> ------------------------ > >> Guillaume Nodet > >> > > > -- ------------------------ Guillaume Nodet
