Re: Mojo execution synchonization with forked lifecycles in parallel builds
Hi Guillaume, [3] is definitely different from your original problem, but it should explain (together with [2]) why [1] did not remove the second "compile" in your problem case. I wanted to understand why MNG-6566 didn't apply to it. I agree that if it would have applied (removing 2nd compile in "compile -> clean -> compile"), it would have failed the build even harder. Anyhow, having said that, I believe that your idea of reordering parts of the build would definitely be an improvement. I just think it is ambitious and might be complex to build. I'm still not fully experienced in those parts of Maven core though, so maybe I am incorrectly estimating this complexity. Looking forward to your locking mechanism! Martin Op di 1 jun. 2021 om 18:20 schreef Guillaume Nodet : > > Le sam. 29 mai 2021 à 10:13, Martin Kanters a > écrit : > >> So we took a look at this. First we wanted to know why the forked >> goals were executed multiple times, while MNG-6566 [1] should have taken >> care of that. But it seems that that only optimizes duplicate goals in the >> context of one project. I attempted to explain that [2]. We created a >> ticket for optimizing this for multi module projects [3], but we think that >> it's not straightforward to fix this. >> > > I still think [3] is slightly different from my original problem. You > seem to indicate that the forked lifecycle is executed after, while in my > case, they were executed at the very beginning of the build, so that the > forked goals were executed before the usual execution from their > respective project. The effect is that for a given project, the following > goals were executed: > compile -> clean -> compile > > The second problem is that the concurrent execution of the clean and > compile goals lead to unpredictable results. > > It seems the issue only stresses a re-execution of the goal as a possible > optimization... That may be just me not reading correctly though... > > >> Back to the problem at hand. IIUC your suggestion is to pause >> project builds halfway, resume them after the required child projects goals >> have been executed (instead of forking), and finally finish the child >> project builds afterwards. I think it's elegant that in this solution, no >> duplicate goals will be executed, since no real forking has to be done. I >> do have the feeling that it would become quite a drastic change. I am also >> not sure whether we can assume that switching halfway through project >> builds works fine in every scenario. For the user it would also become hard >> to follow the build, I guess... >> >> Perhaps it would be easier to ensure projects can only be built once at >> a time (e.g. using locks or something). It wouldn't be the most >> efficient, but I guess it does not involve big changes to both the inner >> workings and user perspective. > > > I could try the second solution, though this could still leave an > execution order: > compile -> clean -> compile > which I think is really wrong. > > I think the first solution brings a few benefits : during parallel builds, > more things could be executed in parallel. For example the ordering could > be improved at the mojo execution level (or at least per group of mojos > execution in the same phase) instead of a global per-project ordering. In > theory, we could even make the project resolution happen in parallel with > some goals execution: compile time dependencies could be resolved during > the clean / resource generation phases, and test dependencies during > compilation, etc... Also, dependencies between projects only really happen > for the compilation (or test) phases, so the clean / resource generation > phases can happen concurrently. > For the output, I'm not so worried. In the case of parallel builds, the > output has to be reordered anyway to be understandable, as done in mvnd. > > I'll have a look at a simple per-project locking mechanism first though... > > >> Martin >> >> [1] https://issues.apache.org/jira/browse/MNG-6566 >> [2] >> >> https://issues.apache.org/jira/browse/MNG-6566?focusedCommentId=17353384&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17353384 >> [3] https://issues.apache.org/jira/browse/MNG-7163 >> >> Op di 18 mei 2021 om 21:45 schreef Martin Kanters < >> martinkant...@apache.org >> >: >> >> > Ah, too bad. I thought it was relevant. Perhaps I can take a look at >> this >> > with Maarten Mulders, but that will not be in the coming week at least. >> > >> > Op di 18 mei 2021 om 21:34 schreef Guillaume Nodet : >> > >> >> I just tried with master and the problem is the same. >> >> The reason is that in my use case, the forked goals are executed before >> >> the >> >> standard executions, and not even in the same project. >> >> >> >> Le mar. 18 mai 2021 à 20:41, Martin Kanters >> a >> >> écrit : >> >> >> >> > Not sure how I've missed this post. Have you tried this build with >> the >> >> > master build of Maven? >> >> > MNG-6566
Re: Mojo execution synchonization with forked lifecycles in parallel builds
I implemented a simple locking mechanism which seems to work correctly for parallel builds: https://github.com/apache/maven/pull/476/ Le mar. 1 juin 2021 à 18:19, Guillaume Nodet a écrit : > > Le sam. 29 mai 2021 à 10:13, Martin Kanters a > écrit : > >> So we took a look at this. First we wanted to know why the forked >> goals were executed multiple times, while MNG-6566 [1] should have taken >> care of that. But it seems that that only optimizes duplicate goals in the >> context of one project. I attempted to explain that [2]. We created a >> ticket for optimizing this for multi module projects [3], but we think that >> it's not straightforward to fix this. >> > > I still think [3] is slightly different from my original problem. You > seem to indicate that the forked lifecycle is executed after, while in my > case, they were executed at the very beginning of the build, so that the > forked goals were executed before the usual execution from their > respective project. The effect is that for a given project, the following > goals were executed: > compile -> clean -> compile > > The second problem is that the concurrent execution of the clean and > compile goals lead to unpredictable results. > > It seems the issue only stresses a re-execution of the goal as a possible > optimization... That may be just me not reading correctly though... > > >> Back to the problem at hand. IIUC your suggestion is to pause >> project builds halfway, resume them after the required child projects goals >> have been executed (instead of forking), and finally finish the child >> project builds afterwards. I think it's elegant that in this solution, no >> duplicate goals will be executed, since no real forking has to be done. I >> do have the feeling that it would become quite a drastic change. I am also >> not sure whether we can assume that switching halfway through project >> builds works fine in every scenario. For the user it would also become hard >> to follow the build, I guess... >> >> Perhaps it would be easier to ensure projects can only be built once at >> a time (e.g. using locks or something). It wouldn't be the most >> efficient, but I guess it does not involve big changes to both the inner >> workings and user perspective. > > > I could try the second solution, though this could still leave an > execution order: > compile -> clean -> compile > which I think is really wrong. > > I think the first solution brings a few benefits : during parallel builds, > more things could be executed in parallel. For example the ordering could > be improved at the mojo execution level (or at least per group of mojos > execution in the same phase) instead of a global per-project ordering. In > theory, we could even make the project resolution happen in parallel with > some goals execution: compile time dependencies could be resolved during > the clean / resource generation phases, and test dependencies during > compilation, etc... Also, dependencies between projects only really happen > for the compilation (or test) phases, so the clean / resource generation > phases can happen concurrently. > For the output, I'm not so worried. In the case of parallel builds, the > output has to be reordered anyway to be understandable, as done in mvnd. > > I'll have a look at a simple per-project locking mechanism first though... > > >> Martin >> >> [1] https://issues.apache.org/jira/browse/MNG-6566 >> [2] >> >> https://issues.apache.org/jira/browse/MNG-6566?focusedCommentId=17353384&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17353384 >> [3] https://issues.apache.org/jira/browse/MNG-7163 >> >> Op di 18 mei 2021 om 21:45 schreef Martin Kanters < >> martinkant...@apache.org >> >: >> >> > Ah, too bad. I thought it was relevant. Perhaps I can take a look at >> this >> > with Maarten Mulders, but that will not be in the coming week at least. >> > >> > Op di 18 mei 2021 om 21:34 schreef Guillaume Nodet : >> > >> >> I just tried with master and the problem is the same. >> >> The reason is that in my use case, the forked goals are executed before >> >> the >> >> standard executions, and not even in the same project. >> >> >> >> Le mar. 18 mai 2021 à 20:41, Martin Kanters >> a >> >> écrit : >> >> >> >> > Not sure how I've missed this post. Have you tried this build with >> the >> >> > master build of Maven? >> >> > MNG-6566 [1] should prevent any unnecessary double executions, thus >> >> > optimizing the buildplan. >> >> > I'm interested if this will solve the problem at hand. >> >> > >> >> > Martin >> >> > >> >> > [1] https://issues.apache.org/jira/browse/MNG-6566 >> >> > >> >> > Op di 18 mei 2021 om 14:57 schreef Guillaume Nodet < >> gno...@apache.org>: >> >> > >> >> > > I'm looking a bit at aggregator goals. >> >> > > At first glance, it seems most of the problem comes from the fact >> that >> >> > the >> >> > > build ordering is done mostly per-project. This causes obvious >> >> problems >> >> > > f
Re: Mojo execution synchonization with forked lifecycles in parallel builds
Le sam. 29 mai 2021 à 10:13, Martin Kanters a écrit : > So we took a look at this. First we wanted to know why the forked > goals were executed multiple times, while MNG-6566 [1] should have taken > care of that. But it seems that that only optimizes duplicate goals in the > context of one project. I attempted to explain that [2]. We created a > ticket for optimizing this for multi module projects [3], but we think that > it's not straightforward to fix this. > I still think [3] is slightly different from my original problem. You seem to indicate that the forked lifecycle is executed after, while in my case, they were executed at the very beginning of the build, so that the forked goals were executed before the usual execution from their respective project. The effect is that for a given project, the following goals were executed: compile -> clean -> compile The second problem is that the concurrent execution of the clean and compile goals lead to unpredictable results. It seems the issue only stresses a re-execution of the goal as a possible optimization... That may be just me not reading correctly though... > Back to the problem at hand. IIUC your suggestion is to pause > project builds halfway, resume them after the required child projects goals > have been executed (instead of forking), and finally finish the child > project builds afterwards. I think it's elegant that in this solution, no > duplicate goals will be executed, since no real forking has to be done. I > do have the feeling that it would become quite a drastic change. I am also > not sure whether we can assume that switching halfway through project > builds works fine in every scenario. For the user it would also become hard > to follow the build, I guess... > > Perhaps it would be easier to ensure projects can only be built once at > a time (e.g. using locks or something). It wouldn't be the most > efficient, but I guess it does not involve big changes to both the inner > workings and user perspective. I could try the second solution, though this could still leave an execution order: compile -> clean -> compile which I think is really wrong. I think the first solution brings a few benefits : during parallel builds, more things could be executed in parallel. For example the ordering could be improved at the mojo execution level (or at least per group of mojos execution in the same phase) instead of a global per-project ordering. In theory, we could even make the project resolution happen in parallel with some goals execution: compile time dependencies could be resolved during the clean / resource generation phases, and test dependencies during compilation, etc... Also, dependencies between projects only really happen for the compilation (or test) phases, so the clean / resource generation phases can happen concurrently. For the output, I'm not so worried. In the case of parallel builds, the output has to be reordered anyway to be understandable, as done in mvnd. I'll have a look at a simple per-project locking mechanism first though... > Martin > > [1] https://issues.apache.org/jira/browse/MNG-6566 > [2] > > https://issues.apache.org/jira/browse/MNG-6566?focusedCommentId=17353384&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17353384 > [3] https://issues.apache.org/jira/browse/MNG-7163 > > Op di 18 mei 2021 om 21:45 schreef Martin Kanters < > martinkant...@apache.org > >: > > > Ah, too bad. I thought it was relevant. Perhaps I can take a look at this > > with Maarten Mulders, but that will not be in the coming week at least. > > > > Op di 18 mei 2021 om 21:34 schreef Guillaume Nodet : > > > >> I just tried with master and the problem is the same. > >> The reason is that in my use case, the forked goals are executed before > >> the > >> standard executions, and not even in the same project. > >> > >> Le mar. 18 mai 2021 à 20:41, Martin Kanters > a > >> écrit : > >> > >> > Not sure how I've missed this post. Have you tried this build with the > >> > master build of Maven? > >> > MNG-6566 [1] should prevent any unnecessary double executions, thus > >> > optimizing the buildplan. > >> > I'm interested if this will solve the problem at hand. > >> > > >> > Martin > >> > > >> > [1] https://issues.apache.org/jira/browse/MNG-6566 > >> > > >> > Op di 18 mei 2021 om 14:57 schreef Guillaume Nodet >: > >> > > >> > > I'm looking a bit at aggregator goals. > >> > > At first glance, it seems most of the problem comes from the fact > that > >> > the > >> > > build ordering is done mostly per-project. This causes obvious > >> problems > >> > > for aggregators when a mojo needs a given phase to be executed for > all > >> > > children before it can run (for example the compilation phase before > >> > > creating the javadoc). Another problem comes from the clean task > >> which > >> > has > >> > > obvious side effects. Calling clean with an aggregator goal > involved > >> in > >> > > the build seems
Re: Mojo execution synchonization with forked lifecycles in parallel builds
So we took a look at this. First we wanted to know why the forked goals were executed multiple times, while MNG-6566 [1] should have taken care of that. But it seems that that only optimizes duplicate goals in the context of one project. I attempted to explain that [2]. We created a ticket for optimizing this for multi module projects [3], but we think that it's not straightforward to fix this. Back to the problem at hand. IIUC your suggestion is to pause project builds halfway, resume them after the required child projects goals have been executed (instead of forking), and finally finish the child project builds afterwards. I think it's elegant that in this solution, no duplicate goals will be executed, since no real forking has to be done. I do have the feeling that it would become quite a drastic change. I am also not sure whether we can assume that switching halfway through project builds works fine in every scenario. For the user it would also become hard to follow the build, I guess... Perhaps it would be easier to ensure projects can only be built once at a time (e.g. using locks or something). It wouldn't be the most efficient, but I guess it does not involve big changes to both the inner workings and user perspective. Martin [1] https://issues.apache.org/jira/browse/MNG-6566 [2] https://issues.apache.org/jira/browse/MNG-6566?focusedCommentId=17353384&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17353384 [3] https://issues.apache.org/jira/browse/MNG-7163 Op di 18 mei 2021 om 21:45 schreef Martin Kanters : > Ah, too bad. I thought it was relevant. Perhaps I can take a look at this > with Maarten Mulders, but that will not be in the coming week at least. > > Op di 18 mei 2021 om 21:34 schreef Guillaume Nodet : > >> I just tried with master and the problem is the same. >> The reason is that in my use case, the forked goals are executed before >> the >> standard executions, and not even in the same project. >> >> Le mar. 18 mai 2021 à 20:41, Martin Kanters a >> écrit : >> >> > Not sure how I've missed this post. Have you tried this build with the >> > master build of Maven? >> > MNG-6566 [1] should prevent any unnecessary double executions, thus >> > optimizing the buildplan. >> > I'm interested if this will solve the problem at hand. >> > >> > Martin >> > >> > [1] https://issues.apache.org/jira/browse/MNG-6566 >> > >> > Op di 18 mei 2021 om 14:57 schreef Guillaume Nodet : >> > >> > > I'm looking a bit at aggregator goals. >> > > At first glance, it seems most of the problem comes from the fact that >> > the >> > > build ordering is done mostly per-project. This causes obvious >> problems >> > > for aggregators when a mojo needs a given phase to be executed for all >> > > children before it can run (for example the compilation phase before >> > > creating the javadoc). Another problem comes from the clean task >> which >> > has >> > > obvious side effects. Calling clean with an aggregator goal involved >> in >> > > the build seems like a recipe for problems... >> > > >> > > Would you see bad side effects to split the build in smaller chunks >> > > (basically down to a mojo execution) so that ordering can be changed >> in a >> > > more meaningful way ? For example when running an aggregate javadoc, >> the >> > > top level project would be built first until before the javadoc >> aggregate >> > > goal. The build would then go to all other projects (like the forked >> > > lifecycle) until all required goals have been run, then resume from >> the >> > > javadoc goal. >> > > Note that with the current behavior, when you run "clean verify", the >> > > execution is the following >> > > >> > > [INFO] Scanning for projects... >> > > [INFO] >> > > >> >> > > [INFO] Reactor Build Order: >> > > [INFO] >> > > [INFO] forked >> > > [pom] >> > > [INFO] forked-mod1 >> > > [jar] >> > > [INFO] forked-mod2 >> > > [jar] >> > > [INFO] >> > > [INFO] ---< org.mvndaemon.mvnd.test.forked:forked >> > > > >> > > [INFO] Building forked 0.0.1-SNAPSHOT >> > > [1/3] >> > > [INFO] [ pom >> > > ]- >> > > [INFO] >> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked --- >> > > [INFO] >>> maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) > >> > > compile @ forked >>> >> > > [INFO] >> > > >> >> > > [INFO] Forking forked-mod1 0.0.1-SNAPSHOT >> > > [INFO] >> > > >> >> > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ >> > > forked-mod1 --- >> > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ >> > > forked-mod1 --- >> > > [INFO] >> > > >> >> > > [INFO] F
Re: Mojo execution synchonization with forked lifecycles in parallel builds
Ah, too bad. I thought it was relevant. Perhaps I can take a look at this with Maarten Mulders, but that will not be in the coming week at least. Op di 18 mei 2021 om 21:34 schreef Guillaume Nodet : > I just tried with master and the problem is the same. > The reason is that in my use case, the forked goals are executed before the > standard executions, and not even in the same project. > > Le mar. 18 mai 2021 à 20:41, Martin Kanters a > écrit : > > > Not sure how I've missed this post. Have you tried this build with the > > master build of Maven? > > MNG-6566 [1] should prevent any unnecessary double executions, thus > > optimizing the buildplan. > > I'm interested if this will solve the problem at hand. > > > > Martin > > > > [1] https://issues.apache.org/jira/browse/MNG-6566 > > > > Op di 18 mei 2021 om 14:57 schreef Guillaume Nodet : > > > > > I'm looking a bit at aggregator goals. > > > At first glance, it seems most of the problem comes from the fact that > > the > > > build ordering is done mostly per-project. This causes obvious > problems > > > for aggregators when a mojo needs a given phase to be executed for all > > > children before it can run (for example the compilation phase before > > > creating the javadoc). Another problem comes from the clean task which > > has > > > obvious side effects. Calling clean with an aggregator goal involved > in > > > the build seems like a recipe for problems... > > > > > > Would you see bad side effects to split the build in smaller chunks > > > (basically down to a mojo execution) so that ordering can be changed > in a > > > more meaningful way ? For example when running an aggregate javadoc, > the > > > top level project would be built first until before the javadoc > aggregate > > > goal. The build would then go to all other projects (like the forked > > > lifecycle) until all required goals have been run, then resume from the > > > javadoc goal. > > > Note that with the current behavior, when you run "clean verify", the > > > execution is the following > > > > > > [INFO] Scanning for projects... > > > [INFO] > > > > > > > [INFO] Reactor Build Order: > > > [INFO] > > > [INFO] forked > > > [pom] > > > [INFO] forked-mod1 > > > [jar] > > > [INFO] forked-mod2 > > > [jar] > > > [INFO] > > > [INFO] ---< org.mvndaemon.mvnd.test.forked:forked > > > > > > > [INFO] Building forked 0.0.1-SNAPSHOT > > > [1/3] > > > [INFO] [ pom > > > ]- > > > [INFO] > > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked --- > > > [INFO] >>> maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) > > > > compile @ forked >>> > > > [INFO] > > > > > > > [INFO] Forking forked-mod1 0.0.1-SNAPSHOT > > > [INFO] > > > > > > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > > > forked-mod1 --- > > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > > > forked-mod1 --- > > > [INFO] > > > > > > > [INFO] Forking forked-mod2 0.0.1-SNAPSHOT > > > [INFO] > > > > > > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > > > forked-mod2 --- > > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > > > forked-mod2 --- > > > [INFO] <<< maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) < > > > compile @ forked <<< > > > [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) @ > > > forked --- > > > [INFO] > > > [INFO] -< org.mvndaemon.mvnd.test.forked:forked-mod1 > > > >- > > > [INFO] Building forked-mod1 0.0.1-SNAPSHOT > > > [2/3] > > > [INFO] [ jar > > > ]- > > > [INFO] > > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod1 > --- > > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > > > forked-mod1 --- > > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > > > forked-mod1 --- > > > [INFO] --- maven-resources-plugin:2.6:testResources > > (default-testResources) > > > @ forked-mod1 --- > > > [INFO] --- maven-compiler-plugin:3.8.0:testCompile > (default-testCompile) > > @ > > > forked-mod1 --- > > > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ > forked-mod1 > > > --- > > > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod1 --- > > > [INFO] > > > [INFO] -< org.mvndaemon.mvnd.test.forked:forked-mod2 > > > >- > > > [INFO] Building forked-mod2 0.0.1-SNAPSHOT > > > [3/3] > > > [INFO] [ jar > > > ]
Re: Mojo execution synchonization with forked lifecycles in parallel builds
I just tried with master and the problem is the same. The reason is that in my use case, the forked goals are executed before the standard executions, and not even in the same project. Le mar. 18 mai 2021 à 20:41, Martin Kanters a écrit : > Not sure how I've missed this post. Have you tried this build with the > master build of Maven? > MNG-6566 [1] should prevent any unnecessary double executions, thus > optimizing the buildplan. > I'm interested if this will solve the problem at hand. > > Martin > > [1] https://issues.apache.org/jira/browse/MNG-6566 > > Op di 18 mei 2021 om 14:57 schreef Guillaume Nodet : > > > I'm looking a bit at aggregator goals. > > At first glance, it seems most of the problem comes from the fact that > the > > build ordering is done mostly per-project. This causes obvious problems > > for aggregators when a mojo needs a given phase to be executed for all > > children before it can run (for example the compilation phase before > > creating the javadoc). Another problem comes from the clean task which > has > > obvious side effects. Calling clean with an aggregator goal involved in > > the build seems like a recipe for problems... > > > > Would you see bad side effects to split the build in smaller chunks > > (basically down to a mojo execution) so that ordering can be changed in a > > more meaningful way ? For example when running an aggregate javadoc, the > > top level project would be built first until before the javadoc aggregate > > goal. The build would then go to all other projects (like the forked > > lifecycle) until all required goals have been run, then resume from the > > javadoc goal. > > Note that with the current behavior, when you run "clean verify", the > > execution is the following > > > > [INFO] Scanning for projects... > > [INFO] > > > > [INFO] Reactor Build Order: > > [INFO] > > [INFO] forked > > [pom] > > [INFO] forked-mod1 > > [jar] > > [INFO] forked-mod2 > > [jar] > > [INFO] > > [INFO] ---< org.mvndaemon.mvnd.test.forked:forked > > > > > [INFO] Building forked 0.0.1-SNAPSHOT > > [1/3] > > [INFO] [ pom > > ]- > > [INFO] > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked --- > > [INFO] >>> maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) > > > compile @ forked >>> > > [INFO] > > > > [INFO] Forking forked-mod1 0.0.1-SNAPSHOT > > [INFO] > > > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > > forked-mod1 --- > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > > forked-mod1 --- > > [INFO] > > > > [INFO] Forking forked-mod2 0.0.1-SNAPSHOT > > [INFO] > > > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > > forked-mod2 --- > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > > forked-mod2 --- > > [INFO] <<< maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) < > > compile @ forked <<< > > [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) @ > > forked --- > > [INFO] > > [INFO] -< org.mvndaemon.mvnd.test.forked:forked-mod1 > > >- > > [INFO] Building forked-mod1 0.0.1-SNAPSHOT > > [2/3] > > [INFO] [ jar > > ]- > > [INFO] > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod1 --- > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > > forked-mod1 --- > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > > forked-mod1 --- > > [INFO] --- maven-resources-plugin:2.6:testResources > (default-testResources) > > @ forked-mod1 --- > > [INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) > @ > > forked-mod1 --- > > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ forked-mod1 > > --- > > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod1 --- > > [INFO] > > [INFO] -< org.mvndaemon.mvnd.test.forked:forked-mod2 > > >- > > [INFO] Building forked-mod2 0.0.1-SNAPSHOT > > [3/3] > > [INFO] [ jar > > ]- > > [INFO] > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod2 --- > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > > forked-mod2 --- > > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > > forked-mod2 --- > > [INFO] --- maven-resources-plugin:2.6:testResources > (default-testResources) > > @ forked-mod2 --- > > [INFO] --- maven-compiler-plugin:3.8.0:
Re: Mojo execution synchonization with forked lifecycles in parallel builds
Not sure how I've missed this post. Have you tried this build with the master build of Maven? MNG-6566 [1] should prevent any unnecessary double executions, thus optimizing the buildplan. I'm interested if this will solve the problem at hand. Martin [1] https://issues.apache.org/jira/browse/MNG-6566 Op di 18 mei 2021 om 14:57 schreef Guillaume Nodet : > I'm looking a bit at aggregator goals. > At first glance, it seems most of the problem comes from the fact that the > build ordering is done mostly per-project. This causes obvious problems > for aggregators when a mojo needs a given phase to be executed for all > children before it can run (for example the compilation phase before > creating the javadoc). Another problem comes from the clean task which has > obvious side effects. Calling clean with an aggregator goal involved in > the build seems like a recipe for problems... > > Would you see bad side effects to split the build in smaller chunks > (basically down to a mojo execution) so that ordering can be changed in a > more meaningful way ? For example when running an aggregate javadoc, the > top level project would be built first until before the javadoc aggregate > goal. The build would then go to all other projects (like the forked > lifecycle) until all required goals have been run, then resume from the > javadoc goal. > Note that with the current behavior, when you run "clean verify", the > execution is the following > > [INFO] Scanning for projects... > [INFO] > > [INFO] Reactor Build Order: > [INFO] > [INFO] forked > [pom] > [INFO] forked-mod1 > [jar] > [INFO] forked-mod2 > [jar] > [INFO] > [INFO] ---< org.mvndaemon.mvnd.test.forked:forked > > > [INFO] Building forked 0.0.1-SNAPSHOT > [1/3] > [INFO] [ pom > ]- > [INFO] > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked --- > [INFO] >>> maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) > > compile @ forked >>> > [INFO] > > [INFO] Forking forked-mod1 0.0.1-SNAPSHOT > [INFO] > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > forked-mod1 --- > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > forked-mod1 --- > [INFO] > > [INFO] Forking forked-mod2 0.0.1-SNAPSHOT > [INFO] > > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > forked-mod2 --- > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > forked-mod2 --- > [INFO] <<< maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) < > compile @ forked <<< > [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) @ > forked --- > [INFO] > [INFO] -< org.mvndaemon.mvnd.test.forked:forked-mod1 > >- > [INFO] Building forked-mod1 0.0.1-SNAPSHOT > [2/3] > [INFO] [ jar > ]- > [INFO] > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod1 --- > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > forked-mod1 --- > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > forked-mod1 --- > [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) > @ forked-mod1 --- > [INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ > forked-mod1 --- > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ forked-mod1 > --- > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod1 --- > [INFO] > [INFO] -< org.mvndaemon.mvnd.test.forked:forked-mod2 > >- > [INFO] Building forked-mod2 0.0.1-SNAPSHOT > [3/3] > [INFO] [ jar > ]- > [INFO] > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod2 --- > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > forked-mod2 --- > [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ > forked-mod2 --- > [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) > @ forked-mod2 --- > [INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ > forked-mod2 --- > [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ forked-mod2 > --- > [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod2 --- > [INFO] > > [INFO] Reactor Summary for forked 0.0.1-SNAPSHOT: > [INFO] > [INFO] forked . SUCCESS [ > 1.921 s] > [INFO] forked-mod1 ..
Re: Mojo execution synchonization with forked lifecycles in parallel builds
I'm looking a bit at aggregator goals. At first glance, it seems most of the problem comes from the fact that the build ordering is done mostly per-project. This causes obvious problems for aggregators when a mojo needs a given phase to be executed for all children before it can run (for example the compilation phase before creating the javadoc). Another problem comes from the clean task which has obvious side effects. Calling clean with an aggregator goal involved in the build seems like a recipe for problems... Would you see bad side effects to split the build in smaller chunks (basically down to a mojo execution) so that ordering can be changed in a more meaningful way ? For example when running an aggregate javadoc, the top level project would be built first until before the javadoc aggregate goal. The build would then go to all other projects (like the forked lifecycle) until all required goals have been run, then resume from the javadoc goal. Note that with the current behavior, when you run "clean verify", the execution is the following [INFO] Scanning for projects... [INFO] [INFO] Reactor Build Order: [INFO] [INFO] forked [pom] [INFO] forked-mod1 [jar] [INFO] forked-mod2 [jar] [INFO] [INFO] ---< org.mvndaemon.mvnd.test.forked:forked > [INFO] Building forked 0.0.1-SNAPSHOT [1/3] [INFO] [ pom ]- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked --- [INFO] >>> maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) > compile @ forked >>> [INFO] [INFO] Forking forked-mod1 0.0.1-SNAPSHOT [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ forked-mod1 --- [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ forked-mod1 --- [INFO] [INFO] Forking forked-mod2 0.0.1-SNAPSHOT [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ forked-mod2 --- [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ forked-mod2 --- [INFO] <<< maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) < compile @ forked <<< [INFO] --- maven-javadoc-plugin:3.2.0:aggregate-jar (aggregate-jar) @ forked --- [INFO] [INFO] -< org.mvndaemon.mvnd.test.forked:forked-mod1 >- [INFO] Building forked-mod1 0.0.1-SNAPSHOT [2/3] [INFO] [ jar ]- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod1 --- [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ forked-mod1 --- [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ forked-mod1 --- [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ forked-mod1 --- [INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ forked-mod1 --- [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ forked-mod1 --- [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod1 --- [INFO] [INFO] -< org.mvndaemon.mvnd.test.forked:forked-mod2 >- [INFO] Building forked-mod2 0.0.1-SNAPSHOT [3/3] [INFO] [ jar ]- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ forked-mod2 --- [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ forked-mod2 --- [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ forked-mod2 --- [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ forked-mod2 --- [INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ forked-mod2 --- [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ forked-mod2 --- [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ forked-mod2 --- [INFO] [INFO] Reactor Summary for forked 0.0.1-SNAPSHOT: [INFO] [INFO] forked . SUCCESS [ 1.921 s] [INFO] forked-mod1 SUCCESS [ 0.498 s] [INFO] forked-mod2 SUCCESS [ 0.046 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 2.535 s [INFO] Finished at: 2021-05-18T14:32:09+02:00 [INFO] As you can see, the resources and compile goals are run twice for each project. Furthermore, given the clean g
Re: Mojo execution synchonization with forked lifecycles in parallel builds
I've raised https://issues.apache.org/jira/browse/MNG-7156 Le mer. 12 mai 2021 à 17:57, Falko Modler a écrit : > Hi Guillaume, > > aggregation goals and parallel builds in combination are a bit of a > mess, e.g.: > > - https://issues.apache.org/jira/browse/MNG-6843 > - https://github.com/apache/maven/pull/413 > - https://www.mail-archive.com/dev@maven.apache.org/msg123439.html > > Cheers, > > Falko > > Am 12.05.2021 um 17:25 schrieb Guillaume Nodet: > > Hi > > > > I've analyzed a bug reported on mvnd this afternoon ( > > https://github.com/mvndaemon/mvnd/issues/408). It appears that the > parent > > pom executes the javadoc aggregate goal, which forks the lifecycle of the > > children modules in order to compile the sources. In a traditional > build, > > this does not cause any real problem, but in a parallel build, a clean > > verify can definitely cause issues if the forked lifecycle and the normal > > project build (and especially the clean) are run concurrently. > > This definitely looks like an issue to me. Any idea where I should look > at > > how to solve the problem ? I wonder if the MojoExecutor should somehow > > delegate to the Builder which is responsible for synchronizing the > > executions in the case of a multithreaded build... > > Thoughts ? > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > > -- Guillaume Nodet
Re: Mojo execution synchonization with forked lifecycles in parallel builds
Hi Guillaume, aggregation goals and parallel builds in combination are a bit of a mess, e.g.: - https://issues.apache.org/jira/browse/MNG-6843 - https://github.com/apache/maven/pull/413 - https://www.mail-archive.com/dev@maven.apache.org/msg123439.html Cheers, Falko Am 12.05.2021 um 17:25 schrieb Guillaume Nodet: Hi I've analyzed a bug reported on mvnd this afternoon ( https://github.com/mvndaemon/mvnd/issues/408). It appears that the parent pom executes the javadoc aggregate goal, which forks the lifecycle of the children modules in order to compile the sources. In a traditional build, this does not cause any real problem, but in a parallel build, a clean verify can definitely cause issues if the forked lifecycle and the normal project build (and especially the clean) are run concurrently. This definitely looks like an issue to me. Any idea where I should look at how to solve the problem ? I wonder if the MojoExecutor should somehow delegate to the Builder which is responsible for synchronizing the executions in the case of a multithreaded build... Thoughts ? - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Mojo execution synchonization with forked lifecycles in parallel builds
Hi I've analyzed a bug reported on mvnd this afternoon ( https://github.com/mvndaemon/mvnd/issues/408). It appears that the parent pom executes the javadoc aggregate goal, which forks the lifecycle of the children modules in order to compile the sources. In a traditional build, this does not cause any real problem, but in a parallel build, a clean verify can definitely cause issues if the forked lifecycle and the normal project build (and especially the clean) are run concurrently. This definitely looks like an issue to me. Any idea where I should look at how to solve the problem ? I wonder if the MojoExecutor should somehow delegate to the Builder which is responsible for synchronizing the executions in the case of a multithreaded build... Thoughts ? -- Guillaume Nodet
