RE: Re: Re: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hi Tamás, Is there an Maven issue out there to track this issue (and your fix for it)? I haven’t been able to get access to the Maven Jira project so have not been able to raise the issue myself. Thanks, Joe On 2024/02/08 10:09:31 Tamás Cservenák wrote: > Seems we are on track with this. To prove my last-night theory I created a > "hack" (is really just that) and guess what? > It makes reproducer behave "as expected": > https://github.com/apache/maven/pull/1406 > > T > > On Wed, Feb 7, 2024 at 10:05 PM Tamás Cservenák wrote: > > > Howdy, > > > > Thank you very much, the reproducer works. Did not dig thru it fully, but > > here are some related issues: > > > > https://issues.apache.org/jira/browse/MNG-8028 (funny thing, I created > > this few weeks ago) > > https://issues.apache.org/jira/browse/MNG-6300 > > > > Will report back tomorrow (EU TZ) > > T > > > > > > On Wed, Feb 7, 2024 at 7:48 PM Joseph Leonard < > > joseph.leon...@alfasystems.com> wrote: > > > >> Hi Tamás, > >> I have created a simple example here: > >> https://github.com/josple/mvn-multibuild-issue > >> Hopefully the README is clear enough – let me know if I can clarify > >> anything. > >> Thanks, > >> Joe > >> > >> On 2024/02/07 17:33:08 Tamás Cservenák wrote: > >> > Howdy, > >> > > >> > In that case, there is something fishy with the project, my blind guess > >> > would be some "hidden" inter-module dependency maybe? > >> > > >> > Can you provide access to source, or, if not feasible, could you provide > >> > some reproducer and publish it on Github/Gitlab/whatever (maybe even > >> just > >> > send it privately as ML strips off attachments and images) for us to see > >> > this in action? > >> > > >> > Thanks > >> > T > >> > > >> > On Wed, Feb 7, 2024 at 6:29 PM Joseph Leonard < > >> > joseph.leon...@alfasystems.com> wrote: > >> > > >> > > Hi Tamás, > >> > > We have previously played around a bit with mvnd but not takari > >> directly – > >> > > I will have a play with it. With regards to this issue, using the > >> takari > >> > > smart builder unfortunately doesn’t resolve the issue. > >> > > Joe > >> > > > >> > > On 2024/02/07 11:41:22 Tamás Cservenák wrote: > >> > > > Can you please try smart builder instead? > >> > > > https://github.com/takari/takari-smart-builder > >> > > > > >> > > > (note: smart builder is used by mvnd as well) > >> > > > > >> > > > The difference between the two can be seen here: > >> > > > http://takari.io/book/30-team-maven.html#takari-smart-builder > >> > > > > >> > > > On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < > >> > > > joseph.leon...@alfasystems.com> wrote: > >> > > > > >> > > > > Hi Tamás, > >> > > > > Yeah, this was unexpected to me initially as well. From what I > >> can tell > >> > > > > the Maven reactor only considers direct dependencies (i.e. not > >> > > transitive > >> > > > > dependencies) between the modules in the reactor when working out > >> the > >> > > build > >> > > > > graph. For example if you have a simple linear dependency chain > >> of: > >> > > > > One --> Two --> Three --> Four --> Five > >> > > > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will > >> result > >> > > in > >> > > > > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). > >> I > >> > > assume > >> > > > > this is as designed because it actually offers quite powerful > >> > > functionality > >> > > > > to improve the parallelism in your build. An example of where > >> this is > >> > > legit > >> > > > > is when: > >> > > > > > >> > > > > * “Four” has a test scope dependency on “Five” > >> > > > > * “One” has a test scoped dependency on “Two” > >> > > > > If you made a src code change to “Five” and “Two” then it would be > >> > > safe to > >> > > > > build [One,Two] and [Four,Five] in parallel because you know the > >> > > changes > >> > > > > within these graphs cannot impact each other. > >> > > > > Joe > >> > > > > > >> > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > >> > > > > > Howdy, > >> > > > > > > >> > > > > > To me this looks like Maven is not aware that the App depends on > >> > > > > ModuleB... > >> > > > > > Are they "plain dependency" linked? Or what kind of dependency > >> we > >> > > talk > >> > > > > > about here? > >> > > > > > In short: why would App start while ModuleB (upstream dep) is > >> not > >> > > done? > >> > > > > > Something is fishy here. > >> > > > > > > >> > > > > > T > >> > > > > > > >> > > > > > > >> > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > >> > > > > > joseph.leon...@alfasystems.com> wrote: > >> > > > > > > >> > > > > > > Hi all, > >> > > > > > > > >> > > > > > > It would be great to get any thoughts on whether the > >> following is a > >> > > > > defect: > >> > > > > > > > >> > > > > > > > >> > > > > > > Issue details: > >> > > > > > > tl;dr > >> > > > > > > > >> > > > > > > Maven can resolve dependencies either from: > >> > > > > > > > >> > > > > > > * an external repo > >> > > > > > > * a class
RE: Re: Re: Re: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hey, Yeah, exactly: ideally we want module-b to be resolved in different ways within the same reactor. I noticed that this kind of already happens (albeit in a simpler way): In my reproducer testing if you play with the test timeouts to dictate the module build order then with the same modules in the reactor you can get different resolution behaviour: * If "app" builds before "module-b" has started compiling then "app" will resolve "module-b" from the local repo * If "app" builds after "module-b" has compiled but before "module-b" has packaged then "app" will resolve the "module-b" class directory (this is the out-of-the-box scenario the reproducer produces) * If "app" builds after "module-b" has packaged then "app" will resolve the "module-b" jar from the "module-b" target directory So within this logic to perform different types of resolves, it would be neat if the reactor could perform an initial check first: * if the dependency being resolved is within the same 'sub-build graph' of the module requesting the resolve then look within the reactor (current behaviour) * otherwise, only resolve from the local repository I don't know whether the code architecture means this is quite easy to do or very difficult. But to be fair, I think this extra parallelism opportunity is a bit of niche edge case and it probably isn't worth it if this is very complex to achieve (or considered a smell generally by the community!) On 2024/02/08 Tamás Cservenák wrote > Howdy, > > Well, sadly, Maven currently has no means to make one module "this and that" > at the same time. > It is either in the reactor or is out of it. > To me, it sounds like you want one reactor, but where module-b is "out" and > 'in", at the same time (within same session) > > > T > > On Thu, Feb 8, 2024 at 1:27 PM Joseph Leonard < > joseph.leon...@alfasystems.com> wrote: > > > Hey, > > We don't need the module-b jar built in the reactor to be included in > > the war if the changes made in module-b won't affect the war. Instead, > > in this scenario, we're fine with using a jar resolved from the local > > repository. > > The requisites for this scenario are: > > > > * Any change made to module-b is only in its test tree > > * The only changes made to the direct and transitive dependencies of > > module-b are to its test scoped dependencies So basically module-b is > > only in the reactor for verification purposes rather than ‘src > > packaging’ purposes. > > Thanks, > > Joe > > > > On 2024/02/08 12:12:03 Tamás Cservenák wrote: > > > Hej, > > > > > > yes, basically the "hack" is well described by you: > > > - it retains "full' reactor project > > > - introduces "needs build" on project > > > - when it comes to building, it skips the project if flag set > > > > > > This results that module-b -> app link is not lost. > > > And as for your parallelization request: the truth is that there IS > > > and edge between app and module-b (via module-a), so despite > > > module-a is "done", its transitive module-b is not done, hence you > > > cannot build these two in parallel, as you do have an edge between app > > > and module-b. > > > > > > Or to invert the question: what module-b JAR you want to have > > > included in app WAR, when building? The one built, or the one from > > > the local repository? If the latter, why are you building it in the first > > > place? > > > > > > On Thu, Feb 8, 2024 at 12:06 PM Joseph Leonard < > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > Hi Tamás, > > > > Thank you for hacking around this so rapidly. I am not familiar > > > > with > > the > > > > maven code base, so let me know if I have misinterpreted the > > > > change: Is your hack to: > > > > > > > > 1. Retain the full multi-module build dependency graph such > > > > that the build graph always remains consistent regardless of whether a > > > > “-pl” > > arg has > > > > been provided > > > > 2. Introduce a skip build option to satisfy the functionality > > > > of the “-pl” arg: The reactor will report the module has been > > > > built but the > > build > > > > is basically a no-op > > > > If so, this does resolve the issue but at the cost of losing the > > > > opportunity to gain extra parallelism in the build. > > > > What I was wondering may be possible (without any idea of how > > > > feasible > > it > > > > would be to implement) would be the opportunity to recognise in > > > > the multi-build distinct build graphs and then keep isolation in > > > > the > > reactor > > > > between these graphs. To elaborate, in the reproducer scenario > > > > there > > would > > > > be two distinct graphs: > > > > > > > > * "testsupport-module-1" followed by "app" > > > > * "testsupport-module-2" followed by "module-b" > > > > Ideally these two graphs build in parallel to each other and even > > > > if ultimately there is a dependency between them they NEVER > > > > resolve that dependency from within reactor
Re: Re: Re: Re: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Howdy, Well, sadly, Maven currently has no means to make one module "this and that" at the same time. It is either in the reactor or is out of it. To me, it sounds like you want one reactor, but where module-b is "out" and 'in", at the same time (within same session) T On Thu, Feb 8, 2024 at 1:27 PM Joseph Leonard < joseph.leon...@alfasystems.com> wrote: > Hey, > We don't need the module-b jar built in the reactor to be included in the > war if the changes made in module-b won't affect the war. Instead, in this > scenario, we're fine with using a jar resolved from the local repository. > The requisites for this scenario are: > > * Any change made to module-b is only in its test tree > * The only changes made to the direct and transitive dependencies of > module-b are to its test scoped dependencies > So basically module-b is only in the reactor for verification purposes > rather than ‘src packaging’ purposes. > Thanks, > Joe > > On 2024/02/08 12:12:03 Tamás Cservenák wrote: > > Hej, > > > > yes, basically the "hack" is well described by you: > > - it retains "full' reactor project > > - introduces "needs build" on project > > - when it comes to building, it skips the project if flag set > > > > This results that module-b -> app link is not lost. > > And as for your parallelization request: the truth is that there IS and > > edge between app and module-b (via module-a), so despite module-a is > > "done", its transitive module-b is not done, hence you cannot build these > > two in parallel, as you do have an edge between app and module-b. > > > > Or to invert the question: what module-b JAR you want to have included in > > app WAR, when building? The one built, or the one from the local > > repository? If the latter, why are you building it in the first place? > > > > On Thu, Feb 8, 2024 at 12:06 PM Joseph Leonard < > > joseph.leon...@alfasystems.com> wrote: > > > > > Hi Tamás, > > > Thank you for hacking around this so rapidly. I am not familiar with > the > > > maven code base, so let me know if I have misinterpreted the change: Is > > > your hack to: > > > > > > 1. Retain the full multi-module build dependency graph such that the > > > build graph always remains consistent regardless of whether a “-pl” > arg has > > > been provided > > > 2. Introduce a skip build option to satisfy the functionality of the > > > “-pl” arg: The reactor will report the module has been built but the > build > > > is basically a no-op > > > If so, this does resolve the issue but at the cost of losing the > > > opportunity to gain extra parallelism in the build. > > > What I was wondering may be possible (without any idea of how feasible > it > > > would be to implement) would be the opportunity to recognise in the > > > multi-build distinct build graphs and then keep isolation in the > reactor > > > between these graphs. To elaborate, in the reproducer scenario there > would > > > be two distinct graphs: > > > > > > * "testsupport-module-1" followed by "app" > > > * "testsupport-module-2" followed by "module-b" > > > Ideally these two graphs build in parallel to each other and even if > > > ultimately there is a dependency between them they NEVER resolve that > > > dependency from within reactor they instead always resolve from the > > > external Maven cache. The key statement here is that if a user > presents a > > > list of modules to be built then they really need to be sure that the > > > DIRECT dependencies between these modules truly represent the build > > > graph(s) they want. For tools like gitflow-incremental-builder which > are > > > designed to carefully work out change sets and what needs to build this > > > offers a really powerful opportunity to optimise builds. I appreciate > this > > > functionality could also be seen as a gotcha (although tbf this already > > > exists). > > > Thanks, > > > Joe > > > > > > > > > On 2024/02/08 10:09:31 Tamás Cservenák wrote: > > > > Seems we are on track with this. To prove my last-night theory I > created > > > a > > > > "hack" (is really just that) and guess what? > > > > It makes reproducer behave "as expected": > > > > https://github.com/apache/maven/pull/1406 > > > > > > > > T > > > > > > > > On Wed, Feb 7, 2024 at 10:05 PM Tamás Cservenák > > > > wrote: > > > > > > > > > Howdy, > > > > > > > > > > Thank you very much, the reproducer works. Did not dig thru it > fully, > > > but > > > > > here are some related issues: > > > > > > > > > > https://issues.apache.org/jira/browse/MNG-8028 (funny thing, I > created > > > > > this few weeks ago) > > > > > https://issues.apache.org/jira/browse/MNG-6300 > > > > > > > > > > Will report back tomorrow (EU TZ) > > > > > T > > > > > > > > > > > > > > > On Wed, Feb 7, 2024 at 7:48 PM Joseph Leonard < > > > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > > > >> Hi Tamás, > > > > >> I have created a simple example here: > > > > >> https://github.com/josple/mvn-multibuild-issue > > > > >> Hopefully the
RE: Re: Re: Re: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hey, We don't need the module-b jar built in the reactor to be included in the war if the changes made in module-b won't affect the war. Instead, in this scenario, we're fine with using a jar resolved from the local repository. The requisites for this scenario are: * Any change made to module-b is only in its test tree * The only changes made to the direct and transitive dependencies of module-b are to its test scoped dependencies So basically module-b is only in the reactor for verification purposes rather than ‘src packaging’ purposes. Thanks, Joe On 2024/02/08 12:12:03 Tamás Cservenák wrote: > Hej, > > yes, basically the "hack" is well described by you: > - it retains "full' reactor project > - introduces "needs build" on project > - when it comes to building, it skips the project if flag set > > This results that module-b -> app link is not lost. > And as for your parallelization request: the truth is that there IS and > edge between app and module-b (via module-a), so despite module-a is > "done", its transitive module-b is not done, hence you cannot build these > two in parallel, as you do have an edge between app and module-b. > > Or to invert the question: what module-b JAR you want to have included in > app WAR, when building? The one built, or the one from the local > repository? If the latter, why are you building it in the first place? > > On Thu, Feb 8, 2024 at 12:06 PM Joseph Leonard < > joseph.leon...@alfasystems.com> wrote: > > > Hi Tamás, > > Thank you for hacking around this so rapidly. I am not familiar with the > > maven code base, so let me know if I have misinterpreted the change: Is > > your hack to: > > > > 1. Retain the full multi-module build dependency graph such that the > > build graph always remains consistent regardless of whether a “-pl” arg has > > been provided > > 2. Introduce a skip build option to satisfy the functionality of the > > “-pl” arg: The reactor will report the module has been built but the build > > is basically a no-op > > If so, this does resolve the issue but at the cost of losing the > > opportunity to gain extra parallelism in the build. > > What I was wondering may be possible (without any idea of how feasible it > > would be to implement) would be the opportunity to recognise in the > > multi-build distinct build graphs and then keep isolation in the reactor > > between these graphs. To elaborate, in the reproducer scenario there would > > be two distinct graphs: > > > > * "testsupport-module-1" followed by "app" > > * "testsupport-module-2" followed by "module-b" > > Ideally these two graphs build in parallel to each other and even if > > ultimately there is a dependency between them they NEVER resolve that > > dependency from within reactor they instead always resolve from the > > external Maven cache. The key statement here is that if a user presents a > > list of modules to be built then they really need to be sure that the > > DIRECT dependencies between these modules truly represent the build > > graph(s) they want. For tools like gitflow-incremental-builder which are > > designed to carefully work out change sets and what needs to build this > > offers a really powerful opportunity to optimise builds. I appreciate this > > functionality could also be seen as a gotcha (although tbf this already > > exists). > > Thanks, > > Joe > > > > > > On 2024/02/08 10:09:31 Tamás Cservenák wrote: > > > Seems we are on track with this. To prove my last-night theory I created > > a > > > "hack" (is really just that) and guess what? > > > It makes reproducer behave "as expected": > > > https://github.com/apache/maven/pull/1406 > > > > > > T > > > > > > On Wed, Feb 7, 2024 at 10:05 PM Tamás Cservenák > > wrote: > > > > > > > Howdy, > > > > > > > > Thank you very much, the reproducer works. Did not dig thru it fully, > > but > > > > here are some related issues: > > > > > > > > https://issues.apache.org/jira/browse/MNG-8028 (funny thing, I created > > > > this few weeks ago) > > > > https://issues.apache.org/jira/browse/MNG-6300 > > > > > > > > Will report back tomorrow (EU TZ) > > > > T > > > > > > > > > > > > On Wed, Feb 7, 2024 at 7:48 PM Joseph Leonard < > > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > >> Hi Tamás, > > > >> I have created a simple example here: > > > >> https://github.com/josple/mvn-multibuild-issue > > > >> Hopefully the README is clear enough – let me know if I can clarify > > > >> anything. > > > >> Thanks, > > > >> Joe > > > >> > > > >> On 2024/02/07 17:33:08 Tamás Cservenák wrote: > > > >> > Howdy, > > > >> > > > > >> > In that case, there is something fishy with the project, my blind > > guess > > > >> > would be some "hidden" inter-module dependency maybe? > > > >> > > > > >> > Can you provide access to source, or, if not feasible, could you > > provide > > > >> > some reproducer and publish it on Github/Gitlab/whatever (maybe even > > > >> just > > > >> > send it privately as ML strips off
Re: Re: Re: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hej, yes, basically the "hack" is well described by you: - it retains "full' reactor project - introduces "needs build" on project - when it comes to building, it skips the project if flag set This results that module-b -> app link is not lost. And as for your parallelization request: the truth is that there IS and edge between app and module-b (via module-a), so despite module-a is "done", its transitive module-b is not done, hence you cannot build these two in parallel, as you do have an edge between app and module-b. Or to invert the question: what module-b JAR you want to have included in app WAR, when building? The one built, or the one from the local repository? If the latter, why are you building it in the first place? On Thu, Feb 8, 2024 at 12:06 PM Joseph Leonard < joseph.leon...@alfasystems.com> wrote: > Hi Tamás, > Thank you for hacking around this so rapidly. I am not familiar with the > maven code base, so let me know if I have misinterpreted the change: Is > your hack to: > > 1. Retain the full multi-module build dependency graph such that the > build graph always remains consistent regardless of whether a “-pl” arg has > been provided > 2. Introduce a skip build option to satisfy the functionality of the > “-pl” arg: The reactor will report the module has been built but the build > is basically a no-op > If so, this does resolve the issue but at the cost of losing the > opportunity to gain extra parallelism in the build. > What I was wondering may be possible (without any idea of how feasible it > would be to implement) would be the opportunity to recognise in the > multi-build distinct build graphs and then keep isolation in the reactor > between these graphs. To elaborate, in the reproducer scenario there would > be two distinct graphs: > > * "testsupport-module-1" followed by "app" > * "testsupport-module-2" followed by "module-b" > Ideally these two graphs build in parallel to each other and even if > ultimately there is a dependency between them they NEVER resolve that > dependency from within reactor they instead always resolve from the > external Maven cache. The key statement here is that if a user presents a > list of modules to be built then they really need to be sure that the > DIRECT dependencies between these modules truly represent the build > graph(s) they want. For tools like gitflow-incremental-builder which are > designed to carefully work out change sets and what needs to build this > offers a really powerful opportunity to optimise builds. I appreciate this > functionality could also be seen as a gotcha (although tbf this already > exists). > Thanks, > Joe > > > On 2024/02/08 10:09:31 Tamás Cservenák wrote: > > Seems we are on track with this. To prove my last-night theory I created > a > > "hack" (is really just that) and guess what? > > It makes reproducer behave "as expected": > > https://github.com/apache/maven/pull/1406 > > > > T > > > > On Wed, Feb 7, 2024 at 10:05 PM Tamás Cservenák > wrote: > > > > > Howdy, > > > > > > Thank you very much, the reproducer works. Did not dig thru it fully, > but > > > here are some related issues: > > > > > > https://issues.apache.org/jira/browse/MNG-8028 (funny thing, I created > > > this few weeks ago) > > > https://issues.apache.org/jira/browse/MNG-6300 > > > > > > Will report back tomorrow (EU TZ) > > > T > > > > > > > > > On Wed, Feb 7, 2024 at 7:48 PM Joseph Leonard < > > > joseph.leon...@alfasystems.com> wrote: > > > > > >> Hi Tamás, > > >> I have created a simple example here: > > >> https://github.com/josple/mvn-multibuild-issue > > >> Hopefully the README is clear enough – let me know if I can clarify > > >> anything. > > >> Thanks, > > >> Joe > > >> > > >> On 2024/02/07 17:33:08 Tamás Cservenák wrote: > > >> > Howdy, > > >> > > > >> > In that case, there is something fishy with the project, my blind > guess > > >> > would be some "hidden" inter-module dependency maybe? > > >> > > > >> > Can you provide access to source, or, if not feasible, could you > provide > > >> > some reproducer and publish it on Github/Gitlab/whatever (maybe even > > >> just > > >> > send it privately as ML strips off attachments and images) for us > to see > > >> > this in action? > > >> > > > >> > Thanks > > >> > T > > >> > > > >> > On Wed, Feb 7, 2024 at 6:29 PM Joseph Leonard < > > >> > joseph.leon...@alfasystems.com> wrote: > > >> > > > >> > > Hi Tamás, > > >> > > We have previously played around a bit with mvnd but not takari > > >> directly – > > >> > > I will have a play with it. With regards to this issue, using the > > >> takari > > >> > > smart builder unfortunately doesn’t resolve the issue. > > >> > > Joe > > >> > > > > >> > > On 2024/02/07 11:41:22 Tamás Cservenák wrote: > > >> > > > Can you please try smart builder instead? > > >> > > > https://github.com/takari/takari-smart-builder > > >> > > > > > >> > > > (note: smart builder is used by mvnd as well) > > >> > > > > > >> > > > The difference between the two
RE: Re: Re: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hi Tamás, Thank you for hacking around this so rapidly. I am not familiar with the maven code base, so let me know if I have misinterpreted the change: Is your hack to: 1. Retain the full multi-module build dependency graph such that the build graph always remains consistent regardless of whether a “-pl” arg has been provided 2. Introduce a skip build option to satisfy the functionality of the “-pl” arg: The reactor will report the module has been built but the build is basically a no-op If so, this does resolve the issue but at the cost of losing the opportunity to gain extra parallelism in the build. What I was wondering may be possible (without any idea of how feasible it would be to implement) would be the opportunity to recognise in the multi-build distinct build graphs and then keep isolation in the reactor between these graphs. To elaborate, in the reproducer scenario there would be two distinct graphs: * "testsupport-module-1" followed by "app" * "testsupport-module-2" followed by "module-b" Ideally these two graphs build in parallel to each other and even if ultimately there is a dependency between them they NEVER resolve that dependency from within reactor they instead always resolve from the external Maven cache. The key statement here is that if a user presents a list of modules to be built then they really need to be sure that the DIRECT dependencies between these modules truly represent the build graph(s) they want. For tools like gitflow-incremental-builder which are designed to carefully work out change sets and what needs to build this offers a really powerful opportunity to optimise builds. I appreciate this functionality could also be seen as a gotcha (although tbf this already exists). Thanks, Joe On 2024/02/08 10:09:31 Tamás Cservenák wrote: > Seems we are on track with this. To prove my last-night theory I created a > "hack" (is really just that) and guess what? > It makes reproducer behave "as expected": > https://github.com/apache/maven/pull/1406 > > T > > On Wed, Feb 7, 2024 at 10:05 PM Tamás Cservenák wrote: > > > Howdy, > > > > Thank you very much, the reproducer works. Did not dig thru it fully, but > > here are some related issues: > > > > https://issues.apache.org/jira/browse/MNG-8028 (funny thing, I created > > this few weeks ago) > > https://issues.apache.org/jira/browse/MNG-6300 > > > > Will report back tomorrow (EU TZ) > > T > > > > > > On Wed, Feb 7, 2024 at 7:48 PM Joseph Leonard < > > joseph.leon...@alfasystems.com> wrote: > > > >> Hi Tamás, > >> I have created a simple example here: > >> https://github.com/josple/mvn-multibuild-issue > >> Hopefully the README is clear enough – let me know if I can clarify > >> anything. > >> Thanks, > >> Joe > >> > >> On 2024/02/07 17:33:08 Tamás Cservenák wrote: > >> > Howdy, > >> > > >> > In that case, there is something fishy with the project, my blind guess > >> > would be some "hidden" inter-module dependency maybe? > >> > > >> > Can you provide access to source, or, if not feasible, could you provide > >> > some reproducer and publish it on Github/Gitlab/whatever (maybe even > >> just > >> > send it privately as ML strips off attachments and images) for us to see > >> > this in action? > >> > > >> > Thanks > >> > T > >> > > >> > On Wed, Feb 7, 2024 at 6:29 PM Joseph Leonard < > >> > joseph.leon...@alfasystems.com> wrote: > >> > > >> > > Hi Tamás, > >> > > We have previously played around a bit with mvnd but not takari > >> directly – > >> > > I will have a play with it. With regards to this issue, using the > >> takari > >> > > smart builder unfortunately doesn’t resolve the issue. > >> > > Joe > >> > > > >> > > On 2024/02/07 11:41:22 Tamás Cservenák wrote: > >> > > > Can you please try smart builder instead? > >> > > > https://github.com/takari/takari-smart-builder > >> > > > > >> > > > (note: smart builder is used by mvnd as well) > >> > > > > >> > > > The difference between the two can be seen here: > >> > > > http://takari.io/book/30-team-maven.html#takari-smart-builder > >> > > > > >> > > > On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < > >> > > > joseph.leon...@alfasystems.com> wrote: > >> > > > > >> > > > > Hi Tamás, > >> > > > > Yeah, this was unexpected to me initially as well. From what I > >> can tell > >> > > > > the Maven reactor only considers direct dependencies (i.e. not > >> > > transitive > >> > > > > dependencies) between the modules in the reactor when working out > >> the > >> > > build > >> > > > > graph. For example if you have a simple linear dependency chain > >> of: > >> > > > > One --> Two --> Three --> Four --> Five > >> > > > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will > >> result > >> > > in > >> > > > > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). > >> I > >> > > assume > >> > > > > this is as designed because it actually offers quite powerful > >> > > functionality > >> > > > > to improve the parallelism in your
Re: Re: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Seems we are on track with this. To prove my last-night theory I created a "hack" (is really just that) and guess what? It makes reproducer behave "as expected": https://github.com/apache/maven/pull/1406 T On Wed, Feb 7, 2024 at 10:05 PM Tamás Cservenák wrote: > Howdy, > > Thank you very much, the reproducer works. Did not dig thru it fully, but > here are some related issues: > > https://issues.apache.org/jira/browse/MNG-8028 (funny thing, I created > this few weeks ago) > https://issues.apache.org/jira/browse/MNG-6300 > > Will report back tomorrow (EU TZ) > T > > > On Wed, Feb 7, 2024 at 7:48 PM Joseph Leonard < > joseph.leon...@alfasystems.com> wrote: > >> Hi Tamás, >> I have created a simple example here: >> https://github.com/josple/mvn-multibuild-issue >> Hopefully the README is clear enough – let me know if I can clarify >> anything. >> Thanks, >> Joe >> >> On 2024/02/07 17:33:08 Tamás Cservenák wrote: >> > Howdy, >> > >> > In that case, there is something fishy with the project, my blind guess >> > would be some "hidden" inter-module dependency maybe? >> > >> > Can you provide access to source, or, if not feasible, could you provide >> > some reproducer and publish it on Github/Gitlab/whatever (maybe even >> just >> > send it privately as ML strips off attachments and images) for us to see >> > this in action? >> > >> > Thanks >> > T >> > >> > On Wed, Feb 7, 2024 at 6:29 PM Joseph Leonard < >> > joseph.leon...@alfasystems.com> wrote: >> > >> > > Hi Tamás, >> > > We have previously played around a bit with mvnd but not takari >> directly – >> > > I will have a play with it. With regards to this issue, using the >> takari >> > > smart builder unfortunately doesn’t resolve the issue. >> > > Joe >> > > >> > > On 2024/02/07 11:41:22 Tamás Cservenák wrote: >> > > > Can you please try smart builder instead? >> > > > https://github.com/takari/takari-smart-builder >> > > > >> > > > (note: smart builder is used by mvnd as well) >> > > > >> > > > The difference between the two can be seen here: >> > > > http://takari.io/book/30-team-maven.html#takari-smart-builder >> > > > >> > > > On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < >> > > > joseph.leon...@alfasystems.com> wrote: >> > > > >> > > > > Hi Tamás, >> > > > > Yeah, this was unexpected to me initially as well. From what I >> can tell >> > > > > the Maven reactor only considers direct dependencies (i.e. not >> > > transitive >> > > > > dependencies) between the modules in the reactor when working out >> the >> > > build >> > > > > graph. For example if you have a simple linear dependency chain >> of: >> > > > > One --> Two --> Three --> Four --> Five >> > > > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will >> result >> > > in >> > > > > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). >> I >> > > assume >> > > > > this is as designed because it actually offers quite powerful >> > > functionality >> > > > > to improve the parallelism in your build. An example of where >> this is >> > > legit >> > > > > is when: >> > > > > >> > > > > * “Four” has a test scope dependency on “Five” >> > > > > * “One” has a test scoped dependency on “Two” >> > > > > If you made a src code change to “Five” and “Two” then it would be >> > > safe to >> > > > > build [One,Two] and [Four,Five] in parallel because you know the >> > > changes >> > > > > within these graphs cannot impact each other. >> > > > > Joe >> > > > > >> > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: >> > > > > > Howdy, >> > > > > > >> > > > > > To me this looks like Maven is not aware that the App depends on >> > > > > ModuleB... >> > > > > > Are they "plain dependency" linked? Or what kind of dependency >> we >> > > talk >> > > > > > about here? >> > > > > > In short: why would App start while ModuleB (upstream dep) is >> not >> > > done? >> > > > > > Something is fishy here. >> > > > > > >> > > > > > T >> > > > > > >> > > > > > >> > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < >> > > > > > joseph.leon...@alfasystems.com> wrote: >> > > > > > >> > > > > > > Hi all, >> > > > > > > >> > > > > > > It would be great to get any thoughts on whether the >> following is a >> > > > > defect: >> > > > > > > >> > > > > > > >> > > > > > > Issue details: >> > > > > > > tl;dr >> > > > > > > >> > > > > > > Maven can resolve dependencies either from: >> > > > > > > >> > > > > > > * an external repo >> > > > > > > * a class directory of a module being built within the >> reactor >> > > > > > > * a packaged jar of a module being built within the >> reactor >> > > > > > > >> > > > > > > If you run a concurrent multi-module build it is possible to >> get a >> > > race >> > > > > > > condition whereby the build of module Foo may resolve module >> Bar >> > > from >> > > > > > > either of the three resolution channels. This inconsistency >> can >> > > result >> > > > > in >> > > > > > > the Maven war plugin sometimes failing to build a functional >> war
Re: Re: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Howdy, Thank you very much, the reproducer works. Did not dig thru it fully, but here are some related issues: https://issues.apache.org/jira/browse/MNG-8028 (funny thing, I created this few weeks ago) https://issues.apache.org/jira/browse/MNG-6300 Will report back tomorrow (EU TZ) T On Wed, Feb 7, 2024 at 7:48 PM Joseph Leonard < joseph.leon...@alfasystems.com> wrote: > Hi Tamás, > I have created a simple example here: > https://github.com/josple/mvn-multibuild-issue > Hopefully the README is clear enough – let me know if I can clarify > anything. > Thanks, > Joe > > On 2024/02/07 17:33:08 Tamás Cservenák wrote: > > Howdy, > > > > In that case, there is something fishy with the project, my blind guess > > would be some "hidden" inter-module dependency maybe? > > > > Can you provide access to source, or, if not feasible, could you provide > > some reproducer and publish it on Github/Gitlab/whatever (maybe even just > > send it privately as ML strips off attachments and images) for us to see > > this in action? > > > > Thanks > > T > > > > On Wed, Feb 7, 2024 at 6:29 PM Joseph Leonard < > > joseph.leon...@alfasystems.com> wrote: > > > > > Hi Tamás, > > > We have previously played around a bit with mvnd but not takari > directly – > > > I will have a play with it. With regards to this issue, using the > takari > > > smart builder unfortunately doesn’t resolve the issue. > > > Joe > > > > > > On 2024/02/07 11:41:22 Tamás Cservenák wrote: > > > > Can you please try smart builder instead? > > > > https://github.com/takari/takari-smart-builder > > > > > > > > (note: smart builder is used by mvnd as well) > > > > > > > > The difference between the two can be seen here: > > > > http://takari.io/book/30-team-maven.html#takari-smart-builder > > > > > > > > On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < > > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > > > Hi Tamás, > > > > > Yeah, this was unexpected to me initially as well. From what I can > tell > > > > > the Maven reactor only considers direct dependencies (i.e. not > > > transitive > > > > > dependencies) between the modules in the reactor when working out > the > > > build > > > > > graph. For example if you have a simple linear dependency chain of: > > > > > One --> Two --> Three --> Four --> Five > > > > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will > result > > > in > > > > > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I > > > assume > > > > > this is as designed because it actually offers quite powerful > > > functionality > > > > > to improve the parallelism in your build. An example of where this > is > > > legit > > > > > is when: > > > > > > > > > > * “Four” has a test scope dependency on “Five” > > > > > * “One” has a test scoped dependency on “Two” > > > > > If you made a src code change to “Five” and “Two” then it would be > > > safe to > > > > > build [One,Two] and [Four,Five] in parallel because you know the > > > changes > > > > > within these graphs cannot impact each other. > > > > > Joe > > > > > > > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > > > > > Howdy, > > > > > > > > > > > > To me this looks like Maven is not aware that the App depends on > > > > > ModuleB... > > > > > > Are they "plain dependency" linked? Or what kind of dependency we > > > talk > > > > > > about here? > > > > > > In short: why would App start while ModuleB (upstream dep) is not > > > done? > > > > > > Something is fishy here. > > > > > > > > > > > > T > > > > > > > > > > > > > > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > > > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > It would be great to get any thoughts on whether the following > is a > > > > > defect: > > > > > > > > > > > > > > > > > > > > > Issue details: > > > > > > > tl;dr > > > > > > > > > > > > > > Maven can resolve dependencies either from: > > > > > > > > > > > > > > * an external repo > > > > > > > * a class directory of a module being built within the > reactor > > > > > > > * a packaged jar of a module being built within the reactor > > > > > > > > > > > > > > If you run a concurrent multi-module build it is possible to > get a > > > race > > > > > > > condition whereby the build of module Foo may resolve module > Bar > > > from > > > > > > > either of the three resolution channels. This inconsistency can > > > result > > > > > in > > > > > > > the Maven war plugin sometimes failing to build a functional > war > > > file. > > > > > I > > > > > > > would expect a consistent resolution would always take place. > > > > > > > > > > > > > > Full details > > > > > > > Scenario > > > > > > > > > > > > > > Consider you have a repo with the following structure: > > > > > > > > > > > > > >App > > > > > > > > > > > > > > / \ > > > > > > > > > > > > > > / \ > > > > > > > >
RE: Re: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hi Tamás, I have created a simple example here: https://github.com/josple/mvn-multibuild-issue Hopefully the README is clear enough – let me know if I can clarify anything. Thanks, Joe On 2024/02/07 17:33:08 Tamás Cservenák wrote: > Howdy, > > In that case, there is something fishy with the project, my blind guess > would be some "hidden" inter-module dependency maybe? > > Can you provide access to source, or, if not feasible, could you provide > some reproducer and publish it on Github/Gitlab/whatever (maybe even just > send it privately as ML strips off attachments and images) for us to see > this in action? > > Thanks > T > > On Wed, Feb 7, 2024 at 6:29 PM Joseph Leonard < > joseph.leon...@alfasystems.com> wrote: > > > Hi Tamás, > > We have previously played around a bit with mvnd but not takari directly – > > I will have a play with it. With regards to this issue, using the takari > > smart builder unfortunately doesn’t resolve the issue. > > Joe > > > > On 2024/02/07 11:41:22 Tamás Cservenák wrote: > > > Can you please try smart builder instead? > > > https://github.com/takari/takari-smart-builder > > > > > > (note: smart builder is used by mvnd as well) > > > > > > The difference between the two can be seen here: > > > http://takari.io/book/30-team-maven.html#takari-smart-builder > > > > > > On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > Hi Tamás, > > > > Yeah, this was unexpected to me initially as well. From what I can tell > > > > the Maven reactor only considers direct dependencies (i.e. not > > transitive > > > > dependencies) between the modules in the reactor when working out the > > build > > > > graph. For example if you have a simple linear dependency chain of: > > > > One --> Two --> Three --> Four --> Five > > > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will result > > in > > > > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I > > assume > > > > this is as designed because it actually offers quite powerful > > functionality > > > > to improve the parallelism in your build. An example of where this is > > legit > > > > is when: > > > > > > > > * “Four” has a test scope dependency on “Five” > > > > * “One” has a test scoped dependency on “Two” > > > > If you made a src code change to “Five” and “Two” then it would be > > safe to > > > > build [One,Two] and [Four,Five] in parallel because you know the > > changes > > > > within these graphs cannot impact each other. > > > > Joe > > > > > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > > > > Howdy, > > > > > > > > > > To me this looks like Maven is not aware that the App depends on > > > > ModuleB... > > > > > Are they "plain dependency" linked? Or what kind of dependency we > > talk > > > > > about here? > > > > > In short: why would App start while ModuleB (upstream dep) is not > > done? > > > > > Something is fishy here. > > > > > > > > > > T > > > > > > > > > > > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > It would be great to get any thoughts on whether the following is a > > > > defect: > > > > > > > > > > > > > > > > > > Issue details: > > > > > > tl;dr > > > > > > > > > > > > Maven can resolve dependencies either from: > > > > > > > > > > > > * an external repo > > > > > > * a class directory of a module being built within the reactor > > > > > > * a packaged jar of a module being built within the reactor > > > > > > > > > > > > If you run a concurrent multi-module build it is possible to get a > > race > > > > > > condition whereby the build of module Foo may resolve module Bar > > from > > > > > > either of the three resolution channels. This inconsistency can > > result > > > > in > > > > > > the Maven war plugin sometimes failing to build a functional war > > file. > > > > I > > > > > > would expect a consistent resolution would always take place. > > > > > > > > > > > > Full details > > > > > > Scenario > > > > > > > > > > > > Consider you have a repo with the following structure: > > > > > > > > > > > >App > > > > > > > > > > > > / \ > > > > > > > > > > > > / \ > > > > > > > > > > > >(compile scope) (test scope) > > > > > > > > > > > > / \ > > > > > > > > > > > > \/_ _\/ > > > > > > > > > > > > ModuleA TestSupportModule1 > > > > > > > > > > > > / > > > > > > > > > > > >/ > > > > > > > > > > > > (compile scope) > > > > > > > > > > > > / > > > > > > > > > > > >\/_ > > > > > > > > > > > > ModuleB > > > > > > > > > > > >/ > > > > > > > > > > > > / > > > > > > > > > > > > (test scope) > > > > > > > > > > > > / > > > > > > > > > > > >
Re: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Howdy, In that case, there is something fishy with the project, my blind guess would be some "hidden" inter-module dependency maybe? Can you provide access to source, or, if not feasible, could you provide some reproducer and publish it on Github/Gitlab/whatever (maybe even just send it privately as ML strips off attachments and images) for us to see this in action? Thanks T On Wed, Feb 7, 2024 at 6:29 PM Joseph Leonard < joseph.leon...@alfasystems.com> wrote: > Hi Tamás, > We have previously played around a bit with mvnd but not takari directly – > I will have a play with it. With regards to this issue, using the takari > smart builder unfortunately doesn’t resolve the issue. > Joe > > On 2024/02/07 11:41:22 Tamás Cservenák wrote: > > Can you please try smart builder instead? > > https://github.com/takari/takari-smart-builder > > > > (note: smart builder is used by mvnd as well) > > > > The difference between the two can be seen here: > > http://takari.io/book/30-team-maven.html#takari-smart-builder > > > > On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < > > joseph.leon...@alfasystems.com> wrote: > > > > > Hi Tamás, > > > Yeah, this was unexpected to me initially as well. From what I can tell > > > the Maven reactor only considers direct dependencies (i.e. not > transitive > > > dependencies) between the modules in the reactor when working out the > build > > > graph. For example if you have a simple linear dependency chain of: > > > One --> Two --> Three --> Four --> Five > > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will result > in > > > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I > assume > > > this is as designed because it actually offers quite powerful > functionality > > > to improve the parallelism in your build. An example of where this is > legit > > > is when: > > > > > > * “Four” has a test scope dependency on “Five” > > > * “One” has a test scoped dependency on “Two” > > > If you made a src code change to “Five” and “Two” then it would be > safe to > > > build [One,Two] and [Four,Five] in parallel because you know the > changes > > > within these graphs cannot impact each other. > > > Joe > > > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > > > Howdy, > > > > > > > > To me this looks like Maven is not aware that the App depends on > > > ModuleB... > > > > Are they "plain dependency" linked? Or what kind of dependency we > talk > > > > about here? > > > > In short: why would App start while ModuleB (upstream dep) is not > done? > > > > Something is fishy here. > > > > > > > > T > > > > > > > > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > > > Hi all, > > > > > > > > > > It would be great to get any thoughts on whether the following is a > > > defect: > > > > > > > > > > > > > > > Issue details: > > > > > tl;dr > > > > > > > > > > Maven can resolve dependencies either from: > > > > > > > > > > * an external repo > > > > > * a class directory of a module being built within the reactor > > > > > * a packaged jar of a module being built within the reactor > > > > > > > > > > If you run a concurrent multi-module build it is possible to get a > race > > > > > condition whereby the build of module Foo may resolve module Bar > from > > > > > either of the three resolution channels. This inconsistency can > result > > > in > > > > > the Maven war plugin sometimes failing to build a functional war > file. > > > I > > > > > would expect a consistent resolution would always take place. > > > > > > > > > > Full details > > > > > Scenario > > > > > > > > > > Consider you have a repo with the following structure: > > > > > > > > > >App > > > > > > > > > > / \ > > > > > > > > > > / \ > > > > > > > > > >(compile scope) (test scope) > > > > > > > > > > / \ > > > > > > > > > > \/_ _\/ > > > > > > > > > > ModuleA TestSupportModule1 > > > > > > > > > > / > > > > > > > > > >/ > > > > > > > > > > (compile scope) > > > > > > > > > > / > > > > > > > > > >\/_ > > > > > > > > > > ModuleB > > > > > > > > > >/ > > > > > > > > > > / > > > > > > > > > > (test scope) > > > > > > > > > > / > > > > > > > > > > \/_ > > > > > > > > > > TestSupportModule2 > > > > > > > > > > If you were to make a src code change to the following test support > > > > > modules: > > > > > > > > > > * TestSupportModule1 > > > > > * TestSupportModule2 > > > > > > > > > > Then the minimum number of modules we need to build to verify the > > > change > > > > > set is OK is: > > > > > > > > > > * TestSupportModule1 > > > > > * TestSupportModule2 > > > > > * ModuleB > > > > > * App > > > > > > > > > > i.e. there is no
RE: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hi Tamás, We have previously played around a bit with mvnd but not takari directly – I will have a play with it. With regards to this issue, using the takari smart builder unfortunately doesn’t resolve the issue. Joe On 2024/02/07 11:41:22 Tamás Cservenák wrote: > Can you please try smart builder instead? > https://github.com/takari/takari-smart-builder > > (note: smart builder is used by mvnd as well) > > The difference between the two can be seen here: > http://takari.io/book/30-team-maven.html#takari-smart-builder > > On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < > joseph.leon...@alfasystems.com> wrote: > > > Hi Tamás, > > Yeah, this was unexpected to me initially as well. From what I can tell > > the Maven reactor only considers direct dependencies (i.e. not transitive > > dependencies) between the modules in the reactor when working out the build > > graph. For example if you have a simple linear dependency chain of: > > One --> Two --> Three --> Four --> Five > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will result in > > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I assume > > this is as designed because it actually offers quite powerful functionality > > to improve the parallelism in your build. An example of where this is legit > > is when: > > > > * “Four” has a test scope dependency on “Five” > > * “One” has a test scoped dependency on “Two” > > If you made a src code change to “Five” and “Two” then it would be safe to > > build [One,Two] and [Four,Five] in parallel because you know the changes > > within these graphs cannot impact each other. > > Joe > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > > Howdy, > > > > > > To me this looks like Maven is not aware that the App depends on > > ModuleB... > > > Are they "plain dependency" linked? Or what kind of dependency we talk > > > about here? > > > In short: why would App start while ModuleB (upstream dep) is not done? > > > Something is fishy here. > > > > > > T > > > > > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > Hi all, > > > > > > > > It would be great to get any thoughts on whether the following is a > > defect: > > > > > > > > > > > > Issue details: > > > > tl;dr > > > > > > > > Maven can resolve dependencies either from: > > > > > > > > * an external repo > > > > * a class directory of a module being built within the reactor > > > > * a packaged jar of a module being built within the reactor > > > > > > > > If you run a concurrent multi-module build it is possible to get a race > > > > condition whereby the build of module Foo may resolve module Bar from > > > > either of the three resolution channels. This inconsistency can result > > in > > > > the Maven war plugin sometimes failing to build a functional war file. > > I > > > > would expect a consistent resolution would always take place. > > > > > > > > Full details > > > > Scenario > > > > > > > > Consider you have a repo with the following structure: > > > > > > > >App > > > > > > > > / \ > > > > > > > > / \ > > > > > > > >(compile scope) (test scope) > > > > > > > > / \ > > > > > > > > \/_ _\/ > > > > > > > > ModuleA TestSupportModule1 > > > > > > > > / > > > > > > > >/ > > > > > > > > (compile scope) > > > > > > > > / > > > > > > > >\/_ > > > > > > > > ModuleB > > > > > > > >/ > > > > > > > > / > > > > > > > > (test scope) > > > > > > > > / > > > > > > > > \/_ > > > > > > > > TestSupportModule2 > > > > > > > > If you were to make a src code change to the following test support > > > > modules: > > > > > > > > * TestSupportModule1 > > > > * TestSupportModule2 > > > > > > > > Then the minimum number of modules we need to build to verify the > > change > > > > set is OK is: > > > > > > > > * TestSupportModule1 > > > > * TestSupportModule2 > > > > * ModuleB > > > > * App > > > > > > > > i.e. there is no requirement to build ModuleA because we know that > > none of > > > > the src code changes could impact the classpaths used in its maven > > build. > > > > > > > > We know that despite 'App' depending (transitively) on ModuleB there > > is no > > > > need for the 'App' build to wait for ModuleB to complete its build > > because > > > > the src code change to TestSupportModule2 will not impact any of the > > > > classpaths used in the App maven build. Therefore to get the most > > efficient > > > > build possible we ideally would invoke Maven to run with 2 threads and > > with > > > > instruction to build two distinct 'dependency graphs': > > > > > > > > * TestSupportModule1 followed by ModuleB > > > > * TestSupportModule1 followed by App > > > >
RE: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hi Nils, Ah I see. We actually already use ‘-pl …’ arguments in conjunction with the Develocity Build Cache implementation because we have found this to still generate very significant time savings. Primarily because even with hits on the build cache there is still fairly substantial time spent resolving cached artifacts (and the reactor doing all its goal steps and determining cache hits). The other advantage of using ‘-pl …’ is that it gives opportunity for even greater parallelism (per my sub-thread example with Tamás). Joe On 2024/02/07 10:59:28 Nils Breunese wrote: > Hi Jospeh, > > I didn’t necessarily expect that enabling the extension would solve/avoid the > issue you described, but I mentioned it, because it would allow you to not > have to specify an argument like '-pl > TestSupportModule1,TestSupportModule2,ModuleB,App’ in the first place, > because the Build Cache Extension should automatically determine which > modules need to be built and for which ones previously cached artifacts can > be used. > > Nils. > > > Op 6 feb 2024, om 15:11 heeft Joseph Leonard het > > volgende geschreven: > > > > Thanks Nils, > > maven-build-cache-extension looks very interesting generally – I will have > > a play with it. > > With regards to this issue, the maven-build-cache-extension overview > > references “Subtree support for multimodule projects builds part of the > > codebase in isolation” which sounded similar to the solution I am proposing > > for this issue. Unfortunately, after enabling the > > maven-build-cache-extension I still hit the same issue. > > Joe > > > > On 2024/02/06 12:54:59 Nils Breunese wrote: > >> I can’t comment on your question directly, but I just wanted to say that > >> your use case sounds like it could benefit from the Maven Build Cache > >> Extension > >> (https://maven.apache.org/extensions/maven-build-cache-extension/). > > > >> > >> Just my 2 cents. > >> > >> Nils. > >> > >>> Op 6 feb 2024 om 11:40 heeft Joseph Leonard het > >>> volgende geschreven: > >>> > >>> Hi all, > >>> > >>> It would be great to get any thoughts on whether the following is a > >>> defect: > >>> > >>> > >>> Issue details: > >>> tl;dr > >>> > >>> Maven can resolve dependencies either from: > >>> > >>> * an external repo > >>> * a class directory of a module being built within the reactor > >>> * a packaged jar of a module being built within the reactor > >>> > >>> If you run a concurrent multi-module build it is possible to get a race > >>> condition whereby the build of module Foo may resolve module Bar from > >>> either of the three resolution channels. This inconsistency can result in > >>> the Maven war plugin sometimes failing to build a functional war file. I > >>> would expect a consistent resolution would always take place. > > > >>> > >>> Full details > >>> Scenario > >>> > >>> Consider you have a repo with the following structure: > >>> > >>> App > >>> > >>>/ \ > >>> > >>> / \ > >>> > >>> (compile scope) (test scope) > >>> > >>> / \ > >>> > >>> \/_ _\/ > >>> > >>>ModuleA TestSupportModule1 > >>> > >>> / > >>> > >>> / > >>> > >>> (compile scope) > >>> > >>>/ > >>> > >>> \/_ > >>> > >>> ModuleB > >>> > >>> / > >>> > >>> / > >>> > >>> (test scope) > >>> > >>> / > >>> > >>> \/_ > >>> > >>> TestSupportModule2 > >>> > >>> If you were to make a src code change to the following test support > >>> modules: > >>> > >>> * TestSupportModule1 > >>> * TestSupportModule2 > >>> > >>> Then the minimum number of modules we need to build to verify the change > >>> set is OK is: > >>> > >>> * TestSupportModule1 > >>> * TestSupportModule2 > >>> * ModuleB > >>> * App > >>> > >>> i.e. there is no requirement to build ModuleA because we know that none > >>> of the src code changes could impact the classpaths used in its maven > >>> build. > > > >>> > >>> We know that despite 'App' depending (transitively) on ModuleB there is > >>> no need for the 'App' build to wait for ModuleB to complete its build > >>> because the src code change to TestSupportModule2 will not impact any of > >>> the classpaths used in the App maven build. Therefore to get the most > >>> efficient build possible we ideally would invoke Maven to run with 2 > >>> threads and with instruction to build two distinct 'dependency graphs': > > > >>> > >>> * TestSupportModule1 followed by ModuleB > >>> * TestSupportModule1 followed by App > >>> > >>> The following Maven command achieves exactly what we want because the > >>> reactor build order is based only on the direct (i.e. non-transitive) > >>> dependencies of the modules provided to the reactor in the build command. > >>> Therefore the absence of ModuleA results in two distinct 'dependency > >>> graphs': >
Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Howdy, see here https://maven.apache.org/mailing-lists.html On Wed, Feb 7, 2024 at 12:55 PM Amn Ojee Uw wrote: > How can I unsubscribe from this mailing list? > Thanks in advance. > > *ArbolOne > Using Fire Fox and Thunderbird. > Developing for Android using Java, C/C++, HTM/CSS and SQLite as our > platform has been exciting and most rewarding. > [ Ñ ]* > > > > On Wed, Feb 7, 2024 at 5:50 AM Joseph Leonard < > joseph.leon...@alfasystems.com> wrote: > > > Hi Tamás, > > Yeah, this was unexpected to me initially as well. From what I can tell > > the Maven reactor only considers direct dependencies (i.e. not transitive > > dependencies) between the modules in the reactor when working out the > build > > graph. For example if you have a simple linear dependency chain of: > > One --> Two --> Three --> Four --> Five > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will result in > > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I > assume > > this is as designed because it actually offers quite powerful > functionality > > to improve the parallelism in your build. An example of where this is > legit > > is when: > > > > * “Four” has a test scope dependency on “Five” > > * “One” has a test scoped dependency on “Two” > > If you made a src code change to “Five” and “Two” then it would be safe > to > > build [One,Two] and [Four,Five] in parallel because you know the changes > > within these graphs cannot impact each other. > > Joe > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > > Howdy, > > > > > > To me this looks like Maven is not aware that the App depends on > > ModuleB... > > > Are they "plain dependency" linked? Or what kind of dependency we talk > > > about here? > > > In short: why would App start while ModuleB (upstream dep) is not done? > > > Something is fishy here. > > > > > > T > > > > > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > Hi all, > > > > > > > > It would be great to get any thoughts on whether the following is a > > defect: > > > > > > > > > > > > Issue details: > > > > tl;dr > > > > > > > > Maven can resolve dependencies either from: > > > > > > > > * an external repo > > > > * a class directory of a module being built within the reactor > > > > * a packaged jar of a module being built within the reactor > > > > > > > > If you run a concurrent multi-module build it is possible to get a > race > > > > condition whereby the build of module Foo may resolve module Bar from > > > > either of the three resolution channels. This inconsistency can > result > > in > > > > the Maven war plugin sometimes failing to build a functional war > file. > > I > > > > would expect a consistent resolution would always take place. > > > > > > > > Full details > > > > Scenario > > > > > > > > Consider you have a repo with the following structure: > > > > > > > >App > > > > > > > > / \ > > > > > > > > / \ > > > > > > > >(compile scope) (test scope) > > > > > > > > / \ > > > > > > > > \/_ _\/ > > > > > > > > ModuleA TestSupportModule1 > > > > > > > > / > > > > > > > >/ > > > > > > > > (compile scope) > > > > > > > > / > > > > > > > >\/_ > > > > > > > > ModuleB > > > > > > > >/ > > > > > > > > / > > > > > > > > (test scope) > > > > > > > > / > > > > > > > > \/_ > > > > > > > > TestSupportModule2 > > > > > > > > If you were to make a src code change to the following test support > > > > modules: > > > > > > > > * TestSupportModule1 > > > > * TestSupportModule2 > > > > > > > > Then the minimum number of modules we need to build to verify the > > change > > > > set is OK is: > > > > > > > > * TestSupportModule1 > > > > * TestSupportModule2 > > > > * ModuleB > > > > * App > > > > > > > > i.e. there is no requirement to build ModuleA because we know that > > none of > > > > the src code changes could impact the classpaths used in its maven > > build. > > > > > > > > We know that despite 'App' depending (transitively) on ModuleB there > > is no > > > > need for the 'App' build to wait for ModuleB to complete its build > > because > > > > the src code change to TestSupportModule2 will not impact any of the > > > > classpaths used in the App maven build. Therefore to get the most > > efficient > > > > build possible we ideally would invoke Maven to run with 2 threads > and > > with > > > > instruction to build two distinct 'dependency graphs': > > > > > > > > * TestSupportModule1 followed by ModuleB > > > > * TestSupportModule1 followed by App > > > > > > > > The following Maven command achieves exactly what we want because the > > > > reactor build order is based only on the direct (i.e.
Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
How can I unsubscribe from this mailing list? Thanks in advance. *ArbolOne Using Fire Fox and Thunderbird. Developing for Android using Java, C/C++, HTM/CSS and SQLite as our platform has been exciting and most rewarding. [ Ñ ]* On Wed, Feb 7, 2024 at 5:50 AM Joseph Leonard < joseph.leon...@alfasystems.com> wrote: > Hi Tamás, > Yeah, this was unexpected to me initially as well. From what I can tell > the Maven reactor only considers direct dependencies (i.e. not transitive > dependencies) between the modules in the reactor when working out the build > graph. For example if you have a simple linear dependency chain of: > One --> Two --> Three --> Four --> Five > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will result in > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I assume > this is as designed because it actually offers quite powerful functionality > to improve the parallelism in your build. An example of where this is legit > is when: > > * “Four” has a test scope dependency on “Five” > * “One” has a test scoped dependency on “Two” > If you made a src code change to “Five” and “Two” then it would be safe to > build [One,Two] and [Four,Five] in parallel because you know the changes > within these graphs cannot impact each other. > Joe > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > Howdy, > > > > To me this looks like Maven is not aware that the App depends on > ModuleB... > > Are they "plain dependency" linked? Or what kind of dependency we talk > > about here? > > In short: why would App start while ModuleB (upstream dep) is not done? > > Something is fishy here. > > > > T > > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > joseph.leon...@alfasystems.com> wrote: > > > > > Hi all, > > > > > > It would be great to get any thoughts on whether the following is a > defect: > > > > > > > > > Issue details: > > > tl;dr > > > > > > Maven can resolve dependencies either from: > > > > > > * an external repo > > > * a class directory of a module being built within the reactor > > > * a packaged jar of a module being built within the reactor > > > > > > If you run a concurrent multi-module build it is possible to get a race > > > condition whereby the build of module Foo may resolve module Bar from > > > either of the three resolution channels. This inconsistency can result > in > > > the Maven war plugin sometimes failing to build a functional war file. > I > > > would expect a consistent resolution would always take place. > > > > > > Full details > > > Scenario > > > > > > Consider you have a repo with the following structure: > > > > > >App > > > > > > / \ > > > > > > / \ > > > > > >(compile scope) (test scope) > > > > > > / \ > > > > > > \/_ _\/ > > > > > > ModuleA TestSupportModule1 > > > > > > / > > > > > >/ > > > > > > (compile scope) > > > > > > / > > > > > >\/_ > > > > > > ModuleB > > > > > >/ > > > > > > / > > > > > > (test scope) > > > > > > / > > > > > > \/_ > > > > > > TestSupportModule2 > > > > > > If you were to make a src code change to the following test support > > > modules: > > > > > > * TestSupportModule1 > > > * TestSupportModule2 > > > > > > Then the minimum number of modules we need to build to verify the > change > > > set is OK is: > > > > > > * TestSupportModule1 > > > * TestSupportModule2 > > > * ModuleB > > > * App > > > > > > i.e. there is no requirement to build ModuleA because we know that > none of > > > the src code changes could impact the classpaths used in its maven > build. > > > > > > We know that despite 'App' depending (transitively) on ModuleB there > is no > > > need for the 'App' build to wait for ModuleB to complete its build > because > > > the src code change to TestSupportModule2 will not impact any of the > > > classpaths used in the App maven build. Therefore to get the most > efficient > > > build possible we ideally would invoke Maven to run with 2 threads and > with > > > instruction to build two distinct 'dependency graphs': > > > > > > * TestSupportModule1 followed by ModuleB > > > * TestSupportModule1 followed by App > > > > > > The following Maven command achieves exactly what we want because the > > > reactor build order is based only on the direct (i.e. non-transitive) > > > dependencies of the modules provided to the reactor in the build > command. > > > Therefore the absence of ModuleA results in two distinct 'dependency > > > graphs': > > > > > > mvn clean verify -pl TestSupportModule1,TestSupportModule2,ModuleB,App > -T 2 > > > > > > Note: In reality the code base I maintain has a very large monobuild > with > > > 100s of modules and this type of build optimisation makes a
Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Can you please try smart builder instead? https://github.com/takari/takari-smart-builder (note: smart builder is used by mvnd as well) The difference between the two can be seen here: http://takari.io/book/30-team-maven.html#takari-smart-builder On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < joseph.leon...@alfasystems.com> wrote: > Hi Tamás, > Yeah, this was unexpected to me initially as well. From what I can tell > the Maven reactor only considers direct dependencies (i.e. not transitive > dependencies) between the modules in the reactor when working out the build > graph. For example if you have a simple linear dependency chain of: > One --> Two --> Three --> Four --> Five > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will result in > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I assume > this is as designed because it actually offers quite powerful functionality > to improve the parallelism in your build. An example of where this is legit > is when: > > * “Four” has a test scope dependency on “Five” > * “One” has a test scoped dependency on “Two” > If you made a src code change to “Five” and “Two” then it would be safe to > build [One,Two] and [Four,Five] in parallel because you know the changes > within these graphs cannot impact each other. > Joe > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > Howdy, > > > > To me this looks like Maven is not aware that the App depends on > ModuleB... > > Are they "plain dependency" linked? Or what kind of dependency we talk > > about here? > > In short: why would App start while ModuleB (upstream dep) is not done? > > Something is fishy here. > > > > T > > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > joseph.leon...@alfasystems.com> wrote: > > > > > Hi all, > > > > > > It would be great to get any thoughts on whether the following is a > defect: > > > > > > > > > Issue details: > > > tl;dr > > > > > > Maven can resolve dependencies either from: > > > > > > * an external repo > > > * a class directory of a module being built within the reactor > > > * a packaged jar of a module being built within the reactor > > > > > > If you run a concurrent multi-module build it is possible to get a race > > > condition whereby the build of module Foo may resolve module Bar from > > > either of the three resolution channels. This inconsistency can result > in > > > the Maven war plugin sometimes failing to build a functional war file. > I > > > would expect a consistent resolution would always take place. > > > > > > Full details > > > Scenario > > > > > > Consider you have a repo with the following structure: > > > > > >App > > > > > > / \ > > > > > > / \ > > > > > >(compile scope) (test scope) > > > > > > / \ > > > > > > \/_ _\/ > > > > > > ModuleA TestSupportModule1 > > > > > > / > > > > > >/ > > > > > > (compile scope) > > > > > > / > > > > > >\/_ > > > > > > ModuleB > > > > > >/ > > > > > > / > > > > > > (test scope) > > > > > > / > > > > > > \/_ > > > > > > TestSupportModule2 > > > > > > If you were to make a src code change to the following test support > > > modules: > > > > > > * TestSupportModule1 > > > * TestSupportModule2 > > > > > > Then the minimum number of modules we need to build to verify the > change > > > set is OK is: > > > > > > * TestSupportModule1 > > > * TestSupportModule2 > > > * ModuleB > > > * App > > > > > > i.e. there is no requirement to build ModuleA because we know that > none of > > > the src code changes could impact the classpaths used in its maven > build. > > > > > > We know that despite 'App' depending (transitively) on ModuleB there > is no > > > need for the 'App' build to wait for ModuleB to complete its build > because > > > the src code change to TestSupportModule2 will not impact any of the > > > classpaths used in the App maven build. Therefore to get the most > efficient > > > build possible we ideally would invoke Maven to run with 2 threads and > with > > > instruction to build two distinct 'dependency graphs': > > > > > > * TestSupportModule1 followed by ModuleB > > > * TestSupportModule1 followed by App > > > > > > The following Maven command achieves exactly what we want because the > > > reactor build order is based only on the direct (i.e. non-transitive) > > > dependencies of the modules provided to the reactor in the build > command. > > > Therefore the absence of ModuleA results in two distinct 'dependency > > > graphs': > > > > > > mvn clean verify -pl TestSupportModule1,TestSupportModule2,ModuleB,App > -T 2 > > > > > > Note: In reality the code base I maintain has a very large monobuild > with > > > 100s of modules and this type of build
Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hi Jospeh, I didn’t necessarily expect that enabling the extension would solve/avoid the issue you described, but I mentioned it, because it would allow you to not have to specify an argument like '-pl TestSupportModule1,TestSupportModule2,ModuleB,App’ in the first place, because the Build Cache Extension should automatically determine which modules need to be built and for which ones previously cached artifacts can be used. Nils. > Op 6 feb 2024, om 15:11 heeft Joseph Leonard > het volgende geschreven: > > Thanks Nils, > maven-build-cache-extension looks very interesting generally – I will have a > play with it. > With regards to this issue, the maven-build-cache-extension overview > references “Subtree support for multimodule projects builds part of the > codebase in isolation” which sounded similar to the solution I am proposing > for this issue. Unfortunately, after enabling the maven-build-cache-extension > I still hit the same issue. > Joe > > On 2024/02/06 12:54:59 Nils Breunese wrote: >> I can’t comment on your question directly, but I just wanted to say that >> your use case sounds like it could benefit from the Maven Build Cache >> Extension (https://maven.apache.org/extensions/maven-build-cache-extension/). > >> >> Just my 2 cents. >> >> Nils. >> >>> Op 6 feb 2024 om 11:40 heeft Joseph Leonard het >>> volgende geschreven: >>> >>> Hi all, >>> >>> It would be great to get any thoughts on whether the following is a defect: >>> >>> >>> Issue details: >>> tl;dr >>> >>> Maven can resolve dependencies either from: >>> >>> * an external repo >>> * a class directory of a module being built within the reactor >>> * a packaged jar of a module being built within the reactor >>> >>> If you run a concurrent multi-module build it is possible to get a race >>> condition whereby the build of module Foo may resolve module Bar from >>> either of the three resolution channels. This inconsistency can result in >>> the Maven war plugin sometimes failing to build a functional war file. I >>> would expect a consistent resolution would always take place. > >>> >>> Full details >>> Scenario >>> >>> Consider you have a repo with the following structure: >>> >>> App >>> >>>/ \ >>> >>> / \ >>> >>> (compile scope) (test scope) >>> >>> / \ >>> >>> \/_ _\/ >>> >>>ModuleA TestSupportModule1 >>> >>> / >>> >>> / >>> >>> (compile scope) >>> >>>/ >>> >>> \/_ >>> >>> ModuleB >>> >>> / >>> >>> / >>> >>> (test scope) >>> >>> / >>> >>> \/_ >>> >>> TestSupportModule2 >>> >>> If you were to make a src code change to the following test support modules: >>> >>> * TestSupportModule1 >>> * TestSupportModule2 >>> >>> Then the minimum number of modules we need to build to verify the change >>> set is OK is: >>> >>> * TestSupportModule1 >>> * TestSupportModule2 >>> * ModuleB >>> * App >>> >>> i.e. there is no requirement to build ModuleA because we know that none of >>> the src code changes could impact the classpaths used in its maven build. > >>> >>> We know that despite 'App' depending (transitively) on ModuleB there is no >>> need for the 'App' build to wait for ModuleB to complete its build because >>> the src code change to TestSupportModule2 will not impact any of the >>> classpaths used in the App maven build. Therefore to get the most efficient >>> build possible we ideally would invoke Maven to run with 2 threads and with >>> instruction to build two distinct 'dependency graphs': > >>> >>> * TestSupportModule1 followed by ModuleB >>> * TestSupportModule1 followed by App >>> >>> The following Maven command achieves exactly what we want because the >>> reactor build order is based only on the direct (i.e. non-transitive) >>> dependencies of the modules provided to the reactor in the build command. >>> Therefore the absence of ModuleA results in two distinct 'dependency >>> graphs': > >>> >>> mvn clean verify -pl TestSupportModule1,TestSupportModule2,ModuleB,App -T 2 >>> >>> Note: In reality the code base I maintain has a very large monobuild with >>> 100s of modules and this type of build optimisation makes a significant >>> difference to the speed of our monobuild (we use >>> https://github.com/gitflow-incremental-builder/gitflow-incremental-builder >>> to automate the logic of determining which modules to include in the >>> reactor based on our change set). > >>> >>> Issue >>> >>> We have encountered an issue in the above scenario because the 'App' build >>> has a race condition with the ModuleB build which will result in one of the >>> following three outcomes: > >>> >>> * If the 'App' build starts before the ModuleB build has compiled its src >>> classes then the 'App' build will resolve
RE: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hi Tamás, Yeah, this was unexpected to me initially as well. From what I can tell the Maven reactor only considers direct dependencies (i.e. not transitive dependencies) between the modules in the reactor when working out the build graph. For example if you have a simple linear dependency chain of: One --> Two --> Three --> Four --> Five Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will result in two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I assume this is as designed because it actually offers quite powerful functionality to improve the parallelism in your build. An example of where this is legit is when: * “Four” has a test scope dependency on “Five” * “One” has a test scoped dependency on “Two” If you made a src code change to “Five” and “Two” then it would be safe to build [One,Two] and [Four,Five] in parallel because you know the changes within these graphs cannot impact each other. Joe On 2024/02/06 21:37:42 Tamás Cservenák wrote: > Howdy, > > To me this looks like Maven is not aware that the App depends on ModuleB... > Are they "plain dependency" linked? Or what kind of dependency we talk > about here? > In short: why would App start while ModuleB (upstream dep) is not done? > Something is fishy here. > > T > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > joseph.leon...@alfasystems.com> wrote: > > > Hi all, > > > > It would be great to get any thoughts on whether the following is a defect: > > > > > > Issue details: > > tl;dr > > > > Maven can resolve dependencies either from: > > > > * an external repo > > * a class directory of a module being built within the reactor > > * a packaged jar of a module being built within the reactor > > > > If you run a concurrent multi-module build it is possible to get a race > > condition whereby the build of module Foo may resolve module Bar from > > either of the three resolution channels. This inconsistency can result in > > the Maven war plugin sometimes failing to build a functional war file. I > > would expect a consistent resolution would always take place. > > > > Full details > > Scenario > > > > Consider you have a repo with the following structure: > > > >App > > > > / \ > > > > / \ > > > >(compile scope) (test scope) > > > > / \ > > > > \/_ _\/ > > > > ModuleA TestSupportModule1 > > > > / > > > >/ > > > > (compile scope) > > > > / > > > >\/_ > > > > ModuleB > > > >/ > > > > / > > > > (test scope) > > > > / > > > > \/_ > > > > TestSupportModule2 > > > > If you were to make a src code change to the following test support > > modules: > > > > * TestSupportModule1 > > * TestSupportModule2 > > > > Then the minimum number of modules we need to build to verify the change > > set is OK is: > > > > * TestSupportModule1 > > * TestSupportModule2 > > * ModuleB > > * App > > > > i.e. there is no requirement to build ModuleA because we know that none of > > the src code changes could impact the classpaths used in its maven build. > > > > We know that despite 'App' depending (transitively) on ModuleB there is no > > need for the 'App' build to wait for ModuleB to complete its build because > > the src code change to TestSupportModule2 will not impact any of the > > classpaths used in the App maven build. Therefore to get the most efficient > > build possible we ideally would invoke Maven to run with 2 threads and with > > instruction to build two distinct 'dependency graphs': > > > > * TestSupportModule1 followed by ModuleB > > * TestSupportModule1 followed by App > > > > The following Maven command achieves exactly what we want because the > > reactor build order is based only on the direct (i.e. non-transitive) > > dependencies of the modules provided to the reactor in the build command. > > Therefore the absence of ModuleA results in two distinct 'dependency > > graphs': > > > > mvn clean verify -pl TestSupportModule1,TestSupportModule2,ModuleB,App -T 2 > > > > Note: In reality the code base I maintain has a very large monobuild with > > 100s of modules and this type of build optimisation makes a significant > > difference to the speed of our monobuild (we use > > https://github.com/gitflow-incremental-builder/gitflow-incremental-builder > > to automate the logic of determining which modules to include in the > > reactor based on our change set). > > > > Issue > > > > We have encountered an issue in the above scenario because the 'App' build > > has a race condition with the ModuleB build which will result in one of the > > following three outcomes: > > > > * If the 'App' build starts before the ModuleB build has compiled its > > src classes then the 'App' build will resolve ModuleB from the
Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Howdy, To me this looks like Maven is not aware that the App depends on ModuleB... Are they "plain dependency" linked? Or what kind of dependency we talk about here? In short: why would App start while ModuleB (upstream dep) is not done? Something is fishy here. T On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < joseph.leon...@alfasystems.com> wrote: > Hi all, > > It would be great to get any thoughts on whether the following is a defect: > > > Issue details: > tl;dr > > Maven can resolve dependencies either from: > > * an external repo > * a class directory of a module being built within the reactor > * a packaged jar of a module being built within the reactor > > If you run a concurrent multi-module build it is possible to get a race > condition whereby the build of module Foo may resolve module Bar from > either of the three resolution channels. This inconsistency can result in > the Maven war plugin sometimes failing to build a functional war file. I > would expect a consistent resolution would always take place. > > Full details > Scenario > > Consider you have a repo with the following structure: > >App > > / \ > > / \ > >(compile scope) (test scope) > > / \ > > \/_ _\/ > > ModuleA TestSupportModule1 > > / > >/ > > (compile scope) > > / > >\/_ > > ModuleB > >/ > > / > > (test scope) > > / > > \/_ > > TestSupportModule2 > > If you were to make a src code change to the following test support > modules: > > * TestSupportModule1 > * TestSupportModule2 > > Then the minimum number of modules we need to build to verify the change > set is OK is: > > * TestSupportModule1 > * TestSupportModule2 > * ModuleB > * App > > i.e. there is no requirement to build ModuleA because we know that none of > the src code changes could impact the classpaths used in its maven build. > > We know that despite 'App' depending (transitively) on ModuleB there is no > need for the 'App' build to wait for ModuleB to complete its build because > the src code change to TestSupportModule2 will not impact any of the > classpaths used in the App maven build. Therefore to get the most efficient > build possible we ideally would invoke Maven to run with 2 threads and with > instruction to build two distinct 'dependency graphs': > > * TestSupportModule1 followed by ModuleB > * TestSupportModule1 followed by App > > The following Maven command achieves exactly what we want because the > reactor build order is based only on the direct (i.e. non-transitive) > dependencies of the modules provided to the reactor in the build command. > Therefore the absence of ModuleA results in two distinct 'dependency > graphs': > > mvn clean verify -pl TestSupportModule1,TestSupportModule2,ModuleB,App -T 2 > > Note: In reality the code base I maintain has a very large monobuild with > 100s of modules and this type of build optimisation makes a significant > difference to the speed of our monobuild (we use > https://github.com/gitflow-incremental-builder/gitflow-incremental-builder > to automate the logic of determining which modules to include in the > reactor based on our change set). > > Issue > > We have encountered an issue in the above scenario because the 'App' build > has a race condition with the ModuleB build which will result in one of the > following three outcomes: > > * If the 'App' build starts before the ModuleB build has compiled its > src classes then the 'App' build will resolve ModuleB from the external > repo (i.e. equivalent to ModuleB not being in the reactor at all) > * If the 'App' build starts after ModuleB has compiled its src classes > but before it has packaged these classes into a jar then the 'App' build > will resolve ModuleB's target/classes directory > * If the 'App' build starts after ModuleB has packaged its jar file > then the 'App' build will resolve ModuleB's target/ModuleB.jar file. > > In many scenarios this dependency resolution inconsistency doesn't > represent a challenge. However, it does cause an issue in our case because > the 'App' POM has its Maven packaging stanza configured to war and in the > scenario where ModuleB's target/classes directory is resolved by the 'App' > then this results in the resultant 'App' war file being packaged with a > completely empty ModuleB.jar file. > > Proposed solution > > Ideally we would like the Maven reactor to retain isolation between the > two distinct 'dependency graphs' it constructs at instantiation throughout > the entire Maven build. This would mean, in the simple example above, that > the 'App' would always resolves ModuleB from the external repo (regardless > of whether the reactor has built ModuleB or not in a separate 'dependency > graph' in
RE: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Note that my diagram formatting is very hard to read in the default formatting on this mailing list. If you view my email in 'raw source' then it is easier to understand! On 2024/02/06 10:40:14 Joseph Leonard wrote: > Hi all, > > It would be great to get any thoughts on whether the following is a defect: > > > Issue details: > tl;dr > > Maven can resolve dependencies either from: > > * an external repo > * a class directory of a module being built within the reactor > * a packaged jar of a module being built within the reactor > > If you run a concurrent multi-module build it is possible to get a race > condition whereby the build of module Foo may resolve module Bar from either > of the three resolution channels. This inconsistency can result in the Maven > war plugin sometimes failing to build a functional war file. I would expect a > consistent resolution would always take place. > > Full details > Scenario > > Consider you have a repo with the following structure: > >App > > / \ > > / \ > >(compile scope) (test scope) > > / \ > > \/_ _\/ > > ModuleA TestSupportModule1 > > / > >/ > > (compile scope) > > / > >\/_ > > ModuleB > >/ > > / > > (test scope) > > / > > \/_ > > TestSupportModule2 > > If you were to make a src code change to the following test support modules: > > * TestSupportModule1 > * TestSupportModule2 > > Then the minimum number of modules we need to build to verify the change set > is OK is: > > * TestSupportModule1 > * TestSupportModule2 > * ModuleB > * App > > i.e. there is no requirement to build ModuleA because we know that none of > the src code changes could impact the classpaths used in its maven build. > > We know that despite 'App' depending (transitively) on ModuleB there is no > need for the 'App' build to wait for ModuleB to complete its build because > the src code change to TestSupportModule2 will not impact any of the > classpaths used in the App maven build. Therefore to get the most efficient > build possible we ideally would invoke Maven to run with 2 threads and with > instruction to build two distinct 'dependency graphs': > > * TestSupportModule1 followed by ModuleB > * TestSupportModule1 followed by App > > The following Maven command achieves exactly what we want because the reactor > build order is based only on the direct (i.e. non-transitive) dependencies of > the modules provided to the reactor in the build command. Therefore the > absence of ModuleA results in two distinct 'dependency graphs': > > mvn clean verify -pl TestSupportModule1,TestSupportModule2,ModuleB,App -T 2 > > Note: In reality the code base I maintain has a very large monobuild with > 100s of modules and this type of build optimisation makes a significant > difference to the speed of our monobuild (we use > https://github.com/gitflow-incremental-builder/gitflow-incremental-builder to > automate the logic of determining which modules to include in the reactor > based on our change set). > > Issue > > We have encountered an issue in the above scenario because the 'App' build > has a race condition with the ModuleB build which will result in one of the > following three outcomes: > > * If the 'App' build starts before the ModuleB build has compiled its src > classes then the 'App' build will resolve ModuleB from the external repo > (i.e. equivalent to ModuleB not being in the reactor at all) > * If the 'App' build starts after ModuleB has compiled its src classes > but before it has packaged these classes into a jar then the 'App' build will > resolve ModuleB's target/classes directory > * If the 'App' build starts after ModuleB has packaged its jar file then > the 'App' build will resolve ModuleB's target/ModuleB.jar file. > > In many scenarios this dependency resolution inconsistency doesn't represent > a challenge. However, it does cause an issue in our case because the 'App' > POM has its Maven packaging stanza configured to war and in the scenario > where ModuleB's target/classes directory is resolved by the 'App' then this > results in the resultant 'App' war file being packaged with a completely > empty ModuleB.jar file. > > Proposed solution > > Ideally we would like the Maven reactor to retain isolation between the two > distinct 'dependency graphs' it constructs at instantiation throughout the > entire Maven build. This would mean, in the simple example above, that the > 'App' would always resolves ModuleB from the external repo (regardless of > whether the reactor has built ModuleB or not in a separate 'dependency graph' > in the reactor). > > > > Joseph Leonard > Manager > > Alfa > > e:
RE: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Thanks Nils, maven-build-cache-extension looks very interesting generally – I will have a play with it. With regards to this issue, the maven-build-cache-extension overview references “Subtree support for multimodule projects builds part of the codebase in isolation” which sounded similar to the solution I am proposing for this issue. Unfortunately, after enabling the maven-build-cache-extension I still hit the same issue. Joe On 2024/02/06 12:54:59 Nils Breunese wrote: > I can’t comment on your question directly, but I just wanted to say that your > use case sounds like it could benefit from the Maven Build Cache Extension > (https://maven.apache.org/extensions/maven-build-cache-extension/). > > Just my 2 cents. > > Nils. > > > Op 6 feb 2024 om 11:40 heeft Joseph Leonard het > > volgende geschreven: > > > > Hi all, > > > > It would be great to get any thoughts on whether the following is a defect: > > > > > > Issue details: > > tl;dr > > > > Maven can resolve dependencies either from: > > > > * an external repo > > * a class directory of a module being built within the reactor > > * a packaged jar of a module being built within the reactor > > > > If you run a concurrent multi-module build it is possible to get a race > > condition whereby the build of module Foo may resolve module Bar from > > either of the three resolution channels. This inconsistency can result in > > the Maven war plugin sometimes failing to build a functional war file. I > > would expect a consistent resolution would always take place. > > > > Full details > > Scenario > > > > Consider you have a repo with the following structure: > > > > App > > > > / \ > > > >/ \ > > > > (compile scope) (test scope) > > > > / \ > > > >\/_ _\/ > > > > ModuleA TestSupportModule1 > > > >/ > > > > / > > > >(compile scope) > > > > / > > > > \/_ > > > >ModuleB > > > > / > > > > / > > > >(test scope) > > > >/ > > > > \/_ > > > > TestSupportModule2 > > > > If you were to make a src code change to the following test support modules: > > > > * TestSupportModule1 > > * TestSupportModule2 > > > > Then the minimum number of modules we need to build to verify the change > > set is OK is: > > > > * TestSupportModule1 > > * TestSupportModule2 > > * ModuleB > > * App > > > > i.e. there is no requirement to build ModuleA because we know that none of > > the src code changes could impact the classpaths used in its maven build. > > > > We know that despite 'App' depending (transitively) on ModuleB there is no > > need for the 'App' build to wait for ModuleB to complete its build because > > the src code change to TestSupportModule2 will not impact any of the > > classpaths used in the App maven build. Therefore to get the most efficient > > build possible we ideally would invoke Maven to run with 2 threads and with > > instruction to build two distinct 'dependency graphs': > > > > * TestSupportModule1 followed by ModuleB > > * TestSupportModule1 followed by App > > > > The following Maven command achieves exactly what we want because the > > reactor build order is based only on the direct (i.e. non-transitive) > > dependencies of the modules provided to the reactor in the build command. > > Therefore the absence of ModuleA results in two distinct 'dependency > > graphs': > > > > mvn clean verify -pl TestSupportModule1,TestSupportModule2,ModuleB,App -T 2 > > > > Note: In reality the code base I maintain has a very large monobuild with > > 100s of modules and this type of build optimisation makes a significant > > difference to the speed of our monobuild (we use > > https://github.com/gitflow-incremental-builder/gitflow-incremental-builder > > to automate the logic of determining which modules to include in the > > reactor based on our change set). > > > > Issue > > > > We have encountered an issue in the above scenario because the 'App' build > > has a race condition with the ModuleB build which will result in one of the > > following three outcomes: > > > > * If the 'App' build starts before the ModuleB build has compiled its > > src classes then the 'App' build will resolve ModuleB from the external > > repo (i.e. equivalent to ModuleB not being in the reactor at all) > > * If the 'App' build starts after ModuleB has compiled its src classes > > but before it has packaged these classes into a jar then the 'App' build > > will resolve ModuleB's target/classes directory > > * If the 'App' build starts after ModuleB has packaged its jar file then > > the 'App' build will resolve ModuleB's target/ModuleB.jar file. > > > > In many scenarios this dependency resolution inconsistency doesn't > > represent a challenge. However, it does
Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
My two cents: a very large monobuild with 100s of modules These are the kinds of problems you find when you're abusing the multi-module system. Multi-module builds are a last resort for me, when there are couplings between artifacts that need to be released simultaneously, e.g. API and implementation artifacts. Do you have 100 artifacts that all depend on each other and need simultaneous release? That sounds like a tightly coupled mess. Keep your builds in independent projects instead of modules and avoid these issues.
Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
I can’t comment on your question directly, but I just wanted to say that your use case sounds like it could benefit from the Maven Build Cache Extension (https://maven.apache.org/extensions/maven-build-cache-extension/). Just my 2 cents. Nils. > Op 6 feb 2024 om 11:40 heeft Joseph Leonard > het volgende geschreven: > > Hi all, > > It would be great to get any thoughts on whether the following is a defect: > > > Issue details: > tl;dr > > Maven can resolve dependencies either from: > > * an external repo > * a class directory of a module being built within the reactor > * a packaged jar of a module being built within the reactor > > If you run a concurrent multi-module build it is possible to get a race > condition whereby the build of module Foo may resolve module Bar from either > of the three resolution channels. This inconsistency can result in the Maven > war plugin sometimes failing to build a functional war file. I would expect a > consistent resolution would always take place. > > Full details > Scenario > > Consider you have a repo with the following structure: > > App > > / \ > >/ \ > > (compile scope) (test scope) > > / \ > >\/_ _\/ > > ModuleA TestSupportModule1 > >/ > > / > >(compile scope) > > / > > \/_ > >ModuleB > > / > > / > >(test scope) > >/ > > \/_ > > TestSupportModule2 > > If you were to make a src code change to the following test support modules: > > * TestSupportModule1 > * TestSupportModule2 > > Then the minimum number of modules we need to build to verify the change set > is OK is: > > * TestSupportModule1 > * TestSupportModule2 > * ModuleB > * App > > i.e. there is no requirement to build ModuleA because we know that none of > the src code changes could impact the classpaths used in its maven build. > > We know that despite 'App' depending (transitively) on ModuleB there is no > need for the 'App' build to wait for ModuleB to complete its build because > the src code change to TestSupportModule2 will not impact any of the > classpaths used in the App maven build. Therefore to get the most efficient > build possible we ideally would invoke Maven to run with 2 threads and with > instruction to build two distinct 'dependency graphs': > > * TestSupportModule1 followed by ModuleB > * TestSupportModule1 followed by App > > The following Maven command achieves exactly what we want because the reactor > build order is based only on the direct (i.e. non-transitive) dependencies of > the modules provided to the reactor in the build command. Therefore the > absence of ModuleA results in two distinct 'dependency graphs': > > mvn clean verify -pl TestSupportModule1,TestSupportModule2,ModuleB,App -T 2 > > Note: In reality the code base I maintain has a very large monobuild with > 100s of modules and this type of build optimisation makes a significant > difference to the speed of our monobuild (we use > https://github.com/gitflow-incremental-builder/gitflow-incremental-builder to > automate the logic of determining which modules to include in the reactor > based on our change set). > > Issue > > We have encountered an issue in the above scenario because the 'App' build > has a race condition with the ModuleB build which will result in one of the > following three outcomes: > > * If the 'App' build starts before the ModuleB build has compiled its src > classes then the 'App' build will resolve ModuleB from the external repo > (i.e. equivalent to ModuleB not being in the reactor at all) > * If the 'App' build starts after ModuleB has compiled its src classes but > before it has packaged these classes into a jar then the 'App' build will > resolve ModuleB's target/classes directory > * If the 'App' build starts after ModuleB has packaged its jar file then > the 'App' build will resolve ModuleB's target/ModuleB.jar file. > > In many scenarios this dependency resolution inconsistency doesn't represent > a challenge. However, it does cause an issue in our case because the 'App' > POM has its Maven packaging stanza configured to war and in the scenario > where ModuleB's target/classes directory is resolved by the 'App' then this > results in the resultant 'App' war file being packaged with a completely > empty ModuleB.jar file. > > Proposed solution > > Ideally we would like the Maven reactor to retain isolation between the two > distinct 'dependency graphs' it constructs at instantiation throughout the > entire Maven build. This would mean, in the simple example above, that the > 'App' would always resolves ModuleB from the external repo (regardless of > whether the reactor has built ModuleB or not in a separate 'dependency
