Re: Can a plugin permit a dependency override by the user setting a simple option?
Jörg,
I do appreciate that you are trying to help me. But in order to do so, it would
be good to pay more attention before replying. Otherwise, you are wasting your
energy and limited time, both of which I also highly appreciate.
Again: There is no inheritance. That should have been clear from my very first
message, and I explained it again and again. I said *nothing* about parent and
child anywhere in this thread and even made it clear again in my last message.
That I need to repeat myself, trying to be more and more assertive about what I
already said or need to correct you about things I never said, is simply a
matter of trying to make you understand that what you think is my question is
really not. That then you do not like my tone, because we are both wasting
cycles, you trying to help me and me trying to help you stop chasing your tail,
is regrettable, but how can I help you if you just do not listen?
Again: There is a *single* project/module using a plugin. And I, as a plugin
author am asking literally here, quoting the very subject of the thread: "Can a
*plugin* permit a dependency override by the user setting a simple option?" No
parent/child, just a plugin and its user. The user wants to override a default
dependency version provided by a plugin, not by a parent. The term "override"
does not imply parent and child POMs, even if you insist it does. The module
using a plugin is not its child, it is just its user and wants to override the
plugin's default.
Sorry to the whole audience to repeat and paraphrase the same thing so many
times. I just want to make sure it sticks this time, so nobody in this
community wastes time trying to solve a problem I do not have.
--
Alexander Kriegisch
https://scrum-master.de
Jörg Schaible schrieb am 08.02.2024 05:28 (GMT +07:00):
> On Wednesday, 7. February 2024, 01:00:50 CET Alexander Kriegisch wrote:
>> 3rd party parent? Are you maybe mixing up my questions about different
>> topics? There is not parent POM involved here, it is about a plugin and an
>> application using it.
>
> You talk about "override by the user" which implies some kind of inheritance,
> at least
> for me.
>
>> Very simple and straightforward. I even gave an
>> example of what does not work for me but you say should.
>
> So strait forward:
>
>
> org.acme
> xy-compiler-plugin
> 1.2.3
>
>
> org.xy
> xy-compiler
> ${xy.compilerVersion}
>
>
>
>
> ... somewhere in a child ...
>
>
> 4.5
>
>
>> So does it?
>
> It does. If this is not, what you want, you will have to ask in a different
> way, since Lasse
> also understood your question the way I answered.
>
>> If you do not know, it is OK. But if you want to help, the ball
>> is in your court now. A counter-example showing me how to do what you
>> suggested right would be most welcome.
>
> Actually I don't like your tone and I get the impression that we waist each
> other's time.
>
> So I'll shut up,
> Jörg
>
>
-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org
Re: Can a plugin permit a dependency override by the user setting a simple option?
On Wednesday, 7. February 2024, 01:00:50 CET Alexander Kriegisch wrote:
> 3rd party parent? Are you maybe mixing up my questions about different
> topics? There is not parent POM involved here, it is about a plugin and an
> application using it.
You talk about "override by the user" which implies some kind of inheritance,
at least
for me.
> Very simple and straightforward. I even gave an
> example of what does not work for me but you say should.
So strait forward:
org.acme
xy-compiler-plugin
1.2.3
org.xy
xy-compiler
${xy.compilerVersion}
... somewhere in a child ...
4.5
> So does it?
It does. If this is not, what you want, you will have to ask in a different
way, since Lasse
also understood your question the way I answered.
> If you do not know, it is OK. But if you want to help, the ball
> is in your court now. A counter-example showing me how to do what you
> suggested right would be most welcome.
Actually I don't like your tone and I get the impression that we waist each
other's time.
So I'll shut up,
Jörg
Re: Re: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Howdy, Thank you very much, the reproducer works. Did not dig thru it fully, but here are some related issues: https://issues.apache.org/jira/browse/MNG-8028 (funny thing, I created this few weeks ago) https://issues.apache.org/jira/browse/MNG-6300 Will report back tomorrow (EU TZ) T On Wed, Feb 7, 2024 at 7:48 PM Joseph Leonard < joseph.leon...@alfasystems.com> wrote: > Hi Tamás, > I have created a simple example here: > https://github.com/josple/mvn-multibuild-issue > Hopefully the README is clear enough – let me know if I can clarify > anything. > Thanks, > Joe > > On 2024/02/07 17:33:08 Tamás Cservenák wrote: > > Howdy, > > > > In that case, there is something fishy with the project, my blind guess > > would be some "hidden" inter-module dependency maybe? > > > > Can you provide access to source, or, if not feasible, could you provide > > some reproducer and publish it on Github/Gitlab/whatever (maybe even just > > send it privately as ML strips off attachments and images) for us to see > > this in action? > > > > Thanks > > T > > > > On Wed, Feb 7, 2024 at 6:29 PM Joseph Leonard < > > joseph.leon...@alfasystems.com> wrote: > > > > > Hi Tamás, > > > We have previously played around a bit with mvnd but not takari > directly – > > > I will have a play with it. With regards to this issue, using the > takari > > > smart builder unfortunately doesn’t resolve the issue. > > > Joe > > > > > > On 2024/02/07 11:41:22 Tamás Cservenák wrote: > > > > Can you please try smart builder instead? > > > > https://github.com/takari/takari-smart-builder > > > > > > > > (note: smart builder is used by mvnd as well) > > > > > > > > The difference between the two can be seen here: > > > > http://takari.io/book/30-team-maven.html#takari-smart-builder > > > > > > > > On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < > > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > > > Hi Tamás, > > > > > Yeah, this was unexpected to me initially as well. From what I can > tell > > > > > the Maven reactor only considers direct dependencies (i.e. not > > > transitive > > > > > dependencies) between the modules in the reactor when working out > the > > > build > > > > > graph. For example if you have a simple linear dependency chain of: > > > > > One --> Two --> Three --> Four --> Five > > > > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will > result > > > in > > > > > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I > > > assume > > > > > this is as designed because it actually offers quite powerful > > > functionality > > > > > to improve the parallelism in your build. An example of where this > is > > > legit > > > > > is when: > > > > > > > > > > * “Four” has a test scope dependency on “Five” > > > > > * “One” has a test scoped dependency on “Two” > > > > > If you made a src code change to “Five” and “Two” then it would be > > > safe to > > > > > build [One,Two] and [Four,Five] in parallel because you know the > > > changes > > > > > within these graphs cannot impact each other. > > > > > Joe > > > > > > > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > > > > > Howdy, > > > > > > > > > > > > To me this looks like Maven is not aware that the App depends on > > > > > ModuleB... > > > > > > Are they "plain dependency" linked? Or what kind of dependency we > > > talk > > > > > > about here? > > > > > > In short: why would App start while ModuleB (upstream dep) is not > > > done? > > > > > > Something is fishy here. > > > > > > > > > > > > T > > > > > > > > > > > > > > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > > > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > It would be great to get any thoughts on whether the following > is a > > > > > defect: > > > > > > > > > > > > > > > > > > > > > Issue details: > > > > > > > tl;dr > > > > > > > > > > > > > > Maven can resolve dependencies either from: > > > > > > > > > > > > > > * an external repo > > > > > > > * a class directory of a module being built within the > reactor > > > > > > > * a packaged jar of a module being built within the reactor > > > > > > > > > > > > > > If you run a concurrent multi-module build it is possible to > get a > > > race > > > > > > > condition whereby the build of module Foo may resolve module > Bar > > > from > > > > > > > either of the three resolution channels. This inconsistency can > > > result > > > > > in > > > > > > > the Maven war plugin sometimes failing to build a functional > war > > > file. > > > > > I > > > > > > > would expect a consistent resolution would always take place. > > > > > > > > > > > > > > Full details > > > > > > > Scenario > > > > > > > > > > > > > > Consider you have a repo with the following structure: > > > > > > > > > > > > > >App > > > > > > > > > > > > > > / \ > > > > > > > > > > > > > > / \ > > > > > > > >
RE: Re: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hi Tamás, I have created a simple example here: https://github.com/josple/mvn-multibuild-issue Hopefully the README is clear enough – let me know if I can clarify anything. Thanks, Joe On 2024/02/07 17:33:08 Tamás Cservenák wrote: > Howdy, > > In that case, there is something fishy with the project, my blind guess > would be some "hidden" inter-module dependency maybe? > > Can you provide access to source, or, if not feasible, could you provide > some reproducer and publish it on Github/Gitlab/whatever (maybe even just > send it privately as ML strips off attachments and images) for us to see > this in action? > > Thanks > T > > On Wed, Feb 7, 2024 at 6:29 PM Joseph Leonard < > joseph.leon...@alfasystems.com> wrote: > > > Hi Tamás, > > We have previously played around a bit with mvnd but not takari directly – > > I will have a play with it. With regards to this issue, using the takari > > smart builder unfortunately doesn’t resolve the issue. > > Joe > > > > On 2024/02/07 11:41:22 Tamás Cservenák wrote: > > > Can you please try smart builder instead? > > > https://github.com/takari/takari-smart-builder > > > > > > (note: smart builder is used by mvnd as well) > > > > > > The difference between the two can be seen here: > > > http://takari.io/book/30-team-maven.html#takari-smart-builder > > > > > > On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > Hi Tamás, > > > > Yeah, this was unexpected to me initially as well. From what I can tell > > > > the Maven reactor only considers direct dependencies (i.e. not > > transitive > > > > dependencies) between the modules in the reactor when working out the > > build > > > > graph. For example if you have a simple linear dependency chain of: > > > > One --> Two --> Three --> Four --> Five > > > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will result > > in > > > > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I > > assume > > > > this is as designed because it actually offers quite powerful > > functionality > > > > to improve the parallelism in your build. An example of where this is > > legit > > > > is when: > > > > > > > > * “Four” has a test scope dependency on “Five” > > > > * “One” has a test scoped dependency on “Two” > > > > If you made a src code change to “Five” and “Two” then it would be > > safe to > > > > build [One,Two] and [Four,Five] in parallel because you know the > > changes > > > > within these graphs cannot impact each other. > > > > Joe > > > > > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > > > > Howdy, > > > > > > > > > > To me this looks like Maven is not aware that the App depends on > > > > ModuleB... > > > > > Are they "plain dependency" linked? Or what kind of dependency we > > talk > > > > > about here? > > > > > In short: why would App start while ModuleB (upstream dep) is not > > done? > > > > > Something is fishy here. > > > > > > > > > > T > > > > > > > > > > > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > It would be great to get any thoughts on whether the following is a > > > > defect: > > > > > > > > > > > > > > > > > > Issue details: > > > > > > tl;dr > > > > > > > > > > > > Maven can resolve dependencies either from: > > > > > > > > > > > > * an external repo > > > > > > * a class directory of a module being built within the reactor > > > > > > * a packaged jar of a module being built within the reactor > > > > > > > > > > > > If you run a concurrent multi-module build it is possible to get a > > race > > > > > > condition whereby the build of module Foo may resolve module Bar > > from > > > > > > either of the three resolution channels. This inconsistency can > > result > > > > in > > > > > > the Maven war plugin sometimes failing to build a functional war > > file. > > > > I > > > > > > would expect a consistent resolution would always take place. > > > > > > > > > > > > Full details > > > > > > Scenario > > > > > > > > > > > > Consider you have a repo with the following structure: > > > > > > > > > > > >App > > > > > > > > > > > > / \ > > > > > > > > > > > > / \ > > > > > > > > > > > >(compile scope) (test scope) > > > > > > > > > > > > / \ > > > > > > > > > > > > \/_ _\/ > > > > > > > > > > > > ModuleA TestSupportModule1 > > > > > > > > > > > > / > > > > > > > > > > > >/ > > > > > > > > > > > > (compile scope) > > > > > > > > > > > > / > > > > > > > > > > > >\/_ > > > > > > > > > > > > ModuleB > > > > > > > > > > > >/ > > > > > > > > > > > > / > > > > > > > > > > > > (test scope) > > > > > > > > > > > > / > > > > > > > > > > > >
Re: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Howdy, In that case, there is something fishy with the project, my blind guess would be some "hidden" inter-module dependency maybe? Can you provide access to source, or, if not feasible, could you provide some reproducer and publish it on Github/Gitlab/whatever (maybe even just send it privately as ML strips off attachments and images) for us to see this in action? Thanks T On Wed, Feb 7, 2024 at 6:29 PM Joseph Leonard < joseph.leon...@alfasystems.com> wrote: > Hi Tamás, > We have previously played around a bit with mvnd but not takari directly – > I will have a play with it. With regards to this issue, using the takari > smart builder unfortunately doesn’t resolve the issue. > Joe > > On 2024/02/07 11:41:22 Tamás Cservenák wrote: > > Can you please try smart builder instead? > > https://github.com/takari/takari-smart-builder > > > > (note: smart builder is used by mvnd as well) > > > > The difference between the two can be seen here: > > http://takari.io/book/30-team-maven.html#takari-smart-builder > > > > On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < > > joseph.leon...@alfasystems.com> wrote: > > > > > Hi Tamás, > > > Yeah, this was unexpected to me initially as well. From what I can tell > > > the Maven reactor only considers direct dependencies (i.e. not > transitive > > > dependencies) between the modules in the reactor when working out the > build > > > graph. For example if you have a simple linear dependency chain of: > > > One --> Two --> Three --> Four --> Five > > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will result > in > > > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I > assume > > > this is as designed because it actually offers quite powerful > functionality > > > to improve the parallelism in your build. An example of where this is > legit > > > is when: > > > > > > * “Four” has a test scope dependency on “Five” > > > * “One” has a test scoped dependency on “Two” > > > If you made a src code change to “Five” and “Two” then it would be > safe to > > > build [One,Two] and [Four,Five] in parallel because you know the > changes > > > within these graphs cannot impact each other. > > > Joe > > > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > > > Howdy, > > > > > > > > To me this looks like Maven is not aware that the App depends on > > > ModuleB... > > > > Are they "plain dependency" linked? Or what kind of dependency we > talk > > > > about here? > > > > In short: why would App start while ModuleB (upstream dep) is not > done? > > > > Something is fishy here. > > > > > > > > T > > > > > > > > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > > > Hi all, > > > > > > > > > > It would be great to get any thoughts on whether the following is a > > > defect: > > > > > > > > > > > > > > > Issue details: > > > > > tl;dr > > > > > > > > > > Maven can resolve dependencies either from: > > > > > > > > > > * an external repo > > > > > * a class directory of a module being built within the reactor > > > > > * a packaged jar of a module being built within the reactor > > > > > > > > > > If you run a concurrent multi-module build it is possible to get a > race > > > > > condition whereby the build of module Foo may resolve module Bar > from > > > > > either of the three resolution channels. This inconsistency can > result > > > in > > > > > the Maven war plugin sometimes failing to build a functional war > file. > > > I > > > > > would expect a consistent resolution would always take place. > > > > > > > > > > Full details > > > > > Scenario > > > > > > > > > > Consider you have a repo with the following structure: > > > > > > > > > >App > > > > > > > > > > / \ > > > > > > > > > > / \ > > > > > > > > > >(compile scope) (test scope) > > > > > > > > > > / \ > > > > > > > > > > \/_ _\/ > > > > > > > > > > ModuleA TestSupportModule1 > > > > > > > > > > / > > > > > > > > > >/ > > > > > > > > > > (compile scope) > > > > > > > > > > / > > > > > > > > > >\/_ > > > > > > > > > > ModuleB > > > > > > > > > >/ > > > > > > > > > > / > > > > > > > > > > (test scope) > > > > > > > > > > / > > > > > > > > > > \/_ > > > > > > > > > > TestSupportModule2 > > > > > > > > > > If you were to make a src code change to the following test support > > > > > modules: > > > > > > > > > > * TestSupportModule1 > > > > > * TestSupportModule2 > > > > > > > > > > Then the minimum number of modules we need to build to verify the > > > change > > > > > set is OK is: > > > > > > > > > > * TestSupportModule1 > > > > > * TestSupportModule2 > > > > > * ModuleB > > > > > * App > > > > > > > > > > i.e. there is no
RE: Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hi Tamás, We have previously played around a bit with mvnd but not takari directly – I will have a play with it. With regards to this issue, using the takari smart builder unfortunately doesn’t resolve the issue. Joe On 2024/02/07 11:41:22 Tamás Cservenák wrote: > Can you please try smart builder instead? > https://github.com/takari/takari-smart-builder > > (note: smart builder is used by mvnd as well) > > The difference between the two can be seen here: > http://takari.io/book/30-team-maven.html#takari-smart-builder > > On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < > joseph.leon...@alfasystems.com> wrote: > > > Hi Tamás, > > Yeah, this was unexpected to me initially as well. From what I can tell > > the Maven reactor only considers direct dependencies (i.e. not transitive > > dependencies) between the modules in the reactor when working out the build > > graph. For example if you have a simple linear dependency chain of: > > One --> Two --> Three --> Four --> Five > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will result in > > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I assume > > this is as designed because it actually offers quite powerful functionality > > to improve the parallelism in your build. An example of where this is legit > > is when: > > > > * “Four” has a test scope dependency on “Five” > > * “One” has a test scoped dependency on “Two” > > If you made a src code change to “Five” and “Two” then it would be safe to > > build [One,Two] and [Four,Five] in parallel because you know the changes > > within these graphs cannot impact each other. > > Joe > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > > Howdy, > > > > > > To me this looks like Maven is not aware that the App depends on > > ModuleB... > > > Are they "plain dependency" linked? Or what kind of dependency we talk > > > about here? > > > In short: why would App start while ModuleB (upstream dep) is not done? > > > Something is fishy here. > > > > > > T > > > > > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > Hi all, > > > > > > > > It would be great to get any thoughts on whether the following is a > > defect: > > > > > > > > > > > > Issue details: > > > > tl;dr > > > > > > > > Maven can resolve dependencies either from: > > > > > > > > * an external repo > > > > * a class directory of a module being built within the reactor > > > > * a packaged jar of a module being built within the reactor > > > > > > > > If you run a concurrent multi-module build it is possible to get a race > > > > condition whereby the build of module Foo may resolve module Bar from > > > > either of the three resolution channels. This inconsistency can result > > in > > > > the Maven war plugin sometimes failing to build a functional war file. > > I > > > > would expect a consistent resolution would always take place. > > > > > > > > Full details > > > > Scenario > > > > > > > > Consider you have a repo with the following structure: > > > > > > > >App > > > > > > > > / \ > > > > > > > > / \ > > > > > > > >(compile scope) (test scope) > > > > > > > > / \ > > > > > > > > \/_ _\/ > > > > > > > > ModuleA TestSupportModule1 > > > > > > > > / > > > > > > > >/ > > > > > > > > (compile scope) > > > > > > > > / > > > > > > > >\/_ > > > > > > > > ModuleB > > > > > > > >/ > > > > > > > > / > > > > > > > > (test scope) > > > > > > > > / > > > > > > > > \/_ > > > > > > > > TestSupportModule2 > > > > > > > > If you were to make a src code change to the following test support > > > > modules: > > > > > > > > * TestSupportModule1 > > > > * TestSupportModule2 > > > > > > > > Then the minimum number of modules we need to build to verify the > > change > > > > set is OK is: > > > > > > > > * TestSupportModule1 > > > > * TestSupportModule2 > > > > * ModuleB > > > > * App > > > > > > > > i.e. there is no requirement to build ModuleA because we know that > > none of > > > > the src code changes could impact the classpaths used in its maven > > build. > > > > > > > > We know that despite 'App' depending (transitively) on ModuleB there > > is no > > > > need for the 'App' build to wait for ModuleB to complete its build > > because > > > > the src code change to TestSupportModule2 will not impact any of the > > > > classpaths used in the App maven build. Therefore to get the most > > efficient > > > > build possible we ideally would invoke Maven to run with 2 threads and > > with > > > > instruction to build two distinct 'dependency graphs': > > > > > > > > * TestSupportModule1 followed by ModuleB > > > > * TestSupportModule1 followed by App > > > >
RE: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hi Nils, Ah I see. We actually already use ‘-pl …’ arguments in conjunction with the Develocity Build Cache implementation because we have found this to still generate very significant time savings. Primarily because even with hits on the build cache there is still fairly substantial time spent resolving cached artifacts (and the reactor doing all its goal steps and determining cache hits). The other advantage of using ‘-pl …’ is that it gives opportunity for even greater parallelism (per my sub-thread example with Tamás). Joe On 2024/02/07 10:59:28 Nils Breunese wrote: > Hi Jospeh, > > I didn’t necessarily expect that enabling the extension would solve/avoid the > issue you described, but I mentioned it, because it would allow you to not > have to specify an argument like '-pl > TestSupportModule1,TestSupportModule2,ModuleB,App’ in the first place, > because the Build Cache Extension should automatically determine which > modules need to be built and for which ones previously cached artifacts can > be used. > > Nils. > > > Op 6 feb 2024, om 15:11 heeft Joseph Leonard het > > volgende geschreven: > > > > Thanks Nils, > > maven-build-cache-extension looks very interesting generally – I will have > > a play with it. > > With regards to this issue, the maven-build-cache-extension overview > > references “Subtree support for multimodule projects builds part of the > > codebase in isolation” which sounded similar to the solution I am proposing > > for this issue. Unfortunately, after enabling the > > maven-build-cache-extension I still hit the same issue. > > Joe > > > > On 2024/02/06 12:54:59 Nils Breunese wrote: > >> I can’t comment on your question directly, but I just wanted to say that > >> your use case sounds like it could benefit from the Maven Build Cache > >> Extension > >> (https://maven.apache.org/extensions/maven-build-cache-extension/). > > > >> > >> Just my 2 cents. > >> > >> Nils. > >> > >>> Op 6 feb 2024 om 11:40 heeft Joseph Leonard het > >>> volgende geschreven: > >>> > >>> Hi all, > >>> > >>> It would be great to get any thoughts on whether the following is a > >>> defect: > >>> > >>> > >>> Issue details: > >>> tl;dr > >>> > >>> Maven can resolve dependencies either from: > >>> > >>> * an external repo > >>> * a class directory of a module being built within the reactor > >>> * a packaged jar of a module being built within the reactor > >>> > >>> If you run a concurrent multi-module build it is possible to get a race > >>> condition whereby the build of module Foo may resolve module Bar from > >>> either of the three resolution channels. This inconsistency can result in > >>> the Maven war plugin sometimes failing to build a functional war file. I > >>> would expect a consistent resolution would always take place. > > > >>> > >>> Full details > >>> Scenario > >>> > >>> Consider you have a repo with the following structure: > >>> > >>> App > >>> > >>>/ \ > >>> > >>> / \ > >>> > >>> (compile scope) (test scope) > >>> > >>> / \ > >>> > >>> \/_ _\/ > >>> > >>>ModuleA TestSupportModule1 > >>> > >>> / > >>> > >>> / > >>> > >>> (compile scope) > >>> > >>>/ > >>> > >>> \/_ > >>> > >>> ModuleB > >>> > >>> / > >>> > >>> / > >>> > >>> (test scope) > >>> > >>> / > >>> > >>> \/_ > >>> > >>> TestSupportModule2 > >>> > >>> If you were to make a src code change to the following test support > >>> modules: > >>> > >>> * TestSupportModule1 > >>> * TestSupportModule2 > >>> > >>> Then the minimum number of modules we need to build to verify the change > >>> set is OK is: > >>> > >>> * TestSupportModule1 > >>> * TestSupportModule2 > >>> * ModuleB > >>> * App > >>> > >>> i.e. there is no requirement to build ModuleA because we know that none > >>> of the src code changes could impact the classpaths used in its maven > >>> build. > > > >>> > >>> We know that despite 'App' depending (transitively) on ModuleB there is > >>> no need for the 'App' build to wait for ModuleB to complete its build > >>> because the src code change to TestSupportModule2 will not impact any of > >>> the classpaths used in the App maven build. Therefore to get the most > >>> efficient build possible we ideally would invoke Maven to run with 2 > >>> threads and with instruction to build two distinct 'dependency graphs': > > > >>> > >>> * TestSupportModule1 followed by ModuleB > >>> * TestSupportModule1 followed by App > >>> > >>> The following Maven command achieves exactly what we want because the > >>> reactor build order is based only on the direct (i.e. non-transitive) > >>> dependencies of the modules provided to the reactor in the build command. > >>> Therefore the absence of ModuleA results in two distinct 'dependency > >>> graphs': >
Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Howdy, see here https://maven.apache.org/mailing-lists.html On Wed, Feb 7, 2024 at 12:55 PM Amn Ojee Uw wrote: > How can I unsubscribe from this mailing list? > Thanks in advance. > > *ArbolOne > Using Fire Fox and Thunderbird. > Developing for Android using Java, C/C++, HTM/CSS and SQLite as our > platform has been exciting and most rewarding. > [ Ñ ]* > > > > On Wed, Feb 7, 2024 at 5:50 AM Joseph Leonard < > joseph.leon...@alfasystems.com> wrote: > > > Hi Tamás, > > Yeah, this was unexpected to me initially as well. From what I can tell > > the Maven reactor only considers direct dependencies (i.e. not transitive > > dependencies) between the modules in the reactor when working out the > build > > graph. For example if you have a simple linear dependency chain of: > > One --> Two --> Three --> Four --> Five > > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will result in > > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I > assume > > this is as designed because it actually offers quite powerful > functionality > > to improve the parallelism in your build. An example of where this is > legit > > is when: > > > > * “Four” has a test scope dependency on “Five” > > * “One” has a test scoped dependency on “Two” > > If you made a src code change to “Five” and “Two” then it would be safe > to > > build [One,Two] and [Four,Five] in parallel because you know the changes > > within these graphs cannot impact each other. > > Joe > > > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > > Howdy, > > > > > > To me this looks like Maven is not aware that the App depends on > > ModuleB... > > > Are they "plain dependency" linked? Or what kind of dependency we talk > > > about here? > > > In short: why would App start while ModuleB (upstream dep) is not done? > > > Something is fishy here. > > > > > > T > > > > > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > > joseph.leon...@alfasystems.com> wrote: > > > > > > > Hi all, > > > > > > > > It would be great to get any thoughts on whether the following is a > > defect: > > > > > > > > > > > > Issue details: > > > > tl;dr > > > > > > > > Maven can resolve dependencies either from: > > > > > > > > * an external repo > > > > * a class directory of a module being built within the reactor > > > > * a packaged jar of a module being built within the reactor > > > > > > > > If you run a concurrent multi-module build it is possible to get a > race > > > > condition whereby the build of module Foo may resolve module Bar from > > > > either of the three resolution channels. This inconsistency can > result > > in > > > > the Maven war plugin sometimes failing to build a functional war > file. > > I > > > > would expect a consistent resolution would always take place. > > > > > > > > Full details > > > > Scenario > > > > > > > > Consider you have a repo with the following structure: > > > > > > > >App > > > > > > > > / \ > > > > > > > > / \ > > > > > > > >(compile scope) (test scope) > > > > > > > > / \ > > > > > > > > \/_ _\/ > > > > > > > > ModuleA TestSupportModule1 > > > > > > > > / > > > > > > > >/ > > > > > > > > (compile scope) > > > > > > > > / > > > > > > > >\/_ > > > > > > > > ModuleB > > > > > > > >/ > > > > > > > > / > > > > > > > > (test scope) > > > > > > > > / > > > > > > > > \/_ > > > > > > > > TestSupportModule2 > > > > > > > > If you were to make a src code change to the following test support > > > > modules: > > > > > > > > * TestSupportModule1 > > > > * TestSupportModule2 > > > > > > > > Then the minimum number of modules we need to build to verify the > > change > > > > set is OK is: > > > > > > > > * TestSupportModule1 > > > > * TestSupportModule2 > > > > * ModuleB > > > > * App > > > > > > > > i.e. there is no requirement to build ModuleA because we know that > > none of > > > > the src code changes could impact the classpaths used in its maven > > build. > > > > > > > > We know that despite 'App' depending (transitively) on ModuleB there > > is no > > > > need for the 'App' build to wait for ModuleB to complete its build > > because > > > > the src code change to TestSupportModule2 will not impact any of the > > > > classpaths used in the App maven build. Therefore to get the most > > efficient > > > > build possible we ideally would invoke Maven to run with 2 threads > and > > with > > > > instruction to build two distinct 'dependency graphs': > > > > > > > > * TestSupportModule1 followed by ModuleB > > > > * TestSupportModule1 followed by App > > > > > > > > The following Maven command achieves exactly what we want because the > > > > reactor build order is based only on the direct (i.e.
Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
How can I unsubscribe from this mailing list? Thanks in advance. *ArbolOne Using Fire Fox and Thunderbird. Developing for Android using Java, C/C++, HTM/CSS and SQLite as our platform has been exciting and most rewarding. [ Ñ ]* On Wed, Feb 7, 2024 at 5:50 AM Joseph Leonard < joseph.leon...@alfasystems.com> wrote: > Hi Tamás, > Yeah, this was unexpected to me initially as well. From what I can tell > the Maven reactor only considers direct dependencies (i.e. not transitive > dependencies) between the modules in the reactor when working out the build > graph. For example if you have a simple linear dependency chain of: > One --> Two --> Three --> Four --> Five > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will result in > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I assume > this is as designed because it actually offers quite powerful functionality > to improve the parallelism in your build. An example of where this is legit > is when: > > * “Four” has a test scope dependency on “Five” > * “One” has a test scoped dependency on “Two” > If you made a src code change to “Five” and “Two” then it would be safe to > build [One,Two] and [Four,Five] in parallel because you know the changes > within these graphs cannot impact each other. > Joe > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > Howdy, > > > > To me this looks like Maven is not aware that the App depends on > ModuleB... > > Are they "plain dependency" linked? Or what kind of dependency we talk > > about here? > > In short: why would App start while ModuleB (upstream dep) is not done? > > Something is fishy here. > > > > T > > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > joseph.leon...@alfasystems.com> wrote: > > > > > Hi all, > > > > > > It would be great to get any thoughts on whether the following is a > defect: > > > > > > > > > Issue details: > > > tl;dr > > > > > > Maven can resolve dependencies either from: > > > > > > * an external repo > > > * a class directory of a module being built within the reactor > > > * a packaged jar of a module being built within the reactor > > > > > > If you run a concurrent multi-module build it is possible to get a race > > > condition whereby the build of module Foo may resolve module Bar from > > > either of the three resolution channels. This inconsistency can result > in > > > the Maven war plugin sometimes failing to build a functional war file. > I > > > would expect a consistent resolution would always take place. > > > > > > Full details > > > Scenario > > > > > > Consider you have a repo with the following structure: > > > > > >App > > > > > > / \ > > > > > > / \ > > > > > >(compile scope) (test scope) > > > > > > / \ > > > > > > \/_ _\/ > > > > > > ModuleA TestSupportModule1 > > > > > > / > > > > > >/ > > > > > > (compile scope) > > > > > > / > > > > > >\/_ > > > > > > ModuleB > > > > > >/ > > > > > > / > > > > > > (test scope) > > > > > > / > > > > > > \/_ > > > > > > TestSupportModule2 > > > > > > If you were to make a src code change to the following test support > > > modules: > > > > > > * TestSupportModule1 > > > * TestSupportModule2 > > > > > > Then the minimum number of modules we need to build to verify the > change > > > set is OK is: > > > > > > * TestSupportModule1 > > > * TestSupportModule2 > > > * ModuleB > > > * App > > > > > > i.e. there is no requirement to build ModuleA because we know that > none of > > > the src code changes could impact the classpaths used in its maven > build. > > > > > > We know that despite 'App' depending (transitively) on ModuleB there > is no > > > need for the 'App' build to wait for ModuleB to complete its build > because > > > the src code change to TestSupportModule2 will not impact any of the > > > classpaths used in the App maven build. Therefore to get the most > efficient > > > build possible we ideally would invoke Maven to run with 2 threads and > with > > > instruction to build two distinct 'dependency graphs': > > > > > > * TestSupportModule1 followed by ModuleB > > > * TestSupportModule1 followed by App > > > > > > The following Maven command achieves exactly what we want because the > > > reactor build order is based only on the direct (i.e. non-transitive) > > > dependencies of the modules provided to the reactor in the build > command. > > > Therefore the absence of ModuleA results in two distinct 'dependency > > > graphs': > > > > > > mvn clean verify -pl TestSupportModule1,TestSupportModule2,ModuleB,App > -T 2 > > > > > > Note: In reality the code base I maintain has a very large monobuild > with > > > 100s of modules and this type of build optimisation makes a
Re: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Can you please try smart builder instead? https://github.com/takari/takari-smart-builder (note: smart builder is used by mvnd as well) The difference between the two can be seen here: http://takari.io/book/30-team-maven.html#takari-smart-builder On Wed, Feb 7, 2024 at 11:50 AM Joseph Leonard < joseph.leon...@alfasystems.com> wrote: > Hi Tamás, > Yeah, this was unexpected to me initially as well. From what I can tell > the Maven reactor only considers direct dependencies (i.e. not transitive > dependencies) between the modules in the reactor when working out the build > graph. For example if you have a simple linear dependency chain of: > One --> Two --> Three --> Four --> Five > Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will result in > two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I assume > this is as designed because it actually offers quite powerful functionality > to improve the parallelism in your build. An example of where this is legit > is when: > > * “Four” has a test scope dependency on “Five” > * “One” has a test scoped dependency on “Two” > If you made a src code change to “Five” and “Two” then it would be safe to > build [One,Two] and [Four,Five] in parallel because you know the changes > within these graphs cannot impact each other. > Joe > > On 2024/02/06 21:37:42 Tamás Cservenák wrote: > > Howdy, > > > > To me this looks like Maven is not aware that the App depends on > ModuleB... > > Are they "plain dependency" linked? Or what kind of dependency we talk > > about here? > > In short: why would App start while ModuleB (upstream dep) is not done? > > Something is fishy here. > > > > T > > > > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > > joseph.leon...@alfasystems.com> wrote: > > > > > Hi all, > > > > > > It would be great to get any thoughts on whether the following is a > defect: > > > > > > > > > Issue details: > > > tl;dr > > > > > > Maven can resolve dependencies either from: > > > > > > * an external repo > > > * a class directory of a module being built within the reactor > > > * a packaged jar of a module being built within the reactor > > > > > > If you run a concurrent multi-module build it is possible to get a race > > > condition whereby the build of module Foo may resolve module Bar from > > > either of the three resolution channels. This inconsistency can result > in > > > the Maven war plugin sometimes failing to build a functional war file. > I > > > would expect a consistent resolution would always take place. > > > > > > Full details > > > Scenario > > > > > > Consider you have a repo with the following structure: > > > > > >App > > > > > > / \ > > > > > > / \ > > > > > >(compile scope) (test scope) > > > > > > / \ > > > > > > \/_ _\/ > > > > > > ModuleA TestSupportModule1 > > > > > > / > > > > > >/ > > > > > > (compile scope) > > > > > > / > > > > > >\/_ > > > > > > ModuleB > > > > > >/ > > > > > > / > > > > > > (test scope) > > > > > > / > > > > > > \/_ > > > > > > TestSupportModule2 > > > > > > If you were to make a src code change to the following test support > > > modules: > > > > > > * TestSupportModule1 > > > * TestSupportModule2 > > > > > > Then the minimum number of modules we need to build to verify the > change > > > set is OK is: > > > > > > * TestSupportModule1 > > > * TestSupportModule2 > > > * ModuleB > > > * App > > > > > > i.e. there is no requirement to build ModuleA because we know that > none of > > > the src code changes could impact the classpaths used in its maven > build. > > > > > > We know that despite 'App' depending (transitively) on ModuleB there > is no > > > need for the 'App' build to wait for ModuleB to complete its build > because > > > the src code change to TestSupportModule2 will not impact any of the > > > classpaths used in the App maven build. Therefore to get the most > efficient > > > build possible we ideally would invoke Maven to run with 2 threads and > with > > > instruction to build two distinct 'dependency graphs': > > > > > > * TestSupportModule1 followed by ModuleB > > > * TestSupportModule1 followed by App > > > > > > The following Maven command achieves exactly what we want because the > > > reactor build order is based only on the direct (i.e. non-transitive) > > > dependencies of the modules provided to the reactor in the build > command. > > > Therefore the absence of ModuleA results in two distinct 'dependency > > > graphs': > > > > > > mvn clean verify -pl TestSupportModule1,TestSupportModule2,ModuleB,App > -T 2 > > > > > > Note: In reality the code base I maintain has a very large monobuild > with > > > 100s of modules and this type of build
Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hi Jospeh, I didn’t necessarily expect that enabling the extension would solve/avoid the issue you described, but I mentioned it, because it would allow you to not have to specify an argument like '-pl TestSupportModule1,TestSupportModule2,ModuleB,App’ in the first place, because the Build Cache Extension should automatically determine which modules need to be built and for which ones previously cached artifacts can be used. Nils. > Op 6 feb 2024, om 15:11 heeft Joseph Leonard > het volgende geschreven: > > Thanks Nils, > maven-build-cache-extension looks very interesting generally – I will have a > play with it. > With regards to this issue, the maven-build-cache-extension overview > references “Subtree support for multimodule projects builds part of the > codebase in isolation” which sounded similar to the solution I am proposing > for this issue. Unfortunately, after enabling the maven-build-cache-extension > I still hit the same issue. > Joe > > On 2024/02/06 12:54:59 Nils Breunese wrote: >> I can’t comment on your question directly, but I just wanted to say that >> your use case sounds like it could benefit from the Maven Build Cache >> Extension (https://maven.apache.org/extensions/maven-build-cache-extension/). > >> >> Just my 2 cents. >> >> Nils. >> >>> Op 6 feb 2024 om 11:40 heeft Joseph Leonard het >>> volgende geschreven: >>> >>> Hi all, >>> >>> It would be great to get any thoughts on whether the following is a defect: >>> >>> >>> Issue details: >>> tl;dr >>> >>> Maven can resolve dependencies either from: >>> >>> * an external repo >>> * a class directory of a module being built within the reactor >>> * a packaged jar of a module being built within the reactor >>> >>> If you run a concurrent multi-module build it is possible to get a race >>> condition whereby the build of module Foo may resolve module Bar from >>> either of the three resolution channels. This inconsistency can result in >>> the Maven war plugin sometimes failing to build a functional war file. I >>> would expect a consistent resolution would always take place. > >>> >>> Full details >>> Scenario >>> >>> Consider you have a repo with the following structure: >>> >>> App >>> >>>/ \ >>> >>> / \ >>> >>> (compile scope) (test scope) >>> >>> / \ >>> >>> \/_ _\/ >>> >>>ModuleA TestSupportModule1 >>> >>> / >>> >>> / >>> >>> (compile scope) >>> >>>/ >>> >>> \/_ >>> >>> ModuleB >>> >>> / >>> >>> / >>> >>> (test scope) >>> >>> / >>> >>> \/_ >>> >>> TestSupportModule2 >>> >>> If you were to make a src code change to the following test support modules: >>> >>> * TestSupportModule1 >>> * TestSupportModule2 >>> >>> Then the minimum number of modules we need to build to verify the change >>> set is OK is: >>> >>> * TestSupportModule1 >>> * TestSupportModule2 >>> * ModuleB >>> * App >>> >>> i.e. there is no requirement to build ModuleA because we know that none of >>> the src code changes could impact the classpaths used in its maven build. > >>> >>> We know that despite 'App' depending (transitively) on ModuleB there is no >>> need for the 'App' build to wait for ModuleB to complete its build because >>> the src code change to TestSupportModule2 will not impact any of the >>> classpaths used in the App maven build. Therefore to get the most efficient >>> build possible we ideally would invoke Maven to run with 2 threads and with >>> instruction to build two distinct 'dependency graphs': > >>> >>> * TestSupportModule1 followed by ModuleB >>> * TestSupportModule1 followed by App >>> >>> The following Maven command achieves exactly what we want because the >>> reactor build order is based only on the direct (i.e. non-transitive) >>> dependencies of the modules provided to the reactor in the build command. >>> Therefore the absence of ModuleA results in two distinct 'dependency >>> graphs': > >>> >>> mvn clean verify -pl TestSupportModule1,TestSupportModule2,ModuleB,App -T 2 >>> >>> Note: In reality the code base I maintain has a very large monobuild with >>> 100s of modules and this type of build optimisation makes a significant >>> difference to the speed of our monobuild (we use >>> https://github.com/gitflow-incremental-builder/gitflow-incremental-builder >>> to automate the logic of determining which modules to include in the >>> reactor based on our change set). > >>> >>> Issue >>> >>> We have encountered an issue in the above scenario because the 'App' build >>> has a race condition with the ModuleB build which will result in one of the >>> following three outcomes: > >>> >>> * If the 'App' build starts before the ModuleB build has compiled its src >>> classes then the 'App' build will resolve
RE: Re: Inconsistent dependency resolution behaviour for concurrent multi-module build can cause failures
Hi Tamás, Yeah, this was unexpected to me initially as well. From what I can tell the Maven reactor only considers direct dependencies (i.e. not transitive dependencies) between the modules in the reactor when working out the build graph. For example if you have a simple linear dependency chain of: One --> Two --> Three --> Four --> Five Then invoking “mvn clean verify -pl One,Two,Four,Five -T 2 will result in two ‘graphs’ being built in parallel ([One,Two] and [Four,Five]). I assume this is as designed because it actually offers quite powerful functionality to improve the parallelism in your build. An example of where this is legit is when: * “Four” has a test scope dependency on “Five” * “One” has a test scoped dependency on “Two” If you made a src code change to “Five” and “Two” then it would be safe to build [One,Two] and [Four,Five] in parallel because you know the changes within these graphs cannot impact each other. Joe On 2024/02/06 21:37:42 Tamás Cservenák wrote: > Howdy, > > To me this looks like Maven is not aware that the App depends on ModuleB... > Are they "plain dependency" linked? Or what kind of dependency we talk > about here? > In short: why would App start while ModuleB (upstream dep) is not done? > Something is fishy here. > > T > > > On Tue, Feb 6, 2024 at 11:40 AM Joseph Leonard < > joseph.leon...@alfasystems.com> wrote: > > > Hi all, > > > > It would be great to get any thoughts on whether the following is a defect: > > > > > > Issue details: > > tl;dr > > > > Maven can resolve dependencies either from: > > > > * an external repo > > * a class directory of a module being built within the reactor > > * a packaged jar of a module being built within the reactor > > > > If you run a concurrent multi-module build it is possible to get a race > > condition whereby the build of module Foo may resolve module Bar from > > either of the three resolution channels. This inconsistency can result in > > the Maven war plugin sometimes failing to build a functional war file. I > > would expect a consistent resolution would always take place. > > > > Full details > > Scenario > > > > Consider you have a repo with the following structure: > > > >App > > > > / \ > > > > / \ > > > >(compile scope) (test scope) > > > > / \ > > > > \/_ _\/ > > > > ModuleA TestSupportModule1 > > > > / > > > >/ > > > > (compile scope) > > > > / > > > >\/_ > > > > ModuleB > > > >/ > > > > / > > > > (test scope) > > > > / > > > > \/_ > > > > TestSupportModule2 > > > > If you were to make a src code change to the following test support > > modules: > > > > * TestSupportModule1 > > * TestSupportModule2 > > > > Then the minimum number of modules we need to build to verify the change > > set is OK is: > > > > * TestSupportModule1 > > * TestSupportModule2 > > * ModuleB > > * App > > > > i.e. there is no requirement to build ModuleA because we know that none of > > the src code changes could impact the classpaths used in its maven build. > > > > We know that despite 'App' depending (transitively) on ModuleB there is no > > need for the 'App' build to wait for ModuleB to complete its build because > > the src code change to TestSupportModule2 will not impact any of the > > classpaths used in the App maven build. Therefore to get the most efficient > > build possible we ideally would invoke Maven to run with 2 threads and with > > instruction to build two distinct 'dependency graphs': > > > > * TestSupportModule1 followed by ModuleB > > * TestSupportModule1 followed by App > > > > The following Maven command achieves exactly what we want because the > > reactor build order is based only on the direct (i.e. non-transitive) > > dependencies of the modules provided to the reactor in the build command. > > Therefore the absence of ModuleA results in two distinct 'dependency > > graphs': > > > > mvn clean verify -pl TestSupportModule1,TestSupportModule2,ModuleB,App -T 2 > > > > Note: In reality the code base I maintain has a very large monobuild with > > 100s of modules and this type of build optimisation makes a significant > > difference to the speed of our monobuild (we use > > https://github.com/gitflow-incremental-builder/gitflow-incremental-builder > > to automate the logic of determining which modules to include in the > > reactor based on our change set). > > > > Issue > > > > We have encountered an issue in the above scenario because the 'App' build > > has a race condition with the ModuleB build which will result in one of the > > following three outcomes: > > > > * If the 'App' build starts before the ModuleB build has compiled its > > src classes then the 'App' build will resolve ModuleB from the
