Re: Build behavior differences between 3.2.5 and 3.3.9 with dependency shading

[Robert Metzger]

Hi Stephen,

Thanks for the offer, I'll try to make sure we return the favor.

Building the current Flink master with a current maven version (3.6.3 on my
machine), leads to the distribution
(in flink/build-target/lib/flink-dist_2.11-1.12-SNAPSHOT.jar) containing
"org/jboss/netty/util" classes.
Building the same code with maven 3.2.5, the distribution will only contain
"org/apache/flink/shaded/akka/org/jboss/netty/util".

The shading is defined here:
https://github.com/apache/flink/blob/master/flink-runtime/pom.xml#L560

We would like to build Flink using current maven versions :)

Regards,
Robert

On Mon, Jul 6, 2020 at 8:43 PM Stephen Connolly <
stephen.alan.conno...@gmail.com> wrote:

> You can make Maven do the right thing if you scope the shades dependencies
> correctly... but it’s a pain.
>
> Can you point me to one pom that’s a good example. I might be able to give
> some pointers in return for a bugfix :rofl:
>
> On Fri 3 Jul 2020 at 09:10, Robert Metzger  wrote:
>
> > Hi all,
> >
> > sorry for bringing up this very old thread again: The Flink project is
> > still using Maven 3.2.5 to have the shading behave as we need it. Using
> > such an old Maven version has some issues in our development workflow, as
> > the http library is quite outdated, and developers are working with much
> > newer maven versions locally, leading to some inconsistent dependency
> > resolutions.
> >
> > I was wondering if there was any progress in the last years, so that we
> can
> > switch to a newer Maven version?
> >
> > On Mon, Nov 7, 2016 at 12:04 AM Stephen Connolly <
> > stephen.alan.conno...@gmail.com> wrote:
> >
> > > yes that was the advice I gave on this thread originally. you do have
> to
> > > fight a bit with the shade plugin to get them shaded in though... which
> > is
> > > why I think we may need to rethink how we do shade
> > >
> > > On 6 November 2016 at 22:36, Jörg Schaible 
> > wrote:
> > >
> > > > Stephen Connolly wrote:
> > > >
> > > > > Hmmm I did some digging...
> > > > >
> > > > >
> > > > https://maven.apache.org/ref/3.2.3/apidocs/org/apache/
> > > > maven/artifact/handler/ArtifactHandler.html#isIncludesDependencies()
> > > > > is i think the idea JvZ was hinting at.
> > > > >
> > > > > For the case where a shaded JAR shades *everything* then a custom
> > > > > packaging will work as we could set this flag and it would stop the
> > > > > transitive dependencies being propagated... but most people do not
> > > shade
> > > > > *all* dependencies, rather they shade a subset.
> > > > >
> > > > > I think we may need to re-think how we do this or rethink the model
> > > being
> > > > > read-only
> > > >
> > > > Set the shaded dependencies as optional. This might at least
> > technically
> > > > produce the proper dependency tree for dependent artifacts.
> > > >
> > > > Cheers,
> > > > Jörg
> > > >
> > > >
> > > > -
> > > > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> > > > For additional commands, e-mail: users-h...@maven.apache.org
> > > >
> > > >
> > >
> >
> --
> Sent from my phone
>

Re: Build behavior differences between 3.2.5 and 3.3.9 with dependency shading

[Stephen Connolly]

You can make Maven do the right thing if you scope the shades dependencies
correctly... but it’s a pain.

Can you point me to one pom that’s a good example. I might be able to give
some pointers in return for a bugfix :rofl:

On Fri 3 Jul 2020 at 09:10, Robert Metzger  wrote:

> Hi all,
>
> sorry for bringing up this very old thread again: The Flink project is
> still using Maven 3.2.5 to have the shading behave as we need it. Using
> such an old Maven version has some issues in our development workflow, as
> the http library is quite outdated, and developers are working with much
> newer maven versions locally, leading to some inconsistent dependency
> resolutions.
>
> I was wondering if there was any progress in the last years, so that we can
> switch to a newer Maven version?
>
> On Mon, Nov 7, 2016 at 12:04 AM Stephen Connolly <
> stephen.alan.conno...@gmail.com> wrote:
>
> > yes that was the advice I gave on this thread originally. you do have to
> > fight a bit with the shade plugin to get them shaded in though... which
> is
> > why I think we may need to rethink how we do shade
> >
> > On 6 November 2016 at 22:36, Jörg Schaible 
> wrote:
> >
> > > Stephen Connolly wrote:
> > >
> > > > Hmmm I did some digging...
> > > >
> > > >
> > > https://maven.apache.org/ref/3.2.3/apidocs/org/apache/
> > > maven/artifact/handler/ArtifactHandler.html#isIncludesDependencies()
> > > > is i think the idea JvZ was hinting at.
> > > >
> > > > For the case where a shaded JAR shades *everything* then a custom
> > > > packaging will work as we could set this flag and it would stop the
> > > > transitive dependencies being propagated... but most people do not
> > shade
> > > > *all* dependencies, rather they shade a subset.
> > > >
> > > > I think we may need to re-think how we do this or rethink the model
> > being
> > > > read-only
> > >
> > > Set the shaded dependencies as optional. This might at least
> technically
> > > produce the proper dependency tree for dependent artifacts.
> > >
> > > Cheers,
> > > Jörg
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> > > For additional commands, e-mail: users-h...@maven.apache.org
> > >
> > >
> >
>
-- 
Sent from my phone

Re: Build behavior differences between 3.2.5 and 3.3.9 with dependency shading

[Robert Metzger]

Hi all,

sorry for bringing up this very old thread again: The Flink project is
still using Maven 3.2.5 to have the shading behave as we need it. Using
such an old Maven version has some issues in our development workflow, as
the http library is quite outdated, and developers are working with much
newer maven versions locally, leading to some inconsistent dependency
resolutions.

I was wondering if there was any progress in the last years, so that we can
switch to a newer Maven version?

On Mon, Nov 7, 2016 at 12:04 AM Stephen Connolly <
stephen.alan.conno...@gmail.com> wrote:

> yes that was the advice I gave on this thread originally. you do have to
> fight a bit with the shade plugin to get them shaded in though... which is
> why I think we may need to rethink how we do shade
>
> On 6 November 2016 at 22:36, Jörg Schaible  wrote:
>
> > Stephen Connolly wrote:
> >
> > > Hmmm I did some digging...
> > >
> > >
> > https://maven.apache.org/ref/3.2.3/apidocs/org/apache/
> > maven/artifact/handler/ArtifactHandler.html#isIncludesDependencies()
> > > is i think the idea JvZ was hinting at.
> > >
> > > For the case where a shaded JAR shades *everything* then a custom
> > > packaging will work as we could set this flag and it would stop the
> > > transitive dependencies being propagated... but most people do not
> shade
> > > *all* dependencies, rather they shade a subset.
> > >
> > > I think we may need to re-think how we do this or rethink the model
> being
> > > read-only
> >
> > Set the shaded dependencies as optional. This might at least technically
> > produce the proper dependency tree for dependent artifacts.
> >
> > Cheers,
> > Jörg
> >
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> > For additional commands, e-mail: users-h...@maven.apache.org
> >
> >
>

Re: Build behavior differences between 3.2.5 and 3.3.9 with dependency shading

[Stephen Connolly]

yes that was the advice I gave on this thread originally. you do have to
fight a bit with the shade plugin to get them shaded in though... which is
why I think we may need to rethink how we do shade

On 6 November 2016 at 22:36, Jörg Schaible  wrote:

> Stephen Connolly wrote:
>
> > Hmmm I did some digging...
> >
> >
> https://maven.apache.org/ref/3.2.3/apidocs/org/apache/
> maven/artifact/handler/ArtifactHandler.html#isIncludesDependencies()
> > is i think the idea JvZ was hinting at.
> >
> > For the case where a shaded JAR shades *everything* then a custom
> > packaging will work as we could set this flag and it would stop the
> > transitive dependencies being propagated... but most people do not shade
> > *all* dependencies, rather they shade a subset.
> >
> > I think we may need to re-think how we do this or rethink the model being
> > read-only
>
> Set the shaded dependencies as optional. This might at least technically
> produce the proper dependency tree for dependent artifacts.
>
> Cheers,
> Jörg
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
> For additional commands, e-mail: users-h...@maven.apache.org
>
>

Re: Build behavior differences between 3.2.5 and 3.3.9 with dependency shading

[Jörg Schaible]

Stephen Connolly wrote:

> Hmmm I did some digging...
> 
> 
https://maven.apache.org/ref/3.2.3/apidocs/org/apache/maven/artifact/handler/ArtifactHandler.html#isIncludesDependencies()
> is i think the idea JvZ was hinting at.
> 
> For the case where a shaded JAR shades *everything* then a custom
> packaging will work as we could set this flag and it would stop the
> transitive dependencies being propagated... but most people do not shade
> *all* dependencies, rather they shade a subset.
> 
> I think we may need to re-think how we do this or rethink the model being
> read-only

Set the shaded dependencies as optional. This might at least technically 
produce the proper dependency tree for dependent artifacts.

Cheers,
Jörg


-
To unsubscribe, e-mail: users-unsubscr...@maven.apache.org
For additional commands, e-mail: users-h...@maven.apache.org

Re: Build behavior differences between 3.2.5 and 3.3.9 with dependency shading

[Stephen Connolly]

Hmmm I did some digging...

https://maven.apache.org/ref/3.2.3/apidocs/org/apache/maven/artifact/handler/ArtifactHandler.html#isIncludesDependencies()
is i think the idea JvZ was hinting at.

For the case where a shaded JAR shades *everything* then a custom packaging
will work as we could set this flag and it would stop the transitive
dependencies being propagated... but most people do not shade *all*
dependencies, rather they shade a subset.

I think we may need to re-think how we do this or rethink the model being
read-only

On 11 December 2015 at 08:27, Robert Metzger  wrote:

> Thank you for the information.
> I've changed now all our Guava dependencies to optional, but it is still
> contained in the first build of the parent-pom.
>
> This is the repository https://github.com/rmetzger/flink/tree/flink3158
> ("flink3158" branch).
> The dependency:tree of "flink-dist" is not showing me guava anywhere, so
> its really hard to figure out where the jars are coming from.
>
>
> For completeness, I'll link to the initial discussion about this issue on
> the Flink mailing list:
> http://mail-archives.apache.org/mod_mbox/flink-dev/201512.
> mbox/%3CCANZa%3DGvFA%2B61968DBYoZc%3D8WfEmoF01DJAkmvzUcUH5XycLQ5w
> %40mail.gmail.com%3E
>
>
> On Thu, Dec 10, 2015 at 5:01 PM, Stephen Connolly <
> stephen.alan.conno...@gmail.com> wrote:
>
> > You need to do this in any module that is producing a dependency reduced
> > pom (and only in those modules)
> >
> > You can leave the version being inherited from dependencyManagement.
> >
> > I advise using the optional technique until Jason gets a new packaging
> for
> > dependency reduced artifacts (or decides to agree with my suggestion of
> > limited ability to modify the model after the reactor has been
> constructed)
> >
> > The 3.2 behaviour is currently viewed as a bug that has been abused by
> the
> > shade plugin rather than a feature. 3.3 enforces the immutability of the
> > model and I don't see that being rolled back
> >
> > On Thursday 10 December 2015, Robert Metzger 
> wrote:
> >
> > > Okay, the true suggestion sounds interesting. I'll try
> that
> > > out.
> > >
> > > However, I'm still wondering why the behavior between 3.2 and 3.3 is
> > > different?
> > > Our CI system runs Maven 3.2, so the shading is done correctly there.
> If
> > a
> > > project committer now adds a guava dependency and forgets the
> ,
> > > we end up with guava in our final fat jar for users building with 3.3.
> > > Putting guava with optional=true into our dependency management is not
> an
> > > option because that would overwrite Hadoop's guava version (we shade
> > hadoop
> > > and its guava in our project as well)
> > >
> > >
> > >
> > > On Thu, Dec 10, 2015 at 2:08 PM, Stephen Connolly <
> > > stephen.alan.conno...@gmail.com > wrote:
> > >
> > > > Dependency reduced poms require mutation of the model after the build
> > > > started. JvZ is investigating a different packaging type to resolve
> > > this...
> > > > Workaround for now is to mark all the dependencies that are removed
> as
> > > > true so that they are no longer transitive and
> > that
> > > > way the effective reactor Pom is the same from a transitive
> dependency
> > > PoV
> > > > as the dependency reduced one that gets published
> > > >
> > > > On Thursday 10 December 2015, Robert Metzger  > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > The Apache Flink project is using Maven for dependency management.
> We
> > > > shade
> > > > > Google's Guava away (to org.apache.flink.shaded.com.
> google.commons)
> > to
> > > > > avoid conflicts with user guava versions.
> > > > >
> > > > > Building Flink with Maven 3.2.5 will create a valid fat-jar without
> > > > guava.
> > > > > However, Maven 3.3.9 (and other 3.3.x versions) are including guava
> > in
> > > > the
> > > > > com/google/commons namespace.
> > > > > Interestingly, doing only a "clean install" in the "flink-dist"
> > package
> > > > > after a build of the parent pom results in a correct "flink-dist"
> fat
> > > > jar.
> > > > >
> > > > > I'm wondering which behavior of Maven is correct 3.2 or 3.3 ?
> > > > > I have the feeling that 3.3 is behaving incorrectly because the
> > > > > dependency:tree of "flink-dist" does not contain Guava.
> > > > > Maybe a "clean install" on the parent pom with Maven 3.3 is not
> > > > respecting
> > > > > the dependency-reduced poms created by the other modules?
> > > > >
> > > > > Regards,
> > > > > Robert
> > > > >
> > > >
> > > >
> > > > --
> > > > Sent from my phone
> > > >
> > >
> >
> >
> > --
> > Sent from my phone
> >
>

Re: Build behavior differences between 3.2.5 and 3.3.9 with dependency shading

[Robert Metzger]

Thank you for the information.
I've changed now all our Guava dependencies to optional, but it is still
contained in the first build of the parent-pom.

This is the repository https://github.com/rmetzger/flink/tree/flink3158
("flink3158" branch).
The dependency:tree of "flink-dist" is not showing me guava anywhere, so
its really hard to figure out where the jars are coming from.


For completeness, I'll link to the initial discussion about this issue on
the Flink mailing list:
http://mail-archives.apache.org/mod_mbox/flink-dev/201512.mbox/%3CCANZa%3DGvFA%2B61968DBYoZc%3D8WfEmoF01DJAkmvzUcUH5XycLQ5w%40mail.gmail.com%3E


On Thu, Dec 10, 2015 at 5:01 PM, Stephen Connolly <
stephen.alan.conno...@gmail.com> wrote:

> You need to do this in any module that is producing a dependency reduced
> pom (and only in those modules)
>
> You can leave the version being inherited from dependencyManagement.
>
> I advise using the optional technique until Jason gets a new packaging for
> dependency reduced artifacts (or decides to agree with my suggestion of
> limited ability to modify the model after the reactor has been constructed)
>
> The 3.2 behaviour is currently viewed as a bug that has been abused by the
> shade plugin rather than a feature. 3.3 enforces the immutability of the
> model and I don't see that being rolled back
>
> On Thursday 10 December 2015, Robert Metzger  wrote:
>
> > Okay, the true suggestion sounds interesting. I'll try that
> > out.
> >
> > However, I'm still wondering why the behavior between 3.2 and 3.3 is
> > different?
> > Our CI system runs Maven 3.2, so the shading is done correctly there. If
> a
> > project committer now adds a guava dependency and forgets the ,
> > we end up with guava in our final fat jar for users building with 3.3.
> > Putting guava with optional=true into our dependency management is not an
> > option because that would overwrite Hadoop's guava version (we shade
> hadoop
> > and its guava in our project as well)
> >
> >
> >
> > On Thu, Dec 10, 2015 at 2:08 PM, Stephen Connolly <
> > stephen.alan.conno...@gmail.com > wrote:
> >
> > > Dependency reduced poms require mutation of the model after the build
> > > started. JvZ is investigating a different packaging type to resolve
> > this...
> > > Workaround for now is to mark all the dependencies that are removed as
> > > true so that they are no longer transitive and
> that
> > > way the effective reactor Pom is the same from a transitive dependency
> > PoV
> > > as the dependency reduced one that gets published
> > >
> > > On Thursday 10 December 2015, Robert Metzger  > > wrote:
> > >
> > > > Hi,
> > > >
> > > > The Apache Flink project is using Maven for dependency management. We
> > > shade
> > > > Google's Guava away (to org.apache.flink.shaded.com.google.commons)
> to
> > > > avoid conflicts with user guava versions.
> > > >
> > > > Building Flink with Maven 3.2.5 will create a valid fat-jar without
> > > guava.
> > > > However, Maven 3.3.9 (and other 3.3.x versions) are including guava
> in
> > > the
> > > > com/google/commons namespace.
> > > > Interestingly, doing only a "clean install" in the "flink-dist"
> package
> > > > after a build of the parent pom results in a correct "flink-dist" fat
> > > jar.
> > > >
> > > > I'm wondering which behavior of Maven is correct 3.2 or 3.3 ?
> > > > I have the feeling that 3.3 is behaving incorrectly because the
> > > > dependency:tree of "flink-dist" does not contain Guava.
> > > > Maybe a "clean install" on the parent pom with Maven 3.3 is not
> > > respecting
> > > > the dependency-reduced poms created by the other modules?
> > > >
> > > > Regards,
> > > > Robert
> > > >
> > >
> > >
> > > --
> > > Sent from my phone
> > >
> >
>
>
> --
> Sent from my phone
>

Re: Build behavior differences between 3.2.5 and 3.3.9 with dependency shading

[Stephen Connolly]

Dependency reduced poms require mutation of the model after the build
started. JvZ is investigating a different packaging type to resolve this...
Workaround for now is to mark all the dependencies that are removed as
true so that they are no longer transitive and that
way the effective reactor Pom is the same from a transitive dependency PoV
as the dependency reduced one that gets published

On Thursday 10 December 2015, Robert Metzger  wrote:

> Hi,
>
> The Apache Flink project is using Maven for dependency management. We shade
> Google's Guava away (to org.apache.flink.shaded.com.google.commons) to
> avoid conflicts with user guava versions.
>
> Building Flink with Maven 3.2.5 will create a valid fat-jar without guava.
> However, Maven 3.3.9 (and other 3.3.x versions) are including guava in the
> com/google/commons namespace.
> Interestingly, doing only a "clean install" in the "flink-dist" package
> after a build of the parent pom results in a correct "flink-dist" fat jar.
>
> I'm wondering which behavior of Maven is correct 3.2 or 3.3 ?
> I have the feeling that 3.3 is behaving incorrectly because the
> dependency:tree of "flink-dist" does not contain Guava.
> Maybe a "clean install" on the parent pom with Maven 3.3 is not respecting
> the dependency-reduced poms created by the other modules?
>
> Regards,
> Robert
>


-- 
Sent from my phone

Re: Build behavior differences between 3.2.5 and 3.3.9 with dependency shading

[Robert Metzger]

Okay, the true suggestion sounds interesting. I'll try that
out.

However, I'm still wondering why the behavior between 3.2 and 3.3 is
different?
Our CI system runs Maven 3.2, so the shading is done correctly there. If a
project committer now adds a guava dependency and forgets the ,
we end up with guava in our final fat jar for users building with 3.3.
Putting guava with optional=true into our dependency management is not an
option because that would overwrite Hadoop's guava version (we shade hadoop
and its guava in our project as well)



On Thu, Dec 10, 2015 at 2:08 PM, Stephen Connolly <
stephen.alan.conno...@gmail.com> wrote:

> Dependency reduced poms require mutation of the model after the build
> started. JvZ is investigating a different packaging type to resolve this...
> Workaround for now is to mark all the dependencies that are removed as
> true so that they are no longer transitive and that
> way the effective reactor Pom is the same from a transitive dependency PoV
> as the dependency reduced one that gets published
>
> On Thursday 10 December 2015, Robert Metzger  wrote:
>
> > Hi,
> >
> > The Apache Flink project is using Maven for dependency management. We
> shade
> > Google's Guava away (to org.apache.flink.shaded.com.google.commons) to
> > avoid conflicts with user guava versions.
> >
> > Building Flink with Maven 3.2.5 will create a valid fat-jar without
> guava.
> > However, Maven 3.3.9 (and other 3.3.x versions) are including guava in
> the
> > com/google/commons namespace.
> > Interestingly, doing only a "clean install" in the "flink-dist" package
> > after a build of the parent pom results in a correct "flink-dist" fat
> jar.
> >
> > I'm wondering which behavior of Maven is correct 3.2 or 3.3 ?
> > I have the feeling that 3.3 is behaving incorrectly because the
> > dependency:tree of "flink-dist" does not contain Guava.
> > Maybe a "clean install" on the parent pom with Maven 3.3 is not
> respecting
> > the dependency-reduced poms created by the other modules?
> >
> > Regards,
> > Robert
> >
>
>
> --
> Sent from my phone
>

Re: Build behavior differences between 3.2.5 and 3.3.9 with dependency shading

[Stephen Connolly]

You need to do this in any module that is producing a dependency reduced
pom (and only in those modules)

You can leave the version being inherited from dependencyManagement.

I advise using the optional technique until Jason gets a new packaging for
dependency reduced artifacts (or decides to agree with my suggestion of
limited ability to modify the model after the reactor has been constructed)

The 3.2 behaviour is currently viewed as a bug that has been abused by the
shade plugin rather than a feature. 3.3 enforces the immutability of the
model and I don't see that being rolled back

On Thursday 10 December 2015, Robert Metzger  wrote:

> Okay, the true suggestion sounds interesting. I'll try that
> out.
>
> However, I'm still wondering why the behavior between 3.2 and 3.3 is
> different?
> Our CI system runs Maven 3.2, so the shading is done correctly there. If a
> project committer now adds a guava dependency and forgets the ,
> we end up with guava in our final fat jar for users building with 3.3.
> Putting guava with optional=true into our dependency management is not an
> option because that would overwrite Hadoop's guava version (we shade hadoop
> and its guava in our project as well)
>
>
>
> On Thu, Dec 10, 2015 at 2:08 PM, Stephen Connolly <
> stephen.alan.conno...@gmail.com > wrote:
>
> > Dependency reduced poms require mutation of the model after the build
> > started. JvZ is investigating a different packaging type to resolve
> this...
> > Workaround for now is to mark all the dependencies that are removed as
> > true so that they are no longer transitive and that
> > way the effective reactor Pom is the same from a transitive dependency
> PoV
> > as the dependency reduced one that gets published
> >
> > On Thursday 10 December 2015, Robert Metzger  > wrote:
> >
> > > Hi,
> > >
> > > The Apache Flink project is using Maven for dependency management. We
> > shade
> > > Google's Guava away (to org.apache.flink.shaded.com.google.commons) to
> > > avoid conflicts with user guava versions.
> > >
> > > Building Flink with Maven 3.2.5 will create a valid fat-jar without
> > guava.
> > > However, Maven 3.3.9 (and other 3.3.x versions) are including guava in
> > the
> > > com/google/commons namespace.
> > > Interestingly, doing only a "clean install" in the "flink-dist" package
> > > after a build of the parent pom results in a correct "flink-dist" fat
> > jar.
> > >
> > > I'm wondering which behavior of Maven is correct 3.2 or 3.3 ?
> > > I have the feeling that 3.3 is behaving incorrectly because the
> > > dependency:tree of "flink-dist" does not contain Guava.
> > > Maybe a "clean install" on the parent pom with Maven 3.3 is not
> > respecting
> > > the dependency-reduced poms created by the other modules?
> > >
> > > Regards,
> > > Robert
> > >
> >
> >
> > --
> > Sent from my phone
> >
>


-- 
Sent from my phone