Re: Build behavior differences between 3.2.5 and 3.3.9 with dependency shading
Hi Stephen, Thanks for the offer, I'll try to make sure we return the favor. Building the current Flink master with a current maven version (3.6.3 on my machine), leads to the distribution (in flink/build-target/lib/flink-dist_2.11-1.12-SNAPSHOT.jar) containing "org/jboss/netty/util" classes. Building the same code with maven 3.2.5, the distribution will only contain "org/apache/flink/shaded/akka/org/jboss/netty/util". The shading is defined here: https://github.com/apache/flink/blob/master/flink-runtime/pom.xml#L560 We would like to build Flink using current maven versions :) Regards, Robert On Mon, Jul 6, 2020 at 8:43 PM Stephen Connolly < stephen.alan.conno...@gmail.com> wrote: > You can make Maven do the right thing if you scope the shades dependencies > correctly... but it’s a pain. > > Can you point me to one pom that’s a good example. I might be able to give > some pointers in return for a bugfix :rofl: > > On Fri 3 Jul 2020 at 09:10, Robert Metzger wrote: > > > Hi all, > > > > sorry for bringing up this very old thread again: The Flink project is > > still using Maven 3.2.5 to have the shading behave as we need it. Using > > such an old Maven version has some issues in our development workflow, as > > the http library is quite outdated, and developers are working with much > > newer maven versions locally, leading to some inconsistent dependency > > resolutions. > > > > I was wondering if there was any progress in the last years, so that we > can > > switch to a newer Maven version? > > > > On Mon, Nov 7, 2016 at 12:04 AM Stephen Connolly < > > stephen.alan.conno...@gmail.com> wrote: > > > > > yes that was the advice I gave on this thread originally. you do have > to > > > fight a bit with the shade plugin to get them shaded in though... which > > is > > > why I think we may need to rethink how we do shade > > > > > > On 6 November 2016 at 22:36, Jörg Schaible > > wrote: > > > > > > > Stephen Connolly wrote: > > > > > > > > > Hmmm I did some digging... > > > > > > > > > > > > > > https://maven.apache.org/ref/3.2.3/apidocs/org/apache/ > > > > maven/artifact/handler/ArtifactHandler.html#isIncludesDependencies() > > > > > is i think the idea JvZ was hinting at. > > > > > > > > > > For the case where a shaded JAR shades *everything* then a custom > > > > > packaging will work as we could set this flag and it would stop the > > > > > transitive dependencies being propagated... but most people do not > > > shade > > > > > *all* dependencies, rather they shade a subset. > > > > > > > > > > I think we may need to re-think how we do this or rethink the model > > > being > > > > > read-only > > > > > > > > Set the shaded dependencies as optional. This might at least > > technically > > > > produce the proper dependency tree for dependent artifacts. > > > > > > > > Cheers, > > > > Jörg > > > > > > > > > > > > - > > > > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > > > > For additional commands, e-mail: users-h...@maven.apache.org > > > > > > > > > > > > > > -- > Sent from my phone >
Re: Build behavior differences between 3.2.5 and 3.3.9 with dependency shading
Hi all, sorry for bringing up this very old thread again: The Flink project is still using Maven 3.2.5 to have the shading behave as we need it. Using such an old Maven version has some issues in our development workflow, as the http library is quite outdated, and developers are working with much newer maven versions locally, leading to some inconsistent dependency resolutions. I was wondering if there was any progress in the last years, so that we can switch to a newer Maven version? On Mon, Nov 7, 2016 at 12:04 AM Stephen Connolly < stephen.alan.conno...@gmail.com> wrote: > yes that was the advice I gave on this thread originally. you do have to > fight a bit with the shade plugin to get them shaded in though... which is > why I think we may need to rethink how we do shade > > On 6 November 2016 at 22:36, Jörg Schaible wrote: > > > Stephen Connolly wrote: > > > > > Hmmm I did some digging... > > > > > > > > https://maven.apache.org/ref/3.2.3/apidocs/org/apache/ > > maven/artifact/handler/ArtifactHandler.html#isIncludesDependencies() > > > is i think the idea JvZ was hinting at. > > > > > > For the case where a shaded JAR shades *everything* then a custom > > > packaging will work as we could set this flag and it would stop the > > > transitive dependencies being propagated... but most people do not > shade > > > *all* dependencies, rather they shade a subset. > > > > > > I think we may need to re-think how we do this or rethink the model > being > > > read-only > > > > Set the shaded dependencies as optional. This might at least technically > > produce the proper dependency tree for dependent artifacts. > > > > Cheers, > > Jörg > > > > > > - > > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > > For additional commands, e-mail: users-h...@maven.apache.org > > > > >
Re: Build behavior differences between 3.2.5 and 3.3.9 with dependency shading
Thank you for the information.
I've changed now all our Guava dependencies to optional, but it is still
contained in the first build of the parent-pom.
This is the repository https://github.com/rmetzger/flink/tree/flink3158
("flink3158" branch).
The dependency:tree of "flink-dist" is not showing me guava anywhere, so
its really hard to figure out where the jars are coming from.
For completeness, I'll link to the initial discussion about this issue on
the Flink mailing list:
http://mail-archives.apache.org/mod_mbox/flink-dev/201512.mbox/%3CCANZa%3DGvFA%2B61968DBYoZc%3D8WfEmoF01DJAkmvzUcUH5XycLQ5w%40mail.gmail.com%3E
On Thu, Dec 10, 2015 at 5:01 PM, Stephen Connolly <
stephen.alan.conno...@gmail.com> wrote:
> You need to do this in any module that is producing a dependency reduced
> pom (and only in those modules)
>
> You can leave the version being inherited from dependencyManagement.
>
> I advise using the optional technique until Jason gets a new packaging for
> dependency reduced artifacts (or decides to agree with my suggestion of
> limited ability to modify the model after the reactor has been constructed)
>
> The 3.2 behaviour is currently viewed as a bug that has been abused by the
> shade plugin rather than a feature. 3.3 enforces the immutability of the
> model and I don't see that being rolled back
>
> On Thursday 10 December 2015, Robert Metzger <rmetz...@apache.org> wrote:
>
> > Okay, the true suggestion sounds interesting. I'll try that
> > out.
> >
> > However, I'm still wondering why the behavior between 3.2 and 3.3 is
> > different?
> > Our CI system runs Maven 3.2, so the shading is done correctly there. If
> a
> > project committer now adds a guava dependency and forgets the ,
> > we end up with guava in our final fat jar for users building with 3.3.
> > Putting guava with optional=true into our dependency management is not an
> > option because that would overwrite Hadoop's guava version (we shade
> hadoop
> > and its guava in our project as well)
> >
> >
> >
> > On Thu, Dec 10, 2015 at 2:08 PM, Stephen Connolly <
> > stephen.alan.conno...@gmail.com <javascript:;>> wrote:
> >
> > > Dependency reduced poms require mutation of the model after the build
> > > started. JvZ is investigating a different packaging type to resolve
> > this...
> > > Workaround for now is to mark all the dependencies that are removed as
> > > true so that they are no longer transitive and
> that
> > > way the effective reactor Pom is the same from a transitive dependency
> > PoV
> > > as the dependency reduced one that gets published
> > >
> > > On Thursday 10 December 2015, Robert Metzger <rmetz...@apache.org
> > <javascript:;>> wrote:
> > >
> > > > Hi,
> > > >
> > > > The Apache Flink project is using Maven for dependency management. We
> > > shade
> > > > Google's Guava away (to org.apache.flink.shaded.com.google.commons)
> to
> > > > avoid conflicts with user guava versions.
> > > >
> > > > Building Flink with Maven 3.2.5 will create a valid fat-jar without
> > > guava.
> > > > However, Maven 3.3.9 (and other 3.3.x versions) are including guava
> in
> > > the
> > > > com/google/commons namespace.
> > > > Interestingly, doing only a "clean install" in the "flink-dist"
> package
> > > > after a build of the parent pom results in a correct "flink-dist" fat
> > > jar.
> > > >
> > > > I'm wondering which behavior of Maven is correct 3.2 or 3.3 ?
> > > > I have the feeling that 3.3 is behaving incorrectly because the
> > > > dependency:tree of "flink-dist" does not contain Guava.
> > > > Maybe a "clean install" on the parent pom with Maven 3.3 is not
> > > respecting
> > > > the dependency-reduced poms created by the other modules?
> > > >
> > > > Regards,
> > > > Robert
> > > >
> > >
> > >
> > > --
> > > Sent from my phone
> > >
> >
>
>
> --
> Sent from my phone
>
Re: Build behavior differences between 3.2.5 and 3.3.9 with dependency shading
Okay, the true suggestion sounds interesting. I'll try that out. However, I'm still wondering why the behavior between 3.2 and 3.3 is different? Our CI system runs Maven 3.2, so the shading is done correctly there. If a project committer now adds a guava dependency and forgets the , we end up with guava in our final fat jar for users building with 3.3. Putting guava with optional=true into our dependency management is not an option because that would overwrite Hadoop's guava version (we shade hadoop and its guava in our project as well) On Thu, Dec 10, 2015 at 2:08 PM, Stephen Connolly < stephen.alan.conno...@gmail.com> wrote: > Dependency reduced poms require mutation of the model after the build > started. JvZ is investigating a different packaging type to resolve this... > Workaround for now is to mark all the dependencies that are removed as > true so that they are no longer transitive and that > way the effective reactor Pom is the same from a transitive dependency PoV > as the dependency reduced one that gets published > > On Thursday 10 December 2015, Robert Metzger <rmetz...@apache.org> wrote: > > > Hi, > > > > The Apache Flink project is using Maven for dependency management. We > shade > > Google's Guava away (to org.apache.flink.shaded.com.google.commons) to > > avoid conflicts with user guava versions. > > > > Building Flink with Maven 3.2.5 will create a valid fat-jar without > guava. > > However, Maven 3.3.9 (and other 3.3.x versions) are including guava in > the > > com/google/commons namespace. > > Interestingly, doing only a "clean install" in the "flink-dist" package > > after a build of the parent pom results in a correct "flink-dist" fat > jar. > > > > I'm wondering which behavior of Maven is correct 3.2 or 3.3 ? > > I have the feeling that 3.3 is behaving incorrectly because the > > dependency:tree of "flink-dist" does not contain Guava. > > Maybe a "clean install" on the parent pom with Maven 3.3 is not > respecting > > the dependency-reduced poms created by the other modules? > > > > Regards, > > Robert > > > > > -- > Sent from my phone >
Build behavior differences between 3.2.5 and 3.3.9 with dependency shading
Hi, The Apache Flink project is using Maven for dependency management. We shade Google's Guava away (to org.apache.flink.shaded.com.google.commons) to avoid conflicts with user guava versions. Building Flink with Maven 3.2.5 will create a valid fat-jar without guava. However, Maven 3.3.9 (and other 3.3.x versions) are including guava in the com/google/commons namespace. Interestingly, doing only a "clean install" in the "flink-dist" package after a build of the parent pom results in a correct "flink-dist" fat jar. I'm wondering which behavior of Maven is correct 3.2 or 3.3 ? I have the feeling that 3.3 is behaving incorrectly because the dependency:tree of "flink-dist" does not contain Guava. Maybe a "clean install" on the parent pom with Maven 3.3 is not respecting the dependency-reduced poms created by the other modules? Regards, Robert
mvn -nsu (--no-snapshot-updates) is still downloading SNAPSHOT versions
Hi, At our project we recently had an issue where we accidentally included outdated artifacts (which were still available on snapshot repositories). To detect those issues in the future by our integration tests, I want to instruct maven not to download snapshot updates, using the -nsu command line argument. However, doing mvn clean install -nsu -DskipTests -Dmaven.repo.local=/tmp/test-mvn will still lead to: [INFO] [INFO] Building flink-language-binding-generic 0.9-SNAPSHOT [INFO] Downloading: http://repository.apache.org/snapshots/org/apache/flink/flink-compiler/0.9-SNAPSHOT/maven-metadata.xmlDownloaded: http://repository.apache.org/snapshots/org/apache/flink/flink-compiler/0.9-SNAPSHOT/maven-metadata.xml (2 KB at 0.8 KB/sec)Downloading: http://repository.apache.org/snapshots/org/apache/flink/flink-compiler/0.9-SNAPSHOT/flink-compiler-0.9-20150313.122611-190.pomDownloaded: http://repository.apache.org/snapshots/org/apache/flink/flink-compiler/0.9-SNAPSHOT/flink-compiler-0.9-20150313.122611-190.pom (4 KB at 5.7 KB/sec)Downloading: http://repository.apache.org/snapshots/org/apache/flink/flink-compiler/0.9-SNAPSHOT/flink-compiler-0.9-20150313.122611-190.jarDownloaded: http://repository.apache.org/snapshots/org/apache/flink/flink-compiler/0.9-SNAPSHOT/flink-compiler-0.9-20150313.122611-190.jar (2660 KB at 379.7 KB/sec) I've added the steps to reproduce here: https://issues.apache.org/jira/browse/MNG-5064?focusedCommentId=14551952page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14551952 I would like to know whether I misunderstood the feature or if this is a bug. Best, Robert
Re: mvn -nsu (--no-snapshot-updates) is still downloading SNAPSHOT versions
Thank you Nick. Your solution would work, but would require us to let users call a bash script which is calling versions:display-dependency-updates and then doing the regular build. We would like to run our builds directly by mvn clean install. Is there a way to let a maven build fail if we depend on nonexistent or SNAPSHOT dependencies? So what happened in our project is the following: We renamed a module from flink-compiler to flink-optimizer. While merging some older patches, a new module was still depending on flink-compiler (the old name) and we didn't notice this because maven was getting the flink-compiler file from apache's snapshot repository. Ideally, the maven build should fail when somebody is doing mvn clean install. On Tue, May 26, 2015 at 11:30 AM, Nick Stolwijk nick.stolw...@gmail.com wrote: Maven is not downloading updates, but only the SNAPSHOTS it is missing. If you build another time it won't download anything and even if your update window (mostly daily) has been passed Maven will not look for new updates but use the ones it has locally. If you want to know which dependencies needs updates you can use the versions plugin. - versions:display-dependency-updates http://mojo.codehaus.org/versions-maven-plugin/display-dependency-updates-mojo.html scans a project's dependencies and produces a report of those dependencies which have newer versions available. - versions:display-plugin-updates http://mojo.codehaus.org/versions-maven-plugin/display-plugin-updates-mojo.html scans a project's plugins and produces a report of those plugins which have newer versions available. - versions:display-property-updates http://mojo.codehaus.org/versions-maven-plugin/display-property-updates-mojo.html scans a projectand produces a report of those properties which are used to control artifact versions and which properies have newer versions available. Hth, Nick Stolwijk ~~~ Try to leave this world a little better than you found it and, when your turn comes to die, you can die happy in feeling that at any rate you have not wasted your time but have done your best ~~~ Lord Baden-Powell On Tue, May 26, 2015 at 11:18 AM, Robert Metzger rmetz...@apache.org wrote: Hi, At our project we recently had an issue where we accidentally included outdated artifacts (which were still available on snapshot repositories). To detect those issues in the future by our integration tests, I want to instruct maven not to download snapshot updates, using the -nsu command line argument. However, doing mvn clean install -nsu -DskipTests -Dmaven.repo.local=/tmp/test-mvn will still lead to: [INFO] [INFO] Building flink-language-binding-generic 0.9-SNAPSHOT [INFO] Downloading: http://repository.apache.org/snapshots/org/apache/flink/flink-compiler/0.9-SNAPSHOT/maven-metadata.xmlDownloaded : http://repository.apache.org/snapshots/org/apache/flink/flink-compiler/0.9-SNAPSHOT/maven-metadata.xml (2 KB at 0.8 KB/sec)Downloading: http://repository.apache.org/snapshots/org/apache/flink/flink-compiler/0.9-SNAPSHOT/flink-compiler-0.9-20150313.122611-190.pomDownloaded : http://repository.apache.org/snapshots/org/apache/flink/flink-compiler/0.9-SNAPSHOT/flink-compiler-0.9-20150313.122611-190.pom (4 KB at 5.7 KB/sec)Downloading: http://repository.apache.org/snapshots/org/apache/flink/flink-compiler/0.9-SNAPSHOT/flink-compiler-0.9-20150313.122611-190.jarDownloaded : http://repository.apache.org/snapshots/org/apache/flink/flink-compiler/0.9-SNAPSHOT/flink-compiler-0.9-20150313.122611-190.jar (2660 KB at 379.7 KB/sec) I've added the steps to reproduce here: https://issues.apache.org/jira/browse/MNG-5064?focusedCommentId=14551952page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14551952 I would like to know whether I misunderstood the feature or if this is a bug. Best, Robert
Re: Build fat jar transitively excluding some dependencies
Cool. Thanks for the pointer. I'll have a closer look at the mentioned IDEA JIRA. On Fri, Feb 27, 2015 at 2:52 PM, Ben Podgursky bpodgur...@gmail.com wrote: Yeah, I understand the problem with the main method. Fyi, there's an active intellij ticket with more discussion about this: https://youtrack.jetbrains.com/issue/IDEA-107048 Another kind of awkward workaround is to create a maven run configuration with the desired scope: https://www.jetbrains.com/idea/help/creating-maven-run-debug-configuration.html (so the run configuration args would be exec:java -Dexec.mainClass=classname -Dexec.classpathScope=test) On Fri, Feb 27, 2015 at 1:13 AM, Robert Metzger rmetz...@apache.org wrote: Thank you for the replies. Martin: Will the maven-assemby-plugin also exclude transitive dependencies? Ben: Our users are used to run Flink programs out of the main() method, so IntelliJ will not include the required jars into the classpath. I basically want some dependencies to be 'provided' when the maven-assembly-plugin runs but otherwise they should be in the 'compile' scope. On Tue, Feb 24, 2015 at 8:51 PM, Ben Podgursky bpodgur...@gmail.com wrote: We package up our job jars using maven assemblies using the 'provided' scope to exclude hadoop jars, and use intellij for local development and testing. We've found that it's easiest to just do all local debugging using junit tests since provided jars will be on the classpath there (if you don't want it to be run during actual unit testing you can @Ignore the class). Not super elegant but it works and encourages people to do testing via actual tests, rather than manual scripts. On Tue, Feb 24, 2015 at 4:53 AM, Robert Metzger rmetz...@apache.org wrote: Hi, I'm a committer at the Apache Flink project (a system for distrib. data processing). We provide our users a quickstart maven archetype to bootstrap new Flink jobs. For the generated Flink job's maven project, I would like to build a fat-jar that contains all the dependencies the user added to the project. However, I don't want to include the flink dependencies into the fat jar. The purpose of the fat-jar is to submit it to the cluster for executing the user's job. So it should contain the usercode, all the user's dependencies BUT NOT the flink dependencies, because we can assume them to be available in the running cluster. A fat-jar with Flink's dependencies is 60MB+, which can be annoying when uploading the jars to a cluster. I'm using the shade plugin to do that. So my first idea was to exclude everything in the org.apache.flink groupId from the fat jar. However, this is not possible because - we can only expect some artifacts to be available at runtime (Flink ships the core jars with the binary builds. Extensions have to be loaded by the user) - If users put code in their archetype project into the org.apache.flink namespace, we exclude usercode. So what I'm looking for is a way to tell the shade (or maven assembly) plugin to exclude a list of artifacts and their transitive dependencies. In case someone asks for it: I can not use the 'provided' scope for the Flink dependencies because users can also start (and debug) their Flink jobs locally. Setting the dependencies to 'provided' would tell IDEs like IntelliJ that the dependencies are not required and the job will fail in IntelliJ. (If there is a way to set the dependencies only during the 'package' phase to provided, let me know) I hope somebody here has a solution for us. Regards, Robert
Re: Build fat jar transitively excluding some dependencies
Thank you for the replies. Martin: Will the maven-assemby-plugin also exclude transitive dependencies? Ben: Our users are used to run Flink programs out of the main() method, so IntelliJ will not include the required jars into the classpath. I basically want some dependencies to be 'provided' when the maven-assembly-plugin runs but otherwise they should be in the 'compile' scope. On Tue, Feb 24, 2015 at 8:51 PM, Ben Podgursky bpodgur...@gmail.com wrote: We package up our job jars using maven assemblies using the 'provided' scope to exclude hadoop jars, and use intellij for local development and testing. We've found that it's easiest to just do all local debugging using junit tests since provided jars will be on the classpath there (if you don't want it to be run during actual unit testing you can @Ignore the class). Not super elegant but it works and encourages people to do testing via actual tests, rather than manual scripts. On Tue, Feb 24, 2015 at 4:53 AM, Robert Metzger rmetz...@apache.org wrote: Hi, I'm a committer at the Apache Flink project (a system for distrib. data processing). We provide our users a quickstart maven archetype to bootstrap new Flink jobs. For the generated Flink job's maven project, I would like to build a fat-jar that contains all the dependencies the user added to the project. However, I don't want to include the flink dependencies into the fat jar. The purpose of the fat-jar is to submit it to the cluster for executing the user's job. So it should contain the usercode, all the user's dependencies BUT NOT the flink dependencies, because we can assume them to be available in the running cluster. A fat-jar with Flink's dependencies is 60MB+, which can be annoying when uploading the jars to a cluster. I'm using the shade plugin to do that. So my first idea was to exclude everything in the org.apache.flink groupId from the fat jar. However, this is not possible because - we can only expect some artifacts to be available at runtime (Flink ships the core jars with the binary builds. Extensions have to be loaded by the user) - If users put code in their archetype project into the org.apache.flink namespace, we exclude usercode. So what I'm looking for is a way to tell the shade (or maven assembly) plugin to exclude a list of artifacts and their transitive dependencies. In case someone asks for it: I can not use the 'provided' scope for the Flink dependencies because users can also start (and debug) their Flink jobs locally. Setting the dependencies to 'provided' would tell IDEs like IntelliJ that the dependencies are not required and the job will fail in IntelliJ. (If there is a way to set the dependencies only during the 'package' phase to provided, let me know) I hope somebody here has a solution for us. Regards, Robert
Build fat jar transitively excluding some dependencies
Hi, I'm a committer at the Apache Flink project (a system for distrib. data processing). We provide our users a quickstart maven archetype to bootstrap new Flink jobs. For the generated Flink job's maven project, I would like to build a fat-jar that contains all the dependencies the user added to the project. However, I don't want to include the flink dependencies into the fat jar. The purpose of the fat-jar is to submit it to the cluster for executing the user's job. So it should contain the usercode, all the user's dependencies BUT NOT the flink dependencies, because we can assume them to be available in the running cluster. A fat-jar with Flink's dependencies is 60MB+, which can be annoying when uploading the jars to a cluster. I'm using the shade plugin to do that. So my first idea was to exclude everything in the org.apache.flink groupId from the fat jar. However, this is not possible because - we can only expect some artifacts to be available at runtime (Flink ships the core jars with the binary builds. Extensions have to be loaded by the user) - If users put code in their archetype project into the org.apache.flink namespace, we exclude usercode. So what I'm looking for is a way to tell the shade (or maven assembly) plugin to exclude a list of artifacts and their transitive dependencies. In case someone asks for it: I can not use the 'provided' scope for the Flink dependencies because users can also start (and debug) their Flink jobs locally. Setting the dependencies to 'provided' would tell IDEs like IntelliJ that the dependencies are not required and the job will fail in IntelliJ. (If there is a way to set the dependencies only during the 'package' phase to provided, let me know) I hope somebody here has a solution for us. Regards, Robert
Re: Change dependency version through property
Hi, thank you Ron and Jörg for the responses! Using the dependency management in the common parent pom resolved my problem! Thank you again, Robert On Fri, Nov 14, 2014 at 7:55 PM, Jörg Schaible joerg.schai...@gmx.de wrote: Hi Robert, Robert Metzger wrote: Hi, I'm a developer at the Apache Flink (incubating) project. I'm trying to pass the version of a dependency using a property from the command line to the binary build. But it seems that the version I've set is not respected everywhere. We have the following setup: - flink-parent - flink-core - ... more modules ... - flink-yarn - flink-dist The flink-yarn module (and others) have dependencies to Apache Hadoop. We have a property (called hadoop.version) to have one place for setting the Hadoop version. The default Hadoop version 2.2.0. The flink-dist module is creating the binary release for Flink. It is using the maven-assembly-plugin for that. All jars we depend on are placed in the lib directory. The flink-dist module has most of our modules (also the flink-yarn) module as a dependency so that we can use the modules in the assembly. What I'm trying to do is building Flink for a different Hadoop version, say 2.2.0-cdh5.0.0-beta-2. Therefore, I'm passing this version as -Dhadoop.version=2.2.0-cdh5.0.0-beta-2. However, some files from Hadoop 2.2.0 end up in the lib directory. mvn dependency:tree reveals for flink-dist: [INFO] +- org.apache.flink:flink-yarn:jar:0.8-incubating-SNAPSHOT:compile [INFO] | +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile [INFO] | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile [INFO] | | +- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile however, for flink-yarn the versions are correct: [INFO] +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0-cdh5.0.0-beta-2:compile [INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile [INFO] | +- commons-lang:commons-lang:jar:2.6:compile Here is the full dependency:tree output: https://gist.github.com/rmetzger/70c5f35d4cfd06a91169 How can I create builds for different Hadoop versions using Maven? General rule: - don't use a version tag for direct dependencies - use a common parent for all modules with - a dependency management section for any dependency you want to control - use properties for these versions These properties can be overwritten from command line or setting the property again explicitly in a module. Cheers, Jörg - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Change dependency version through property
Hi, I'm a developer at the Apache Flink (incubating) project. I'm trying to pass the version of a dependency using a property from the command line to the binary build. But it seems that the version I've set is not respected everywhere. We have the following setup: - flink-parent - flink-core - ... more modules ... - flink-yarn - flink-dist The flink-yarn module (and others) have dependencies to Apache Hadoop. We have a property (called hadoop.version) to have one place for setting the Hadoop version. The default Hadoop version 2.2.0. The flink-dist module is creating the binary release for Flink. It is using the maven-assembly-plugin for that. All jars we depend on are placed in the lib directory. The flink-dist module has most of our modules (also the flink-yarn) module as a dependency so that we can use the modules in the assembly. What I'm trying to do is building Flink for a different Hadoop version, say 2.2.0-cdh5.0.0-beta-2. Therefore, I'm passing this version as -Dhadoop.version=2.2.0-cdh5.0.0-beta-2. However, some files from Hadoop 2.2.0 end up in the lib directory. mvn dependency:tree reveals for flink-dist: [INFO] +- org.apache.flink:flink-yarn:jar:0.8-incubating-SNAPSHOT:compile [INFO] | +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile [INFO] | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:compile [INFO] | | +- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile however, for flink-yarn the versions are correct: [INFO] +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0-cdh5.0.0-beta-2:compile [INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile [INFO] | +- commons-lang:commons-lang:jar:2.6:compile Here is the full dependency:tree output: https://gist.github.com/rmetzger/70c5f35d4cfd06a91169 How can I create builds for different Hadoop versions using Maven? Regards, Robert
archetype:integration-test generated archetype build depends on parent project
Hi, we use the archetype:integration-test to test if the archetypes we generate successfully compile. Our archetypes (we call them quickstart) are located within the main project. So we have - parent - core - runtime - clients - quickstart - quickstart-java - quickstart-scala as a project hierarchy. To build the archetype generated by quickstart-java we need the artifacts from core and runtime. So if we do a mvn clean verify on our project, and the user has not executed mvn clean install before (and the artifacts are not available online), the build fails. Is there a way to tell maven to add the artifacts from core and runtime into the tests classpath? Or can we somehow automatically execute the install phase when a user requests the verify phase? You can find our project here: https://github.com/apache/incubator-flink -- Robert
