Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts
Per offline discussion with Robert: He or I will get onto this, probably next week. On Thu, Feb 19, 2015 at 10:07 AM, Till Rohrmann wrote: > Dependency conflicts were also the reason why we have to use a different > Akka version for the Hadoop 2.0.0-alpha build profile. > > Thus, +1. > > On Wed, Feb 18, 2015 at 3:48 PM, Robert Metzger > wrote: > > > I'm also in favor of shading commonly used libraries to resolve this > issue > > for our upstream users. > > > > I recently wrote this distributed TPC-H datagenerator, which had a hard > > dependency on a newer guava version. So I needed to shade guava in my > > project to make it work. > > Another candidate to shade is the ASM library. > > > > So you have my +1 for doing this. > > > > On Tue, Feb 17, 2015 at 4:26 PM, Stephan Ewen wrote: > > > > > I think the way we shaded Guava is a problem for the way IntelliJ uses > > > maven (compile dependency projects, not package them). > > > > > > Since we do not apply relocation to our code for this, it should have > no > > > effect on the IDE usability. > > > > > > On Tue, Feb 17, 2015 at 3:37 PM, Ufuk Celebi wrote: > > > > > > > > > > > On 17 Feb 2015, at 09:40, Stephan Ewen wrote: > > > > > > > > > Hi everyone! > > > > > > > > > > We have been time and time again struck by the problem that Hadoop > > > > bundles > > > > > many dependencies in certain versions, that conflict either with > > > versions > > > > > of the dependencies we use, or with versions that users use. > > > > > > > > > > The most prominent examples are Guava and Protobuf. > > > > > > > > > > One way to solve this is to create ourselves a "custom" Hadoop > > > > dependency: > > > > > We build a fat jar from Hadoop (with all its dependencies) and in > > this > > > > fat > > > > > jar shade Guava and Protobuf. > > > > > > > > > > We can use the flink-shading project for this - it is already used > to > > > > build > > > > > a custom (shaded) version of Guava for Flink. > > > > > > > > > > Any opinions on this? > > > > > > > > I'm not familiar with all the details, but from past experience I > > recall > > > > that this has been an issue for various users. I don't see all > > > implications > > > > of such a change (are there any down sides?), but in general I think > > that > > > > it will improve the overall user experience. Hence: +1. > > > > > >
Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts
Dependency conflicts were also the reason why we have to use a different Akka version for the Hadoop 2.0.0-alpha build profile. Thus, +1. On Wed, Feb 18, 2015 at 3:48 PM, Robert Metzger wrote: > I'm also in favor of shading commonly used libraries to resolve this issue > for our upstream users. > > I recently wrote this distributed TPC-H datagenerator, which had a hard > dependency on a newer guava version. So I needed to shade guava in my > project to make it work. > Another candidate to shade is the ASM library. > > So you have my +1 for doing this. > > On Tue, Feb 17, 2015 at 4:26 PM, Stephan Ewen wrote: > > > I think the way we shaded Guava is a problem for the way IntelliJ uses > > maven (compile dependency projects, not package them). > > > > Since we do not apply relocation to our code for this, it should have no > > effect on the IDE usability. > > > > On Tue, Feb 17, 2015 at 3:37 PM, Ufuk Celebi wrote: > > > > > > > > On 17 Feb 2015, at 09:40, Stephan Ewen wrote: > > > > > > > Hi everyone! > > > > > > > > We have been time and time again struck by the problem that Hadoop > > > bundles > > > > many dependencies in certain versions, that conflict either with > > versions > > > > of the dependencies we use, or with versions that users use. > > > > > > > > The most prominent examples are Guava and Protobuf. > > > > > > > > One way to solve this is to create ourselves a "custom" Hadoop > > > dependency: > > > > We build a fat jar from Hadoop (with all its dependencies) and in > this > > > fat > > > > jar shade Guava and Protobuf. > > > > > > > > We can use the flink-shading project for this - it is already used to > > > build > > > > a custom (shaded) version of Guava for Flink. > > > > > > > > Any opinions on this? > > > > > > I'm not familiar with all the details, but from past experience I > recall > > > that this has been an issue for various users. I don't see all > > implications > > > of such a change (are there any down sides?), but in general I think > that > > > it will improve the overall user experience. Hence: +1. > > >
Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts
I'm also in favor of shading commonly used libraries to resolve this issue for our upstream users. I recently wrote this distributed TPC-H datagenerator, which had a hard dependency on a newer guava version. So I needed to shade guava in my project to make it work. Another candidate to shade is the ASM library. So you have my +1 for doing this. On Tue, Feb 17, 2015 at 4:26 PM, Stephan Ewen wrote: > I think the way we shaded Guava is a problem for the way IntelliJ uses > maven (compile dependency projects, not package them). > > Since we do not apply relocation to our code for this, it should have no > effect on the IDE usability. > > On Tue, Feb 17, 2015 at 3:37 PM, Ufuk Celebi wrote: > > > > > On 17 Feb 2015, at 09:40, Stephan Ewen wrote: > > > > > Hi everyone! > > > > > > We have been time and time again struck by the problem that Hadoop > > bundles > > > many dependencies in certain versions, that conflict either with > versions > > > of the dependencies we use, or with versions that users use. > > > > > > The most prominent examples are Guava and Protobuf. > > > > > > One way to solve this is to create ourselves a "custom" Hadoop > > dependency: > > > We build a fat jar from Hadoop (with all its dependencies) and in this > > fat > > > jar shade Guava and Protobuf. > > > > > > We can use the flink-shading project for this - it is already used to > > build > > > a custom (shaded) version of Guava for Flink. > > > > > > Any opinions on this? > > > > I'm not familiar with all the details, but from past experience I recall > > that this has been an issue for various users. I don't see all > implications > > of such a change (are there any down sides?), but in general I think that > > it will improve the overall user experience. Hence: +1. >
Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts
I think the way we shaded Guava is a problem for the way IntelliJ uses maven (compile dependency projects, not package them). Since we do not apply relocation to our code for this, it should have no effect on the IDE usability. On Tue, Feb 17, 2015 at 3:37 PM, Ufuk Celebi wrote: > > On 17 Feb 2015, at 09:40, Stephan Ewen wrote: > > > Hi everyone! > > > > We have been time and time again struck by the problem that Hadoop > bundles > > many dependencies in certain versions, that conflict either with versions > > of the dependencies we use, or with versions that users use. > > > > The most prominent examples are Guava and Protobuf. > > > > One way to solve this is to create ourselves a "custom" Hadoop > dependency: > > We build a fat jar from Hadoop (with all its dependencies) and in this > fat > > jar shade Guava and Protobuf. > > > > We can use the flink-shading project for this - it is already used to > build > > a custom (shaded) version of Guava for Flink. > > > > Any opinions on this? > > I'm not familiar with all the details, but from past experience I recall > that this has been an issue for various users. I don't see all implications > of such a change (are there any down sides?), but in general I think that > it will improve the overall user experience. Hence: +1.
Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts
On 17 Feb 2015, at 09:40, Stephan Ewen wrote: > Hi everyone! > > We have been time and time again struck by the problem that Hadoop bundles > many dependencies in certain versions, that conflict either with versions > of the dependencies we use, or with versions that users use. > > The most prominent examples are Guava and Protobuf. > > One way to solve this is to create ourselves a "custom" Hadoop dependency: > We build a fat jar from Hadoop (with all its dependencies) and in this fat > jar shade Guava and Protobuf. > > We can use the flink-shading project for this - it is already used to build > a custom (shaded) version of Guava for Flink. > > Any opinions on this? I'm not familiar with all the details, but from past experience I recall that this has been an issue for various users. I don't see all implications of such a change (are there any down sides?), but in general I think that it will improve the overall user experience. Hence: +1.
