Re: Gradle dependency problem with spark
@Sean perhaps I could leverage this when this
http://openjdk.java.net/jeps/261 becomes available.
On Fri, Dec 16, 2016 at 4:05 AM, Steve Loughran
wrote:
> FWIW, although the underlying Hadoop declared guava dependency is pretty
> low, everything in org.apache.hadoop is set up to run against later
> versions. It just sticks with the old one to avoid breaking anything
> donwstream which does expect a low version number. See HADOOP-10101 for the
> ongoing pain there —and complain on there if you do find something in the
> Hadoop layer which can't handle later guava versions.
>
>
>
>
> On 16 Dec 2016, at 11:07, Sean Owen wrote:
>
> Yes, that's the problem. Guava isn't generally mutually compatible across
> more than a couple major releases. You may have to hunt for a version that
> happens to have the functionality that both dependencies want, and hope
> that exists. Spark should shade Guava at this point but doesn't mean that
> you won't hit this problem from transitive dependencies.
>
> On Fri, Dec 16, 2016 at 11:05 AM kant kodali wrote:
>
>> I replaced *guava-14.0.1.jar* with *guava-19.0.jar in *SPARK_HOME/jars
>> and seem to work ok but I am not sure if it is the right thing to do. My
>> fear is that if Spark uses features from Guava that are only present in
>> 14.0.1 but not in 19.0 I guess my app will break.
>>
>>
>>
>> On Fri, Dec 16, 2016 at 2:22 AM, kant kodali wrote:
>>
>> Hi Guys,
>>
>> Here is the simplified version of my problem. I have the following
>> problem and I new to gradle
>>
>>
>> dependencies {
>> compile group: 'org.apache.spark', name: 'spark-core_2.11', version:
>> '2.0.2'
>> compile group: 'com.github.brainlag', name: 'nsq-client', version:
>> '1.0.0.RC2'
>> }
>>
>>
>> I took out the other dependencies for simplicity. The problem here
>> is spark-core_2.11 uses com.google.guava:14.0.1 and nsq-client uses
>> com.google.guava:19.0 so when I submit my fat uber jar using spark-submit I
>> get the following error
>>
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;
>> at com.github.brainlag.nsq.NSQProducer.(NSQProducer.java:22)
>> at com.hello.streamprocessing.app.SparkDriver2.main(SparkDriver2.java:37)
>>
>>
>> any help would be great. Also if you need more description you can find
>> it here
>>
>>
>> Thanks!
>>
>>
>>
>
Re: Gradle dependency problem with spark
FWIW, although the underlying Hadoop declared guava dependency is pretty low,
everything in org.apache.hadoop is set up to run against later versions. It
just sticks with the old one to avoid breaking anything donwstream which does
expect a low version number. See HADOOP-10101 for the ongoing pain there —and
complain on there if you do find something in the Hadoop layer which can't
handle later guava versions.
On 16 Dec 2016, at 11:07, Sean Owen
mailto:so...@cloudera.com>> wrote:
Yes, that's the problem. Guava isn't generally mutually compatible across more
than a couple major releases. You may have to hunt for a version that happens
to have the functionality that both dependencies want, and hope that exists.
Spark should shade Guava at this point but doesn't mean that you won't hit this
problem from transitive dependencies.
On Fri, Dec 16, 2016 at 11:05 AM kant kodali
mailto:kanth...@gmail.com>> wrote:
I replaced guava-14.0.1.jar with guava-19.0.jar in SPARK_HOME/jars and seem to
work ok but I am not sure if it is the right thing to do. My fear is that if
Spark uses features from Guava that are only present in 14.0.1 but not in 19.0
I guess my app will break.
On Fri, Dec 16, 2016 at 2:22 AM, kant kodali
mailto:kanth...@gmail.com>> wrote:
Hi Guys,
Here is the simplified version of my problem. I have the following problem and
I new to gradle
dependencies {
compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.0.2'
compile group: 'com.github.brainlag', name: 'nsq-client', version:
'1.0.0.RC2'
}
I took out the other dependencies for simplicity. The problem here is
spark-core_2.11 uses com.google.guava:14.0.1 and nsq-client uses
com.google.guava:19.0 so when I submit my fat uber jar using spark-submit I get
the following error
Exception in thread "main" java.lang.NoSuchMethodError:
com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;
at com.github.brainlag.nsq.NSQProducer.(NSQProducer.java:22)
at com.hello.streamprocessing.app.SparkDriver2.main(SparkDriver2.java:37)
any help would be great. Also if you need more description you can find it
here
Thanks!
Re: Gradle dependency problem with spark
Yes, that's the problem. Guava isn't generally mutually compatible across
more than a couple major releases. You may have to hunt for a version that
happens to have the functionality that both dependencies want, and hope
that exists. Spark should shade Guava at this point but doesn't mean that
you won't hit this problem from transitive dependencies.
On Fri, Dec 16, 2016 at 11:05 AM kant kodali wrote:
> I replaced *guava-14.0.1.jar* with *guava-19.0.jar in *SPARK_HOME/jars
> and seem to work ok but I am not sure if it is the right thing to do. My
> fear is that if Spark uses features from Guava that are only present in
> 14.0.1 but not in 19.0 I guess my app will break.
>
>
>
> On Fri, Dec 16, 2016 at 2:22 AM, kant kodali wrote:
>
> Hi Guys,
>
> Here is the simplified version of my problem. I have the following problem
> and I new to gradle
>
>
> dependencies {
> compile group: 'org.apache.spark', name: 'spark-core_2.11', version:
> '2.0.2'
> compile group: 'com.github.brainlag', name: 'nsq-client', version:
> '1.0.0.RC2'
> }
>
>
> I took out the other dependencies for simplicity. The problem here
> is spark-core_2.11 uses com.google.guava:14.0.1 and nsq-client uses
> com.google.guava:19.0 so when I submit my fat uber jar using spark-submit I
> get the following error
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;
> at com.github.brainlag.nsq.NSQProducer.(NSQProducer.java:22)
> at com.hello.streamprocessing.app.SparkDriver2.main(SparkDriver2.java:37)
>
>
> any help would be great. Also if you need more description you can find it
> here
>
>
> Thanks!
>
>
>
Re: Gradle dependency problem with spark
I replaced *guava-14.0.1.jar* with *guava-19.0.jar in *SPARK_HOME/jars and
seem to work ok but I am not sure if it is the right thing to do. My fear
is that if Spark uses features from Guava that are only present in 14.0.1
but not in 19.0 I guess my app will break.
On Fri, Dec 16, 2016 at 2:22 AM, kant kodali wrote:
> Hi Guys,
>
> Here is the simplified version of my problem. I have the following problem
> and I new to gradle
>
>
> dependencies {
> compile group: 'org.apache.spark', name: 'spark-core_2.11', version:
> '2.0.2'
> compile group: 'com.github.brainlag', name: 'nsq-client', version:
> '1.0.0.RC2'
> }
>
>
> I took out the other dependencies for simplicity. The problem here
> is spark-core_2.11 uses com.google.guava:14.0.1 and nsq-client uses
> com.google.guava:19.0 so when I submit my fat uber jar using spark-submit I
> get the following error
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;
> at com.github.brainlag.nsq.NSQProducer.(NSQProducer.java:22)
> at com.hello.streamprocessing.app.SparkDriver2.main(SparkDriver2.java:37)
>
>
> any help would be great. Also if you need more description you can find it
> here
>
>
> Thanks!
>
>
