Hi,
It looks like you are building from master
(spark-cassandra-connector-assembly-1.2.0).
- Append this to your com.google.guava declaration: % provided
- Be sure your version of the connector dependency is the same as the assembly
build. For instance, if you are using 1.1.0-beta1, build your assembly with
that vs master.
- You can upgrade your version of cassandra if that is plausible for your
deploy environment, to 2.1.0. Side note: we are releasing 1.1.0-beta2 today or
tomorrow which allows usage of Cassandra 2.1.1 and fixes any guava issues
- Make your version of cassandra server + dependencies match your cassandra
driver version. You currently have 2.0.9 with 2.0.4
- Helena
@helenaedelson
On Nov 11, 2014, at 6:13 AM, shahab shahab.mok...@gmail.com wrote:
Hi,
I have a spark application which uses Cassandra
connectorspark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar to load data
from Cassandra into spark.
Everything works fine in the local mode, when I run in my IDE. But when I
submit the application to be executed in standalone Spark server, I get the
following exception, (which is apparently related to Guava versions ???!).
Does any one know how to solve this?
I create a jar file of my spark application using assembly.bat, and the
followings is the dependencies I used:
I put the connectorspark-cassandra-connector-assembly-1.2.0-SNAPSHOT.ja in
the lib/ folder of my eclipse project thats why it is not included in the
dependencies
libraryDependencies ++= Seq(
org.apache.spark%% spark-catalyst% 1.1.0 %
provided,
org.apache.cassandra % cassandra-all % 2.0.9 intransitive(),
org.apache.cassandra % cassandra-thrift % 2.0.9 intransitive(),
net.jpountz.lz4 % lz4 % 1.2.0,
org.apache.thrift % libthrift % 0.9.1 exclude(org.slf4j,
slf4j-api) exclude(javax.servlet, servlet-api),
com.datastax.cassandra % cassandra-driver-core % 2.0.4
intransitive(),
org.apache.spark %% spark-core % 1.1.0 % provided
exclude(org.apache.hadoop, hadoop-core),
org.apache.spark %% spark-streaming % 1.1.0 % provided,
org.apache.hadoop % hadoop-client % 1.0.4 % provided,
com.github.nscala-time %% nscala-time % 1.0.0,
org.scalatest %% scalatest % 1.9.1 % test,
org.apache.spark %% spark-sql % 1.1.0 % provided,
org.apache.spark %% spark-hive % 1.1.0 % provided,
org.json4s %% json4s-jackson % 3.2.5,
junit % junit % 4.8.1 % test,
org.slf4j % slf4j-api % 1.7.7,
org.slf4j % slf4j-simple % 1.7.7,
org.clapper %% grizzled-slf4j % 1.0.2,
log4j % log4j % 1.2.17,
com.google.guava % guava % 16.0
)
best,
/Shahab
And this is the exception I get:
Exception in thread main com.google.common.util.concurrent.ExecutionError:
java.lang.NoSuchMethodError:
com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;
at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2261)
at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
at
org.apache.spark.sql.cassandra.CassandraCatalog.lookupRelation(CassandraCatalog.scala:39)
at
org.apache.spark.sql.cassandra.CassandraSQLContext$$anon$2.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation(CassandraSQLContext.scala:60)
at
org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:123)
at
org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:123)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:123)
at
org.apache.spark.sql.cassandra.CassandraSQLContext$$anon$2.lookupRelation(CassandraSQLContext.scala:65)
