Hi Helena, It's great to e-meet you! I've actually been following along your blogs and talks trying to get this to work.
I just solved it, and you were absolutely correct. I've been using 1.1.0-alpha3 as my dependency, but my assembly is the 1.2.0-SNAPSHOT. Thanks for looking through all my other threads to piece together the problem. So the solution is that I just included the spark-cassandra-connector-assembly-1.2.0-SNAPSHOT straight into a "lib" folder, removed the line from 1.1.0-alpha3 line from libraryDepenencies, and re-packaged it. It all works great now, and I couldn't be happier. I had to piece together 6 different forums and sites to get that working (being absolutely new to Spark and Scala and sbt). I'll write a blog post on how to get this working later, in case it can help someone. I really appreciate the help! Harold On Tue, Oct 28, 2014 at 11:55 AM, Helena Edelson < helena.edel...@datastax.com> wrote: > Hi Harold, > It seems like, based on your previous post, you are using one version of > the connector as a dependency yet building the assembly jar from master? > You were using 1.1.0-alpha3 (you can upgrade to alpha4, beta coming this > week) yet your assembly is > spark-cassandra-connector-assembly-1.2.0-SNAPSHOT. > > libraryDependencies ++= Seq( > "org.apache.spark" % "spark-streaming_2.10" % "1.1.0", > "com.datastax.spark" %% "spark-cassandra-connector" % "1.1.0-alpha3" > withSources() withJavadoc(), > "org.apache.spark" %% "spark-sql" % "1.1.0" > ) > > - Helena > > On Oct 28, 2014, at 2:08 PM, Harold Nguyen <har...@nexgate.com> wrote: > > > Hi all, > > > > The following works fine when submitting dependency jars through > Spark-Shell: > > > > ./bin/spark-shell --master spark://ip-172-31-38-112:7077 --jars > /home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar > > > > But not through spark-submit: > > > > ./bin/spark-submit --class > "org.apache.spark.examples.streaming.CassandraSave" --master > spark://ip-172-31-38-112:7077 > streaming-test/target/scala-2.10/simple-streaming_2.10-1.0.jar --jars > local:///home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar > > > > Am I issuing the spark-submit command incorrectly ? Each of the workers > has that built jar in their respective directories > (spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar) > > > > Thanks, > > > > Harold > >
