Re: sbt package failed: wrong libraryDependencies for spark-streaming?
Hi Tathagata, I was using the "raw" tag in the web-editor. Seems like this doesn't make it into the mail. Here's the message again, this time without those tags: I've added the following to my spark-env.sh: SPARK_CLASSPATH="/disk.b/spark-master-2014-07-28/external/twitter/target/spark-streaming-twitter_2.10-1.1.0-SNAPSHOT.jar" I can now execute import org.apache.spark.streaming.twitter._ import org.apache.spark.streaming.StreamingContext._ without an error in the shell. However, I will get an error when doing this: scala> val ssc = new StreamingContext(sc, Seconds(1)) ssc: org.apache.spark.streaming.StreamingContext = org.apache.spark.streaming.StreamingContext@6e78177b scala> val tweets = TwitterUtils.createStream(ssc, "twitter.txt") error: bad symbolic reference. A signature in TwitterUtils.class refers to term twitter4j in package which is not available. It may be completely missing from the current classpath, or the version on the classpath might be incompatible with the version used when compiling TwitterUtils.class. What am I missing? Do I have to import another jar? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sbt-package-failed-wrong-libraryDependencies-for-spark-streaming-tp11103p6.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: sbt package failed: wrong libraryDependencies for spark-streaming?
Hey Simon, The stuff you are trying to show - logs, contents of spark-env.sh, etc. are missing from the email. At least I am not able to see it (viewing through gmail). Are you pasting screenshots? Those might get blocked out somehow! TD On Thu, Jul 31, 2014 at 6:55 PM, durin wrote: > I've added the following to my spark-env.sh: > > > I can now execute > without an error in the shell. However, I will get an error when doing this: > > > > > What am I missing? Do I have to import another jar? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/sbt-package-failed-wrong-libraryDependencies-for-spark-streaming-tp11103p11108.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: sbt package failed: wrong libraryDependencies for spark-streaming?
I've added the following to my spark-env.sh: I can now execute without an error in the shell. However, I will get an error when doing this: What am I missing? Do I have to import another jar? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sbt-package-failed-wrong-libraryDependencies-for-spark-streaming-tp11103p11108.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: sbt package failed: wrong libraryDependencies for spark-streaming?
Hi Tathagata, I didn't mean to say this was an error. According to the other thread I linked, right now there shouldn't be any conflicts, so I wanted to use streaming in the shell for easy testing. I thought I had to create my own project in which I'd add streaming as a dependency, but if I can add it into the config that' even simpler and gets rid of my sbt problem. I'll try that. Simon -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sbt-package-failed-wrong-libraryDependencies-for-spark-streaming-tp11103p11106.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: sbt package failed: wrong libraryDependencies for spark-streaming?
I dont see the error. The twitter stuff (as well as kafka and flume stuff) are treated as "external" projects and are not included in the spark shell. This is because we dont want the dependencies of such non-core functionalities to cause random conflicts with that of core spark. Hence its not possible to use twitter spark streaming in the spark shell. Unless you explicitly add the jar generated in external/twitter/target/scala... To the spark shells extra class path (see spark config page) TD On Thu, Jul 31, 2014 at 5:48 PM, durin wrote: > As suggested here > < http://apache-spark-user-list.1001560.n3.nabble.com/import-org-apache-spark-streaming-twitter-in-Shell-tp9665p9743.html > > , I want to create a minimal project using sbt to be able to use > org.apache.spark.streaming.twitter in the shell. My Spark version is the > latest Master branch compiled with maven3. > > I was following this > < http://spark.apache.org/docs/latest/quick-start.html#standalone-applications > > tutorial to create a small standalone application. > > My simple.sbt file looks like this: > > > > > However, I will get this error when using sbt package: > > > > > I don't quite understand the whole standalone thing yet. sbt is installed on > a different machine than my Spark Cluster, but the tutorial never mentions > any variables that would have to be set. So I did assume sbt didn't actually > need a Spark installation on the machine. > Also, I could imagine that 1.0.1 isn't the correct version when I compiled > it myself. > > > Can someone help me out? In the end, all I want for now is to be able to use > twitterUtils in the spark-shell. > > > Simon > > > > -- > View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sbt-package-failed-wrong-libraryDependencies-for-spark-streaming-tp11103.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.
sbt package failed: wrong libraryDependencies for spark-streaming?
As suggested here <http://apache-spark-user-list.1001560.n3.nabble.com/import-org-apache-spark-streaming-twitter-in-Shell-tp9665p9743.html> , I want to create a minimal project using sbt to be able to use org.apache.spark.streaming.twitter in the shell. My Spark version is the latest Master branch compiled with maven3. I was following this <http://spark.apache.org/docs/latest/quick-start.html#standalone-applications> tutorial to create a small standalone application. My simple.sbt file looks like this: However, I will get this error when using sbt package: I don't quite understand the whole standalone thing yet. sbt is installed on a different machine than my Spark Cluster, but the tutorial never mentions any variables that would have to be set. So I did assume sbt didn't actually need a Spark installation on the machine. Also, I could imagine that 1.0.1 isn't the correct version when I compiled it myself. Can someone help me out? In the end, all I want for now is to be able to use twitterUtils in the spark-shell. Simon -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sbt-package-failed-wrong-libraryDependencies-for-spark-streaming-tp11103.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
