You can pick the dependency version from here
http://mvnrepository.com/artifact/org.apache.spark/spark-streaming-twitter_2.10
Thanks
Best Regards
On Tue, Nov 11, 2014 at 6:36 PM, Jishnu Menath Prathap (WT01 - BAS) <
jishnu.prat...@wipro.com> wrote:
> Hi
>
> Thank you Akhil for reply.
>
> I am not using cluster mode I am doing in local mode
>
> val sparkConf = new SparkConf().setAppName("TwitterPopularTags").
> *setMaster("local")*.set("spark.eventLog.enabled","true")
>
> Also is there anywhere documented which Twitter4j version to
> be used for different versions of spark.
>
>
>
> Thanks & Regards
>
> Jishnu Menath Prathap
>
>
>
> *From:* Akhil [via Apache Spark User List] [mailto:
> ml-node+s1001560n18576...@n3.nabble.com]
> *Sent:* Tuesday, November 11, 2014 6:30 PM
> *To:* Jishnu Menath Prathap (WT01 - BAS)
> *Subject:* Re: java.lang.NoSuchMethodError:
> twitter4j.TwitterStream.addListener
>
>
>
> You are not having the twitter4j jars in the classpath. While running it
> in the cluster mode you need to ship those dependency jars.
>
>
>
> You can do like:
>
>
>
>
> sparkConf.setJars("/home/akhld/jars/twitter4j-core-3.0.3.jar,/home/akhld/jars/twitter4j-stream-3.0.3.jar")
>
>
>
> You can make sure they are shipped by checking the Application WebUI
> (4040) environment tab.
>
>
>
>
> Thanks
>
> Best Regards
>
>
>
> On Tue, Nov 11, 2014 at 5:48 PM, Jishnu Menath Prathap (WT01 - BAS) <[hidden
> email] <http://user/SendEmail.jtp?type=node&node=18576&i=0>> wrote:
>
> *Hi I am getting the following error while executing a scala_twitter
> program for spark*
> 14/11/11 16:39:23 ERROR receiver.ReceiverSupervisorImpl: Stopped executor
> with error: java.lang.NoSuchMethodError:
> twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V
> 14/11/11 16:39:23 ERROR executor.Executor: Exception in task 0.0 in stage
> 0.0 (TID 0)
> java.lang.NoSuchMethodError:
> twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V
> at
> org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72)
>
> at
> org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121)
>
> at
> org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106)
>
> at
> org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264)
>
> at
> org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257)
>
> at
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
>
> at
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
>
> at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
> at org.apache.spark.scheduler.Task.run(Task.scala:54)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
> 14/11/11 16:39:23 ERROR executor.ExecutorUncaughtExceptionHandler:
> Uncaught exception in thread Thread[Executor task launch worker-0,5,main]
> java.lang.NoSuchMethodError:
> twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V
> at
> org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72)
>
> at
> org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121)
>
> at
> org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106)
>
> at
> org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264)
>
> at
> org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257)
>
> at
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
>
> at
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
>
> at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
> at org.apache.spark.scheduler.Task.run(Task.scala:54)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
> 1
> *I think it might be a dependency issue so sharing pom.xml too*
>
> <dependencies>
> <dependency>
> <groupId>org.twitter4j</groupId>
> <artifactId>twitter4j-core</artifactId>
> <version>3.0.3</version>
> </dependency>
> <dependency>
> <groupId>org.apache.httpcomponents</groupId>
> <artifactId>httpclient</artifactId>
> <version>4.0-beta1</version>
> </dependency>
> <dependency>
> <groupId>org.apache.httpcomponents</groupId>
> <artifactId>httpclient</artifactId>
> <version>4.3.5</version>
> </dependency>
> <dependency>
> <groupId>oauth.signpost</groupId>
> <artifactId>signpost-commonshttp4</artifactId>
> <version>1.2</version>
> </dependency>
> <dependency>
> <groupId>org.scalatest</groupId>
> <artifactId>scalatest_2.10</artifactId>
> <version>3.0.0-SNAP2</version>
> </dependency>
> <dependency>
> <groupId>commons-io</groupId>
> <artifactId>commons-io</artifactId>
> <version>2.4</version>
> </dependency>
> <dependency>
> <groupId>junit</groupId>
> <artifactId>junit</artifactId>
> <version>4.4</version>
> </dependency>
> <dependency>
> <groupId>org.twitter4j</groupId>
> <artifactId>twitter4j-stream</artifactId>
> <version>3.0.3</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-streaming_2.10</artifactId>
> <version>1.1.0</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-core_2.10</artifactId>
> <version>1.1.0</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
>
> <artifactId>spark-streaming-twitter_2.10</artifactId>
> <version>1.1.0</version>
> </dependency>
>
> </dependencies>
>
> * Code of the twitter streaming spark *
>
> object sparktwitteruser {
> def main(args: Array[String]) {
>
>
> val filters = Array("hi")
> // Set the system properties so
> that Twitter4j library used by twitter stream
> // can use them to generat OAuth
> credentials
>
>
> System.setProperty("twitter4j.oauth.consumerKey", "******")
>
> System.setProperty("twitter4j.oauth.consumerSecret","*****")
>
> System.setProperty("twitter4j.oauth.accessToken","*******")
>
> System.setProperty("twitter4j.oauth.accessTokenSecret", "*******")
>
> val sparkConf = new
> SparkConf().setAppName("TwitterPopularTags").setMaster("local").set("spark.eventLog.enabled","true")
>
> val ssc = new
> StreamingContext(sparkConf, Seconds(2))
> val stream =
> TwitterUtils.createStream(ssc, None, filters)
> val hashTags =
> stream.flatMap(status => status.getText.split("
> ").filter(_.startsWith("#")))
> val topCounts60 = hashTags.map((_,
> 1)).reduceByKeyAndWindow(_ + _, Seconds(60))
> .map{case (topic, count) =>
> (count, topic)}
> .transform(_.sortByKey(false))
>
> val topCounts10 = hashTags.map((_,
> 1)).reduceByKeyAndWindow(_ + _, Seconds(10))
> .map{case (topic, count) =>
> (count, topic)}
> .transform(_.sortByKey(false))
> // Print popular hashtags
>
> topCounts60.foreachRDD(rdd => {
> val topList = rdd.take(10)
>
> println("\nPopular topics in last 60 seconds (%s
> total):".format(rdd.count()))
>
> topList.foreach{case (count, tag) => println("%s (%s tweets)".format(tag,
> count))}
> })
>
> topCounts10.foreachRDD(rdd => {
> val topList = rdd.take(10)
>
> println("\nPopular topics in last 10 seconds (%s
> total):".format(rdd.count()))
>
> topList.foreach{case (count, tag) => println("%s (%s tweets)".format(tag,
> count))}
> })
>
> ssc.start()
> ssc.awaitTermination()
> }
> }
>
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments.
>
> WARNING: Computer viruses can be transmitted via email. The recipient
> should check this email and any attachments for the presence of viruses.
> The company accepts no liability for any damage caused by any virus
> transmitted by this email.
>
> www.wipro.com
>
>
>
>
> ------------------------------
>
> *If you reply to this email, your message will be added to the discussion
> below:*
>
>
> http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-NoSuchMethodError-twitter4j-TwitterStream-addListener-tp18571p18576.html
>
> To unsubscribe from java.lang.NoSuchMethodError:
> twitter4j.TwitterStream.addListener, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=18571&code=amlzaG51LnByYXRoYXBAd2lwcm8uY29tfDE4NTcxfC0xNzUwOTc3MjE3>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments.
>
> WARNING: Computer viruses can be transmitted via email. The recipient
> should check this email and any attachments for the presence of viruses.
> The company accepts no liability for any damage caused by any virus
> transmitted by this email.
>
> www.wipro.com
>
