Hi
Thank you Akhil for reply.
I am not using cluster mode I am doing in local mode
val sparkConf = new
SparkConf().setAppName(TwitterPopularTags).setMaster(local).set(spark.eventLog.enabled,true)
Also is there anywhere documented which Twitter4j version to be
used for different versions of spark.
Thanks Regards
Jishnu Menath Prathap
From: Akhil [via Apache Spark User List]
[mailto:ml-node+s1001560n18576...@n3.nabble.com]
Sent: Tuesday, November 11, 2014 6:30 PM
To: Jishnu Menath Prathap (WT01 - BAS)
Subject: Re: java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener
You are not having the twitter4j jars in the classpath. While running it in the
cluster mode you need to ship those dependency jars.
You can do like:
sparkConf.setJars(/home/akhld/jars/twitter4j-core-3.0.3.jar,/home/akhld/jars/twitter4j-stream-3.0.3.jar)
You can make sure they are shipped by checking the Application WebUI (4040)
environment tab.
Thanks
Best Regards
On Tue, Nov 11, 2014 at 5:48 PM, Jishnu Menath Prathap (WT01 - BAS) [hidden
email]/user/SendEmail.jtp?type=nodenode=18576i=0 wrote:
Hi I am getting the following error while executing a scala_twitter program for
spark
14/11/11 16:39:23 ERROR receiver.ReceiverSupervisorImpl: Stopped executor with
error: java.lang.NoSuchMethodError:
twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V
14/11/11 16:39:23 ERROR executor.Executor: Exception in task 0.0 in stage 0.0
(TID 0)
java.lang.NoSuchMethodError:
twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V
at
org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72)
at
org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121)
at
org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106)
at
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264)
at
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257)
at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
at org.apache.spark.scheduler.Task.run(Task.scala:54)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
14/11/11 16:39:23 ERROR executor.ExecutorUncaughtExceptionHandler: Uncaught
exception in thread Thread[Executor task launch worker-0,5,main]
java.lang.NoSuchMethodError:
twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V
at
org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72)
at
org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121)
at
org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106)
at
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264)
at
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257)
at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
at
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
at org.apache.spark.scheduler.Task.run(Task.scala:54)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
1
I think it might be a dependency issue so sharing pom.xml too
dependencies
dependency
groupIdorg.twitter4j/groupId
artifactIdtwitter4j-core/artifactId
version3.0.3/version
/dependency
dependency
groupIdorg.apache.httpcomponents/groupId
artifactIdhttpclient/artifactId
version4.0-beta1/version
/dependency
dependency
groupIdorg.apache.httpcomponents/groupId
artifactIdhttpclient/artifactId
version4.3.5/version
/dependency
dependency
groupIdoauth.signpost/groupId
