java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener
Hi I am getting the following error while executing a scala_twitter program for spark 14/11/11 16:39:23 ERROR receiver.ReceiverSupervisorImpl: Stopped executor with error: java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V 14/11/11 16:39:23 ERROR executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0) java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V at org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72) at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121) at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) at org.apache.spark.scheduler.Task.run(Task.scala:54) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 14/11/11 16:39:23 ERROR executor.ExecutorUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-0,5,main] java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V at org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72) at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121) at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) at org.apache.spark.scheduler.Task.run(Task.scala:54) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 1 I think it might be a dependency issue so sharing pom.xml too dependencies dependency groupIdorg.twitter4j/groupId artifactIdtwitter4j-core/artifactId version3.0.3/version /dependency dependency groupIdorg.apache.httpcomponents/groupId artifactIdhttpclient/artifactId version4.0-beta1/version /dependency dependency groupIdorg.apache.httpcomponents/groupId artifactIdhttpclient/artifactId version4.3.5/version /dependency dependency groupIdoauth.signpost/groupId artifactIdsignpost-commonshttp4/artifactId version1.2/version /dependency dependency groupIdorg.scalatest/groupId artifactIdscalatest_2.10/artifactId version3.0.0-SNAP2/version /dependency dependency groupIdcommons-io/groupId artifactIdcommons-io/artifactId version2.4/version /dependency dependency groupIdjunit/groupId artifactIdjunit/artifactId version4.4/version /dependency dependency groupIdorg.twitter4j/groupId artifactIdtwitter4j-stream/artifactId version3.0.3/version /dependency dependency groupIdorg.apache.spark/groupId artifactIdspark-streaming_2.10/artifactId
Re: java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener
You are not having the twitter4j jars in the classpath. While running it in the cluster mode you need to ship those dependency jars. You can do like: sparkConf.setJars(/home/akhld/jars/twitter4j-core-3.0.3.jar, /home/akhld/jars/twitter4j-stream-3.0.3.jar) You can make sure they are shipped by checking the Application WebUI (4040) environment tab. Thanks Best Regards On Tue, Nov 11, 2014 at 5:48 PM, Jishnu Menath Prathap (WT01 - BAS) jishnu.prat...@wipro.com wrote: *Hi I am getting the following error while executing a scala_twitter program for spark* 14/11/11 16:39:23 ERROR receiver.ReceiverSupervisorImpl: Stopped executor with error: java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V 14/11/11 16:39:23 ERROR executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0) java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V at org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72) at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121) at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) at org.apache.spark.scheduler.Task.run(Task.scala:54) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 14/11/11 16:39:23 ERROR executor.ExecutorUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-0,5,main] java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V at org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72) at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121) at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) at org.apache.spark.scheduler.Task.run(Task.scala:54) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 1 *I think it might be a dependency issue so sharing pom.xml too* dependencies dependency groupIdorg.twitter4j/groupId artifactIdtwitter4j-core/artifactId version3.0.3/version /dependency dependency groupIdorg.apache.httpcomponents/groupId artifactIdhttpclient/artifactId version4.0-beta1/version /dependency dependency groupIdorg.apache.httpcomponents/groupId artifactIdhttpclient/artifactId version4.3.5/version /dependency dependency groupIdoauth.signpost/groupId artifactIdsignpost-commonshttp4/artifactId version1.2/version /dependency dependency groupIdorg.scalatest/groupId artifactIdscalatest_2.10/artifactId version3.0.0-SNAP2/version /dependency dependency groupIdcommons-io/groupId artifactIdcommons-io/artifactId version2.4/version /dependency
RE: java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener
Hi Thank you Akhil for reply. I am not using cluster mode I am doing in local mode val sparkConf = new SparkConf().setAppName(TwitterPopularTags).setMaster(local).set(spark.eventLog.enabled,true) Also is there anywhere documented which Twitter4j version to be used for different versions of spark. Thanks Regards Jishnu Menath Prathap From: Akhil [via Apache Spark User List] [mailto:ml-node+s1001560n18576...@n3.nabble.com] Sent: Tuesday, November 11, 2014 6:30 PM To: Jishnu Menath Prathap (WT01 - BAS) Subject: Re: java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener You are not having the twitter4j jars in the classpath. While running it in the cluster mode you need to ship those dependency jars. You can do like: sparkConf.setJars(/home/akhld/jars/twitter4j-core-3.0.3.jar,/home/akhld/jars/twitter4j-stream-3.0.3.jar) You can make sure they are shipped by checking the Application WebUI (4040) environment tab. Thanks Best Regards On Tue, Nov 11, 2014 at 5:48 PM, Jishnu Menath Prathap (WT01 - BAS) [hidden email]/user/SendEmail.jtp?type=nodenode=18576i=0 wrote: Hi I am getting the following error while executing a scala_twitter program for spark 14/11/11 16:39:23 ERROR receiver.ReceiverSupervisorImpl: Stopped executor with error: java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V 14/11/11 16:39:23 ERROR executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0) java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V at org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72) at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121) at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) at org.apache.spark.scheduler.Task.run(Task.scala:54) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 14/11/11 16:39:23 ERROR executor.ExecutorUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-0,5,main] java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V at org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72) at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121) at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) at org.apache.spark.scheduler.Task.run(Task.scala:54) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 1 I think it might be a dependency issue so sharing pom.xml too dependencies dependency groupIdorg.twitter4j/groupId artifactIdtwitter4j-core/artifactId version3.0.3/version /dependency dependency groupIdorg.apache.httpcomponents/groupId artifactIdhttpclient/artifactId version4.0-beta1/version /dependency dependency groupIdorg.apache.httpcomponents/groupId artifactIdhttpclient/artifactId version4.3.5/version /dependency dependency groupIdoauth.signpost/groupId
Re: java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener
You can pick the dependency version from here http://mvnrepository.com/artifact/org.apache.spark/spark-streaming-twitter_2.10 Thanks Best Regards On Tue, Nov 11, 2014 at 6:36 PM, Jishnu Menath Prathap (WT01 - BAS) jishnu.prat...@wipro.com wrote: Hi Thank you Akhil for reply. I am not using cluster mode I am doing in local mode val sparkConf = new SparkConf().setAppName(TwitterPopularTags). *setMaster(local)*.set(spark.eventLog.enabled,true) Also is there anywhere documented which Twitter4j version to be used for different versions of spark. Thanks Regards Jishnu Menath Prathap *From:* Akhil [via Apache Spark User List] [mailto: ml-node+s1001560n18576...@n3.nabble.com] *Sent:* Tuesday, November 11, 2014 6:30 PM *To:* Jishnu Menath Prathap (WT01 - BAS) *Subject:* Re: java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener You are not having the twitter4j jars in the classpath. While running it in the cluster mode you need to ship those dependency jars. You can do like: sparkConf.setJars(/home/akhld/jars/twitter4j-core-3.0.3.jar,/home/akhld/jars/twitter4j-stream-3.0.3.jar) You can make sure they are shipped by checking the Application WebUI (4040) environment tab. Thanks Best Regards On Tue, Nov 11, 2014 at 5:48 PM, Jishnu Menath Prathap (WT01 - BAS) [hidden email] http://user/SendEmail.jtp?type=nodenode=18576i=0 wrote: *Hi I am getting the following error while executing a scala_twitter program for spark* 14/11/11 16:39:23 ERROR receiver.ReceiverSupervisorImpl: Stopped executor with error: java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V 14/11/11 16:39:23 ERROR executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0) java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V at org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72) at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121) at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) at org.apache.spark.scheduler.Task.run(Task.scala:54) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 14/11/11 16:39:23 ERROR executor.ExecutorUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-0,5,main] java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V at org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:72) at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121) at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:264) at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1121) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) at org.apache.spark.scheduler.Task.run(Task.scala:54) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 1 *I think it might be a dependency issue so sharing pom.xml too* dependencies dependency groupIdorg.twitter4j/groupId artifactIdtwitter4j-core/artifactId version3.0.3/version /dependency dependency groupIdorg.apache.httpcomponents/groupId artifactIdhttpclient
