Thanks a lot. It worked after keeping all versions to same.1.2.0 On Wed, Jun 24, 2015 at 2:24 AM, Tathagata Das <t...@databricks.com> wrote:
> Why are you mixing spark versions between streaming and core?? > Your core is 1.2.0 and streaming is 1.4.0. > > On Tue, Jun 23, 2015 at 1:32 PM, Shushant Arora <shushantaror...@gmail.com > > wrote: > >> It throws exception for WriteAheadLogUtils after excluding core and >> streaming jar. >> >> Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/spark/streaming/util/WriteAheadLogUtils$ >> at >> org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:84) >> at >> org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:65) >> at >> org.apache.spark.streaming.kafka.KafkaUtils$.createStream(KafkaUtils.scala:103) >> at >> org.apache.spark.streaming.kafka.KafkaUtils.createStream(KafkaUtils.scala) >> at >> com.adobe.hadoop.saprk.sample.SampleSparkStreamApp.main(SampleSparkStreamApp.java:25) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> >> pom.xml is : >> >> <project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi=" >> http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation=" >> http://maven.apache.org/POM/4.0.0 >> http://maven.apache.org/xsd/maven-4.0.0.xsd";> >> <modelVersion>4.0.0</modelVersion> >> <groupId>xxxx</groupId> >> <artifactId>SampleSparkStreamApp</artifactId> >> <version>1.0</version> >> >> >> <dependencies> >> <dependency> >> <groupId>org.apache.spark</groupId> >> <artifactId>spark-core_2.10</artifactId> >> <version>1.2.0</version> >> <scope>provided</scope> >> </dependency> >> <dependency> >> <groupId>org.apache.spark</groupId> >> <artifactId>spark-streaming-kafka_2.10</artifactId> >> <version>1.4.0</version> >> </dependency> >> <dependency> >> <groupId>org.apache.spark</groupId> >> <artifactId>spark-streaming_2.10</artifactId> >> <scope>provided</scope> >> <version>1.4.0</version> >> </dependency> >> </dependencies> >> <build> >> <plugins> >> <!-- any other plugins --> >> <plugin> >> <artifactId>maven-assembly-plugin</artifactId> >> <executions> >> <execution> >> <phase>package</phase> >> <goals> >> <goal>single</goal> >> </goals> >> </execution> >> </executions> >> <configuration> >> <descriptorRefs> >> <descriptorRef>jar-with-dependencies</descriptorRef> >> </descriptorRefs> >> </configuration> >> </plugin> >> </plugins> >> </build> >> >> </project> >> >> And when I pass streaming jar using --jar option , it threw >> same java.lang.NoClassDefFoundError: org/apache/spark/util/ThreadUtils$. >> >> Thanks >> >> On Wed, Jun 24, 2015 at 1:17 AM, Tathagata Das <t...@databricks.com> >> wrote: >> >>> You must not include spark-core and spark-streaming in the assembly. >>> They are already present in the installation and the presence of multiple >>> versions of spark may throw off the classloaders in weird ways. So make the >>> assembly marking the those dependencies as scope=provided. >>> >>> >>> >>> On Tue, Jun 23, 2015 at 11:56 AM, Shushant Arora < >>> shushantaror...@gmail.com> wrote: >>> >>>> hi >>>> >>>> While using spark streaming (1.2) with kafka . I am getting below error >>>> and receivers are getting killed but jobs get scheduled at each stream >>>> interval. >>>> >>>> 15/06/23 18:42:35 WARN TaskSetManager: Lost task 0.1 in stage 18.0 (TID >>>> 82, ip(XXXXXX)): java.io.IOException: Failed to connect to ip(XXXXXXXX) >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:191) >>>> at >>>> org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:156) >>>> at >>>> org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:78) >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140) >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43) >>>> at >>>> org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:744) >>>> >>>> >>>> 15/06/23 18:42:36 ERROR ReceiverTracker: Deregistered receiver for >>>> stream 0: Error starting receiver 0 - java.lang.NoClassDefFoundError: >>>> org/apache/spark/util/ThreadUtils$ >>>> at >>>> org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:115) >>>> at >>>> org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121) >>>> at >>>> org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106) >>>> at >>>> org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:277) >>>> at >>>> org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:269) >>>> at >>>> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1319) >>>> at >>>> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1319) >>>> at >>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) >>>> at org.apache.spark.scheduler.Task.run(Task.scala:56) >>>> at >>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:744) >>>> >>>> >>>> I created jar with include all dependencies. Which jar is missing here >>>> ? >>>> >>>> >>>> >>>> >>>> >>> >> >
