For 2.0, the kafka dstream support is in two separate subprojects depending on which version of Kafka you are using
spark-streaming-kafka-0-10 or spark-streaming-kafka-0-8 corresponding to brokers that are version 0.10+ or 0.8+ On Mon, Jul 25, 2016 at 12:29 PM, Reynold Xin <r...@databricks.com> wrote: > The presentation at Spark Summit SF was probably referring to Structured > Streaming. The existing Spark Streaming (dstream) in Spark 2.0 has the same > production stability level as Spark 1.6. There is also Kafka 0.10 support in > dstream. > > On July 25, 2016 at 10:26:49 AM, Andy Davidson > (a...@santacruzintegration.com) wrote: > > Hi Kevin > > Just a heads up at the recent spark summit in S.F. There was a presentation > about streaming in 2.0. They said that streaming was not going to production > ready in 2.0. > > I am not sure if the older 1.6.x version will be supported. My project will > not be able to upgrade with streaming support. We also use kafka > > Andy > > From: Marco Mistroni <mmistr...@gmail.com> > Date: Monday, July 25, 2016 at 2:33 AM > To: kevin <kiss.kevin...@gmail.com> > Cc: "user @spark" <u...@spark.apache.org>, "dev.spark" > <dev@spark.apache.org> > Subject: Re: where I can find spark-streaming-kafka for spark2.0 > > Hi Kevin > you should not need to rebuild everything. > Instead, i believe you should launch spark-submit by specifying the kafka > jar file in your --packages... i had to follow same when integrating spark > streaming with flume > > have you checked this link ? > https://spark.apache.org/docs/latest/streaming-kafka-integration.html > > > hth > > > > On Mon, Jul 25, 2016 at 10:20 AM, kevin <kiss.kevin...@gmail.com> wrote: >> >> I have compile it from source code >> >> 2016-07-25 12:05 GMT+08:00 kevin <kiss.kevin...@gmail.com>: >>> >>> hi,all : >>> I try to run example org.apache.spark.examples.streaming.KafkaWordCount , >>> I got error : >>> Exception in thread "main" java.lang.NoClassDefFoundError: >>> org/apache/spark/streaming/kafka/KafkaUtils$ >>> at >>> org.apache.spark.examples.streaming.KafkaWordCount$.main(KafkaWordCount.scala:57) >>> at >>> org.apache.spark.examples.streaming.KafkaWordCount.main(KafkaWordCount.scala) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> at >>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:724) >>> at >>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) >>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) >>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> Caused by: java.lang.ClassNotFoundException: >>> org.apache.spark.streaming.kafka.KafkaUtils$ >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>> ... 11 more >>> >>> so where I can find spark-streaming-kafka for spark2.0 >> >> > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
