The presentation at Spark Summit SF was probably referring to Structured Streaming. The existing Spark Streaming (dstream) in Spark 2.0 has the same production stability level as Spark 1.6. There is also Kafka 0.10 support in dstream.
On July 25, 2016 at 10:26:49 AM, Andy Davidson ( a...@santacruzintegration.com) wrote: Hi Kevin Just a heads up at the recent spark summit in S.F. There was a presentation about streaming in 2.0. They said that streaming was not going to production ready in 2.0. I am not sure if the older 1.6.x version will be supported. My project will not be able to upgrade with streaming support. We also use kafka Andy From: Marco Mistroni <mmistr...@gmail.com> Date: Monday, July 25, 2016 at 2:33 AM To: kevin <kiss.kevin...@gmail.com> Cc: "user @spark" <user@spark.apache.org>, "dev.spark" <d...@spark.apache.org > Subject: Re: where I can find spark-streaming-kafka for spark2.0 Hi Kevin you should not need to rebuild everything. Instead, i believe you should launch spark-submit by specifying the kafka jar file in your --packages... i had to follow same when integrating spark streaming with flume have you checked this link ? https://spark.apache.org/docs/latest/streaming-kafka-integration.html hth On Mon, Jul 25, 2016 at 10:20 AM, kevin <kiss.kevin...@gmail.com> wrote: > I have compile it from source code > > 2016-07-25 12:05 GMT+08:00 kevin <kiss.kevin...@gmail.com>: > >> hi,all : >> I try to run example org.apache.spark.examples.streaming.KafkaWordCount , >> I got error : >> Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/spark/streaming/kafka/KafkaUtils$ >> at >> org.apache.spark.examples.streaming.KafkaWordCount$.main(KafkaWordCount.scala:57) >> at >> org.apache.spark.examples.streaming.KafkaWordCount.main(KafkaWordCount.scala) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at >> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:724) >> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.spark.streaming.kafka.KafkaUtils$ >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> ... 11 more >> >> so where I can find spark-streaming-kafka for spark2.0 >> > >
