Hi, Thanks for the response, The issue I am facing is only for the clustered Kafka 2.11 based version 0.10.0.1 and Spark 1.6.1 with following dependencies. org.apache.spark:spark-core_2.10:1.6.1 compile group: 'org.apache.spark', name: 'spark-streaming_2.10', version:'1.6.1' compile group: 'org.apache.spark', name: 'spark-streaming-kafka_2.10', version:'1.6.1'
For standalone kafka with rest versions remaining same, I am perfectly able to connect and consume from Kafka using Spark Streaming. DO you know if we have to do anything special while doing createDirectStream/RDD in Clustered 0.10 Kafka consumed via Spark 1.6.1? -Regards Sagar On Fri, Sep 23, 2016 at 7:05 AM, Cody Koeninger <c...@koeninger.org> wrote: > For Spark 2.0 there are two kafka artifacts, > spark-streaming-kafka-0-10 (0.10 and later brokers only) and > spark-streaming-kafka-0-8 (should work with 0.8 and later brokers). > The docs explaining this were merged to master just after 2.0 > released, so they haven't been published yet. > > There are usage examples of the 0-10 connector at > https://github.com/koeninger/kafka-exactly-once > > The change to Spark 2.0 was really straightforward for the one or two > jobs I switched over, for what it's worth. > > On Fri, Sep 23, 2016 at 12:31 AM, sagarcasual . <sagarcas...@gmail.com> > wrote: > > Also you mentioned about streaming-kafka-0-10 connector, what connector > is > > this, do you know the dependency ? I did not see mention of it in the > > documents > > For current Spark 1.6.1 to Kafka 0.10.0.1 standalone, the only > dependencies > > I have are > > > > org.apache.spark:spark-core_2.10:1.6.1 > > compile group: 'org.apache.spark', name: 'spark-streaming_2.10', > > version:'1.6.1' > > compile group: 'org.apache.spark', name: 'spark-streaming-kafka_2.10', > > version:'1.6.1' > > compile group: 'org.apache.spark', name: 'spark-sql_2.10', version: > '1.6.1' > > > > For Spark 2.0 with Kafka 0.10.0.1 do I need to have a different kafka > > connector dependency? > > > > > > On Thu, Sep 22, 2016 at 2:21 PM, sagarcasual . <sagarcas...@gmail.com> > > wrote: > >> > >> Hi Cody, > >> Thanks for the response. > >> One thing I forgot to mention is I am using a Direct Approach (No > >> receivers) in Spark streaming. > >> > >> I am not sure if I have that leverage to upgrade at this point, but do > you > >> know if Spark 1.6.1 to Spark 2.0 jump is smooth usually or does it > involve > >> lot of hick-ups. > >> Also is there a migration guide or something? > >> > >> -Regards > >> Sagar > >> > >> On Thu, Sep 22, 2016 at 1:39 PM, Cody Koeninger <c...@koeninger.org> > >> wrote: > >>> > >>> Do you have the ability to try using Spark 2.0 with the > >>> streaming-kafka-0-10 connector? > >>> > >>> I'd expect the 1.6.1 version to be compatible with kafka 0.10, but it > >>> would be good to rule that out. > >>> > >>> On Thu, Sep 22, 2016 at 1:37 PM, sagarcasual . <sagarcas...@gmail.com> > >>> wrote: > >>> > Hello, > >>> > > >>> > I am trying to stream data out of kafka cluster (2.11_0.10.0.1) using > >>> > Spark > >>> > 1.6.1 > >>> > I am receiving following error, and I confirmed that Topic to which I > >>> > am > >>> > trying to connect exists with the data . > >>> > > >>> > Any idea what could be the case? > >>> > > >>> > kafka.common.UnknownTopicOrPartitionException > >>> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > >>> > Method) > >>> > at > >>> > > >>> > sun.reflect.NativeConstructorAccessorImpl.newInstance( > NativeConstructorAccessorImpl.java:62) > >>> > at > >>> > > >>> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > DelegatingConstructorAccessorImpl.java:45) > >>> > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > >>> > at java.lang.Class.newInstance(Class.java:442) > >>> > at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:102) > >>> > at > >>> > > >>> > org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator. > handleFetchErr(KafkaRDD.scala:184) > >>> > at > >>> > > >>> > org.apache.spark.streaming.kafka.KafkaRDD$ > KafkaRDDIterator.fetchBatch(KafkaRDD.scala:193) > >>> > at > >>> > > >>> > org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext( > KafkaRDD.scala:208) > >>> > at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) > >>> > at > >>> > > >>> > scala.collection.convert.Wrappers$IteratorWrapper. > hasNext(Wrappers.scala:29) > >>> > > >>> > > >> > >> > > > > >
