Thanks for the reply Cody. I found the below article on the same, very helpful. Thanks for the details, much appreciated.
http://blog.cloudera.com/blog/2015/03/exactly-once-spark-streaming-from-apache-kafka/ On Tue, Sep 13, 2016 at 8:14 PM, Cody Koeninger <c...@koeninger.org> wrote: > 1. see http://spark.apache.org/docs/latest/streaming-kafka- > integration.html#approach-2-direct-approach-no-receivers > look for HasOffsetRange. If you really want the info per-message > rather than per-partition, createRDD has an overload that takes a > messageHandler from MessageAndMetadata to whatever you need > > 2. createRDD takes type parameters for the key and value decoder, so > specify them there > > 3. you can use spark-streaming-kafka-0-8 against 0.9 or 0.10 brokers. > There is a spark-streaming-kafka-0-10 package with additional features > that only works on brokers 0.10 or higher. A pull request for > documenting it has been merged, but not deployed. > > On Tue, Sep 13, 2016 at 6:46 PM, Mukesh Jha <me.mukesh....@gmail.com> > wrote: > > Hello fellow sparkers, > > > > I'm using spark to consume messages from kafka in a non streaming > fashion. > > I'm suing the using spark-streaming-kafka-0-8_2.10 & sparkv2.0to do the > > same. > > > > I have a few queries for the same, please get back if you guys have > clues on > > the same. > > > > 1) Is there anyway to get the have the topic and partition & offset > > information for each item from the KafkaRDD. I'm using the > > KafkaUtils.createRDD[String, String, StringDecoder, StringDecoder] to > create > > my kafka RDD. > > 2) How to pass my custom Decoder instead of using the String or Byte > decoder > > are there any examples for the same? > > 3) is there a newer version to consumer from kafka-0.10 & kafka-0.9 > clusters > > > > -- > > Thanks & Regards, > > > > Mukesh Jha > -- Thanks & Regards, *Mukesh Jha <me.mukesh....@gmail.com>*
