Hi

I am processing kafka messages using spark streaming 1.3.

I am using mapPartitions function to process kafka message.
 How can I access offset no of individual message getting being processed.


JavaPairInputDStream<byte[], byte[]> directKafkaStream
=KafkaUtils.createDirectStream(..);

directKafkaStream.mapPartitions(new
FlatMapFunction<Iterator<Tuple2<byte[],byte[]>>, String>() {
public Iterable<String> call(Iterator<Tuple2<byte[], byte[]>> t)
throws Exception {

while(t.hasNext()){
Tuple2<byte[], byte[]> tuple = t.next();
byte[] key = tuple._1();
byte[] msg = tuple._2();
 ///how to get kafka partition no and offset of this message
 }
}
});

Reply via email to