The last time someone brought this up on the mailing list, the issue actually was that the topic(s) didn't exist in Kafka at the time the spark job was running.
On Wed, Jul 29, 2015 at 6:17 PM, Tathagata Das <t...@databricks.com> wrote: > There is a known issue that Kafka cannot return leader if there is not > data in the topic. I think it was raised in another thread in this forum. > Is that the issue? > > On Wed, Jul 29, 2015 at 10:38 AM, unk1102 <umesh.ka...@gmail.com> wrote: > >> Hi I have Spark Streaming code which streams from Kafka topic it used to >> work >> fine but suddenly it started throwing the following exception >> >> Exception in thread "main" org.apache.spark.SparkException: >> org.apache.spark.SparkException: Couldn't find leader offsets for Set() >> at >> >> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413) >> at >> >> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413) >> at scala.util.Either.fold(Either.scala:97) >> at >> >> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412) >> at >> >> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528) >> at >> >> org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala) >> My Spark Streaming client code is very simple I just create one receiver >> using the following code and trying to print messages it consumed >> >> JavaPairInputDStream<String, String> messages = >> KafkaUtils.createDirectStream(jssc, >> String.class, >> String.class, >> StringDecoder.class, >> StringDecoder.class, >> kafkaParams, >> topicSet); >> >> Kafka param is only one I specify kafka.ofset.reset=largest. Kafka topic >> has >> data I can see data using other Kafka consumers but above Spark Streaming >> code throws exception saying leader offset not found. I tried both >> smallest >> and largest offset. I wonder what happened this code used to work >> earlier. I >> am using Spark-Streaming 1.3.1 as it was working in this version I tried >> in >> 1.4.1 and same exception. Please guide. I am new to Spark thanks in >> advance. >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-could-not-find-leader-offset-for-Set-tp24066.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
