Hi, I see this sometimes in our Kafka Direct approach in our Streaming job. How do we make sure that the job recovers from such errors and works normally thereafter?
15/09/30 05:14:18 ERROR KafkaRDD: Lost leader for topic x_stream partition 19, sleeping for 200ms 15/09/30 05:14:18 ERROR KafkaRDD: Lost leader for topic x_stream partition 5, sleeping for 200ms Followed by every task failing with something like this: 15/09/30 05:26:20 ERROR Executor: Exception in task 4.0 in stage 84281.0 (TID 818804) kafka.common.NotLeaderForPartitionException And: org.apache.spark.SparkException: Job aborted due to stage failure: Task 15 in stage 84958.0 failed 4 times, most recent failure: Lost task 15.3 in stage 84958.0 (TID 819461, 10.227.68.102): java.lang.AssertionError: assertion failed: Beginning offset 15027734702 is after the ending offset 15027725493 for topic hubble_stream partition 12. You either provided an invalid fromOffset, or the Kafka topic has been damaged Thanks, Swetha -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Lost-leader-exception-in-Kafka-Direct-for-Streaming-tp24891.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
