Greetings all,
I’ve recently started hitting on the following error in Spark Streaming in
Kafka. Adjusting maxRetries and spark.streaming.kafka.consumer.poll.ms even to
five minutes doesn’t seem to be helping. The problem only manifested in the
last few days, restarting with a new consumer group seems to remedy the issue
for a few hours (< retention, which is 12 hours).
Error:
Caused by: java.lang.AssertionError: assertion failed: Got wrong record for
spark-executor-<consumergrouphere> <topichere> 76 even after seeking to offset
1759148155
at scala.Predef$.assert(Predef.scala:170)
at
org.apache.spark.streaming.kafka010.CachedKafkaConsumer.get(CachedKafkaConsumer.scala:85)
at
org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:223)
at
org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:189)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
I guess my questions are, why is that assertion a job killer vs a warning and
is there anything I can tweak settings wise that may keep it at bay.
I wouldn’t be surprised if this issue were exacerbated by the volume we do on
Kafka topics (~150k/sec on the persister that’s crashing).
Thank you!
Justin
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
