Github user gaborgsomogyi commented on the issue: https://github.com/apache/spark/pull/20997 In the meantime found a small glitch in the SQL part. Namely if reattempt happens this line https://github.com/apache/spark/blob/1d758dc73b54e802fdc92be204185fe7414e6553/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala#L445 removes the consumer from cache which will end up in this log message: ``` 13:27:07.556 INFO org.apache.spark.sql.kafka010.KafkaDataConsumer: Released a supposedly cached consumer that was not found in the cache ``` I've solved this here by removing only the closed consumer. The marked for close will be removed in `release`.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org