Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/22097
Looks like there is a race condition during terminating Kafka cluster:
{code}
18/08/13 15:34:44.148 kafka-log-cleaner-thread-0 ERROR LogCleaner: Failed
to access checkpoint file cleaner-offset-checkpoint in dir
/home/jenkins/workspace/SparkPullRequestBuilder@3/target/tmp/spark-5ad98c9e-0d75-4f23-a948-9e29246651d2
org.apache.kafka.common.errors.KafkaStorageException: Error while reading
checkpoint file
/home/jenkins/workspace/SparkPullRequestBuilder@3/target/tmp/spark-5ad98c9e-0d75-4f23-a948-9e29246651d2/cleaner-offset-checkpoint
Caused by: java.io.FileNotFoundException:
/home/jenkins/workspace/SparkPullRequestBuilder@3/target/tmp/spark-5ad98c9e-0d75-4f23-a948-9e29246651d2/cleaner-offset-checkpoint
(No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at
kafka.server.checkpoints.CheckpointFile.liftedTree2$1(CheckpointFile.scala:87)
at kafka.server.checkpoints.CheckpointFile.read(CheckpointFile.scala:86)
at
kafka.server.checkpoints.OffsetCheckpointFile.read(OffsetCheckpointFile.scala:61)
at
kafka.log.LogCleanerManager$$anonfun$allCleanerCheckpoints$1$$anonfun$apply$1.apply(LogCleanerManager.scala:89)
at
kafka.log.LogCleanerManager$$anonfun$allCleanerCheckpoints$1$$anonfun$apply$1.apply(LogCleanerManager.scala:87)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at
scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206)
at
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at
kafka.log.LogCleanerManager$$anonfun$allCleanerCheckpoints$1.apply(LogCleanerManager.scala:87)
at
kafka.log.LogCleanerManager$$anonfun$allCleanerCheckpoints$1.apply(LogCleanerManager.scala:95)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at
kafka.log.LogCleanerManager.allCleanerCheckpoints(LogCleanerManager.scala:86)
at
kafka.log.LogCleanerManager$$anonfun$grabFilthiestCompactedLog$1.apply(LogCleanerManager.scala:126)
at
kafka.log.LogCleanerManager$$anonfun$grabFilthiestCompactedLog$1.apply(LogCleanerManager.scala:123)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at
kafka.log.LogCleanerManager.grabFilthiestCompactedLog(LogCleanerManager.scala:123)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:296)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:289)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
{code}
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
