Jay Lee created KAFKA-8533:
------------------------------
Summary: Kafka 2.1.0 version crashes when deleting log
Key: KAFKA-8533
URL: https://issues.apache.org/jira/browse/KAFKA-8533
Project: Kafka
Issue Type: Bug
Components: log cleaner
Affects Versions: 2.1.0
Reporter: Jay Lee
We are running Kafka 2.1.0 version in RedHat Linux.
On 28th-May, Kafka crashed with the log in the bottom.
Please let me know what may be the issue.
For your information, here below are the versions of our OS & kafka:
* *OS: RHEL 7.6*
* *Kafka: 2.12-2.1.0*
======================================================
May 28 08:31:51 tsea-bc-db02 sh: [2019-05-28 08:31:51,291] ERROR Failed to
clean up log for __consumer_offsets-46 in dir /tmp/kafka-logs due to
IOException (kafka.server.LogDirFailureChannel)
May 28 08:31:51 tsea-bc-db02 sh: java.nio.file.NoSuchFileException:
/tmp/kafka-logs/__consumer_offsets-46/00000000000000000000.log
May 28 08:31:51 tsea-bc-db02 sh: at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
May 28 08:31:51 tsea-bc-db02 sh: at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
May 28 08:31:51 tsea-bc-db02 sh: at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
May 28 08:31:51 tsea-bc-db02 sh: at
sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
May 28 08:31:51 tsea-bc-db02 sh: at
sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
May 28 08:31:51 tsea-bc-db02 sh: at java.nio.file.Files.move(Files.java:1395)
May 28 08:31:51 tsea-bc-db02 sh: at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:809)
May 28 08:31:51 tsea-bc-db02 sh: at
org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:222)
May 28 08:31:51 tsea-bc-db02 sh: at
kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:488)
May 28 08:31:51 tsea-bc-db02 sh: at
kafka.log.Log.asyncDeleteSegment(Log.scala:1838)
May 28 08:31:51 tsea-bc-db02 sh: at
kafka.log.Log.$anonfun$replaceSegments$6(Log.scala:1901)
May 28 08:31:51 tsea-bc-db02 sh: at
kafka.log.Log.$anonfun$replaceSegments$6$adapted(Log.scala:1896)
May 28 08:31:51 tsea-bc-db02 sh: at
scala.collection.immutable.List.foreach(List.scala:388)
May 28 08:31:51 tsea-bc-db02 sh: at
kafka.log.Log.replaceSegments(Log.scala:1896)
May 28 08:31:51 tsea-bc-db02 sh: at
kafka.log.Cleaner.cleanSegments(LogCleaner.scala:583)
May 28 08:31:51 tsea-bc-db02 sh: at
kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:515)
May 28 08:31:51 tsea-bc-db02 sh: at
kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:514)
May 28 08:31:51 tsea-bc-db02 sh: at
scala.collection.immutable.List.foreach(List.scala:388)
May 28 08:31:51 tsea-bc-db02 sh: at
kafka.log.Cleaner.doClean(LogCleaner.scala:514)
May 28 08:31:51 tsea-bc-db02 sh: at
kafka.log.Cleaner.clean(LogCleaner.scala:492)
May 28 08:31:51 tsea-bc-db02 sh: at
kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:353)
May 28 08:31:51 tsea-bc-db02 sh: at
kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:319)
May 28 08:31:51 tsea-bc-db02 sh: at
kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:300)
May 28 08:31:51 tsea-bc-db02 sh: at
kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
May 28 08:31:51 tsea-bc-db02 sh: Suppressed: java.nio.file.NoSuchFileException:
/tmp/kafka-logs/__consumer_offsets-46/00000000000000000000.log ->
/tmp/kafka-logs/__consumer_offsets-46/00000000000000000000.log.deleted
May 28 08:31:51 tsea-bc-db02 sh: at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
May 28 08:31:51 tsea-bc-db02 sh: at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
May 28 08:31:51 tsea-bc-db02 sh: at
sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
May 28 08:31:51 tsea-bc-db02 sh: at
sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
May 28 08:31:51 tsea-bc-db02 sh: at java.nio.file.Files.move(Files.java:1395)
May 28 08:31:51 tsea-bc-db02 sh: at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:806)
May 28 08:31:51 tsea-bc-db02 sh: ... 17 more
May 28 08:31:51 tsea-bc-db02 sh: [2019-05-28 08:31:51,309] INFO [ReplicaManager
broker=2] Stopping serving replicas in dir /tmp/kafka-logs
(kafka.server.ReplicaManager)
May 28 08:31:51 tsea-bc-db02 sh: [2019-05-28 08:31:51,312] INFO
[ReplicaFetcherManager on broker 2] Removed fetcher for partitions
Set(__consumer_offsets-22, CONSUME_MESSAGE_TOPIC-0, __consumer_offsets-4,
__consumer_offsets-7, __consumer_offsets-46, __consumer_offsets-25,
__consumer_offsets-49, __consumer_offsets-16, __consumer_offsets-28,
__consumer_offsets-31, __consumer_offsets-37, __consumer_offsets-19,
__consumer_offsets-13, __consumer_offsets-43, __consumer_offsets-1,
__consumer_offsets-34, __consumer_offsets-10, __consumer_offsets-40,
PUBLISH_MESSAGE_TOPIC-0) (kafka.server.ReplicaFetcherManager)
May 28 08:31:51 tsea-bc-db02 sh: [2019-05-28 08:31:51,313] INFO
[ReplicaAlterLogDirsManager on broker 2] Removed fetcher for partitions
Set(__consumer_offsets-22, CONSUME_MESSAGE_TOPIC-0, __consumer_offsets-4,
__consumer_offsets-7, __consumer_offsets-46, __consumer_offsets-25,
__consumer_offsets-49, __consumer_offsets-16, __consumer_offsets-28,
__consumer_offsets-31, __consumer_offsets-37, __consumer_offsets-19,
__consumer_offsets-13, __consumer_offsets-43, __consumer_offsets-1,
__consumer_offsets-34, __consumer_offsets-10, __consumer_offsets-40,
PUBLISH_MESSAGE_TOPIC-0) (kafka.server.ReplicaAlterLogDirsManager)
May 28 08:31:51 tsea-bc-db02 sh: [2019-05-28 08:31:51,322] INFO [ReplicaManager
broker=2] Broker 2 stopped fetcher for partitions
__consumer_offsets-22,CONSUME_MESSAGE_TOPIC-0,__consumer_offsets-4,__consumer_offsets-7,__consumer_offsets-46,__consumer_offsets-25,__consumer_offsets-49,__consumer_offsets-16,__consumer_offsets-28,__consumer_offsets-31,__consumer_offsets-37,__consumer_offsets-19,__consumer_offsets-13,__consumer_offsets-43,__consumer_offsets-1,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-40,PUBLISH_MESSAGE_TOPIC-0
and stopped moving logs for partitions because they are in the failed log
directory /tmp/kafka-logs. (kafka.server.ReplicaManager)
May 28 08:31:51 tsea-bc-db02 sh: [2019-05-28 08:31:51,323] INFO Stopping
serving logs in dir /tmp/kafka-logs (kafka.log.LogManager)
May 28 08:31:51 tsea-bc-db02 sh: [2019-05-28 08:31:51,328] ERROR Shutdown
broker because all log dirs in /tmp/kafka-logs have failed
(kafka.log.LogManager)
May 28 08:31:51 tsea-bc-db02 systemd: kafka.service: main process exited,
code=exited, status=1/FAILURE
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
