[jira] [Commented] (KAFKA-6188) Broker fails with FATAL Shutdown - log dirs have failed
[ https://issues.apache.org/jira/browse/KAFKA-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925309#comment-16925309 ] prehistoricpenguin commented on KAFKA-6188: --- [~jhandey] We are running Kafka on Windows with my [patch|[https://github.com/apache/kafka/pull/6403]] in our company, if you have the knowledge to build Kafka, you may have a try. > Broker fails with FATAL Shutdown - log dirs have failed > --- > > Key: KAFKA-6188 > URL: https://issues.apache.org/jira/browse/KAFKA-6188 > Project: Kafka > Issue Type: Bug > Components: clients, log >Affects Versions: 1.0.0, 1.0.1 > Environment: Windows 10 >Reporter: Valentina Baljak >Assignee: Dong Lin >Priority: Blocker > Labels: windows > Attachments: Segments are opened before deletion, > kafka_2.10-0.10.2.1.zip, output.txt > > > Just started with version 1.0.0 after a 4-5 months of using 0.10.2.1. The > test environment is very simple, with only one producer and one consumer. > Initially, everything started fine, stand alone tests worked as expected. > However, running my code, Kafka clients fail after approximately 10 minutes. > Kafka won't start after that and it fails with the same error. > Deleting logs helps to start again, and the same problem occurs. > Here is the error traceback: > [2017-11-08 08:21:57,532] INFO Starting log cleanup with a period of 30 > ms. (kafka.log.LogManager) > [2017-11-08 08:21:57,548] INFO Starting log flusher with a default period of > 9223372036854775807 ms. (kafka.log.LogManager) > [2017-11-08 08:21:57,798] INFO Awaiting socket connections on 0.0.0.0:9092. > (kafka.network.Acceptor) > [2017-11-08 08:21:57,813] INFO [SocketServer brokerId=0] Started 1 acceptor > threads (kafka.network.SocketServer) > [2017-11-08 08:21:57,829] INFO [ExpirationReaper-0-Produce]: Starting > (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) > [2017-11-08 08:21:57,845] INFO [ExpirationReaper-0-DeleteRecords]: Starting > (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) > [2017-11-08 08:21:57,845] INFO [ExpirationReaper-0-Fetch]: Starting > (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) > [2017-11-08 08:21:57,845] INFO [LogDirFailureHandler]: Starting > (kafka.server.ReplicaManager$LogDirFailureHandler) > [2017-11-08 08:21:57,860] INFO [ReplicaManager broker=0] Stopping serving > replicas in dir C:\Kafka\kafka_2.12-1.0.0\kafka-logs > (kafka.server.ReplicaManager) > [2017-11-08 08:21:57,860] INFO [ReplicaManager broker=0] Partitions are > offline due to failure on log directory C:\Kafka\kafka_2.12-1.0.0\kafka-logs > (kafka.server.ReplicaManager) > [2017-11-08 08:21:57,860] INFO [ReplicaFetcherManager on broker 0] Removed > fetcher for partitions (kafka.server.ReplicaFetcherManager) > [2017-11-08 08:21:57,892] INFO [ReplicaManager broker=0] Broker 0 stopped > fetcher for partitions because they are in the failed log dir > C:\Kafka\kafka_2.12-1.0.0\kafka-logs (kafka.server.ReplicaManager) > [2017-11-08 08:21:57,892] INFO Stopping serving logs in dir > C:\Kafka\kafka_2.12-1.0.0\kafka-logs (kafka.log.LogManager) > [2017-11-08 08:21:57,892] FATAL Shutdown broker because all log dirs in > C:\Kafka\kafka_2.12-1.0.0\kafka-logs have failed (kafka.log.LogManager) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (KAFKA-6188) Broker fails with FATAL Shutdown - log dirs have failed
[ https://issues.apache.org/jira/browse/KAFKA-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925309#comment-16925309 ] prehistoricpenguin edited comment on KAFKA-6188 at 9/9/19 2:38 AM: --- [~jhandey] We are running Kafka on Windows with my [patch |[https://github.com/apache/kafka/pull/6403]] in our company, if you have the knowledge to build Kafka, you may have a try. was (Author: prehistoricpenguin): [~jhandey] We are running Kafka on Windows with my [patch|[https://github.com/apache/kafka/pull/6403]] in our company, if you have the knowledge to build Kafka, you may have a try. > Broker fails with FATAL Shutdown - log dirs have failed > --- > > Key: KAFKA-6188 > URL: https://issues.apache.org/jira/browse/KAFKA-6188 > Project: Kafka > Issue Type: Bug > Components: clients, log >Affects Versions: 1.0.0, 1.0.1 > Environment: Windows 10 >Reporter: Valentina Baljak >Assignee: Dong Lin >Priority: Blocker > Labels: windows > Attachments: Segments are opened before deletion, > kafka_2.10-0.10.2.1.zip, output.txt > > > Just started with version 1.0.0 after a 4-5 months of using 0.10.2.1. The > test environment is very simple, with only one producer and one consumer. > Initially, everything started fine, stand alone tests worked as expected. > However, running my code, Kafka clients fail after approximately 10 minutes. > Kafka won't start after that and it fails with the same error. > Deleting logs helps to start again, and the same problem occurs. > Here is the error traceback: > [2017-11-08 08:21:57,532] INFO Starting log cleanup with a period of 30 > ms. (kafka.log.LogManager) > [2017-11-08 08:21:57,548] INFO Starting log flusher with a default period of > 9223372036854775807 ms. (kafka.log.LogManager) > [2017-11-08 08:21:57,798] INFO Awaiting socket connections on 0.0.0.0:9092. > (kafka.network.Acceptor) > [2017-11-08 08:21:57,813] INFO [SocketServer brokerId=0] Started 1 acceptor > threads (kafka.network.SocketServer) > [2017-11-08 08:21:57,829] INFO [ExpirationReaper-0-Produce]: Starting > (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) > [2017-11-08 08:21:57,845] INFO [ExpirationReaper-0-DeleteRecords]: Starting > (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) > [2017-11-08 08:21:57,845] INFO [ExpirationReaper-0-Fetch]: Starting > (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) > [2017-11-08 08:21:57,845] INFO [LogDirFailureHandler]: Starting > (kafka.server.ReplicaManager$LogDirFailureHandler) > [2017-11-08 08:21:57,860] INFO [ReplicaManager broker=0] Stopping serving > replicas in dir C:\Kafka\kafka_2.12-1.0.0\kafka-logs > (kafka.server.ReplicaManager) > [2017-11-08 08:21:57,860] INFO [ReplicaManager broker=0] Partitions are > offline due to failure on log directory C:\Kafka\kafka_2.12-1.0.0\kafka-logs > (kafka.server.ReplicaManager) > [2017-11-08 08:21:57,860] INFO [ReplicaFetcherManager on broker 0] Removed > fetcher for partitions (kafka.server.ReplicaFetcherManager) > [2017-11-08 08:21:57,892] INFO [ReplicaManager broker=0] Broker 0 stopped > fetcher for partitions because they are in the failed log dir > C:\Kafka\kafka_2.12-1.0.0\kafka-logs (kafka.server.ReplicaManager) > [2017-11-08 08:21:57,892] INFO Stopping serving logs in dir > C:\Kafka\kafka_2.12-1.0.0\kafka-logs (kafka.log.LogManager) > [2017-11-08 08:21:57,892] FATAL Shutdown broker because all log dirs in > C:\Kafka\kafka_2.12-1.0.0\kafka-logs have failed (kafka.log.LogManager) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-6188) Broker fails with FATAL Shutdown - log dirs have failed
[ https://issues.apache.org/jira/browse/KAFKA-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16790418#comment-16790418 ] prehistoricpenguin commented on KAFKA-6188: --- Hi, [~lindong], [~TeilaRei] I have made a fix for this issue and our manually test passed via this [PR|https://github.com/apache/kafka/pull/6403], can you please help me review my code? Thank you so much! > Broker fails with FATAL Shutdown - log dirs have failed > --- > > Key: KAFKA-6188 > URL: https://issues.apache.org/jira/browse/KAFKA-6188 > Project: Kafka > Issue Type: Bug > Components: clients, log >Affects Versions: 1.0.0, 1.0.1 > Environment: Windows 10 >Reporter: Valentina Baljak >Assignee: Dong Lin >Priority: Blocker > Labels: windows > Attachments: Segments are opened before deletion, > kafka_2.10-0.10.2.1.zip, output.txt > > > Just started with version 1.0.0 after a 4-5 months of using 0.10.2.1. The > test environment is very simple, with only one producer and one consumer. > Initially, everything started fine, stand alone tests worked as expected. > However, running my code, Kafka clients fail after approximately 10 minutes. > Kafka won't start after that and it fails with the same error. > Deleting logs helps to start again, and the same problem occurs. > Here is the error traceback: > [2017-11-08 08:21:57,532] INFO Starting log cleanup with a period of 30 > ms. (kafka.log.LogManager) > [2017-11-08 08:21:57,548] INFO Starting log flusher with a default period of > 9223372036854775807 ms. (kafka.log.LogManager) > [2017-11-08 08:21:57,798] INFO Awaiting socket connections on 0.0.0.0:9092. > (kafka.network.Acceptor) > [2017-11-08 08:21:57,813] INFO [SocketServer brokerId=0] Started 1 acceptor > threads (kafka.network.SocketServer) > [2017-11-08 08:21:57,829] INFO [ExpirationReaper-0-Produce]: Starting > (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) > [2017-11-08 08:21:57,845] INFO [ExpirationReaper-0-DeleteRecords]: Starting > (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) > [2017-11-08 08:21:57,845] INFO [ExpirationReaper-0-Fetch]: Starting > (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) > [2017-11-08 08:21:57,845] INFO [LogDirFailureHandler]: Starting > (kafka.server.ReplicaManager$LogDirFailureHandler) > [2017-11-08 08:21:57,860] INFO [ReplicaManager broker=0] Stopping serving > replicas in dir C:\Kafka\kafka_2.12-1.0.0\kafka-logs > (kafka.server.ReplicaManager) > [2017-11-08 08:21:57,860] INFO [ReplicaManager broker=0] Partitions are > offline due to failure on log directory C:\Kafka\kafka_2.12-1.0.0\kafka-logs > (kafka.server.ReplicaManager) > [2017-11-08 08:21:57,860] INFO [ReplicaFetcherManager on broker 0] Removed > fetcher for partitions (kafka.server.ReplicaFetcherManager) > [2017-11-08 08:21:57,892] INFO [ReplicaManager broker=0] Broker 0 stopped > fetcher for partitions because they are in the failed log dir > C:\Kafka\kafka_2.12-1.0.0\kafka-logs (kafka.server.ReplicaManager) > [2017-11-08 08:21:57,892] INFO Stopping serving logs in dir > C:\Kafka\kafka_2.12-1.0.0\kafka-logs (kafka.log.LogManager) > [2017-11-08 08:21:57,892] FATAL Shutdown broker because all log dirs in > C:\Kafka\kafka_2.12-1.0.0\kafka-logs have failed (kafka.log.LogManager) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8145) Broker fails with FATAL Shutdown on Windows, log or index renaming fail
prehistoricpenguin created KAFKA-8145: - Summary: Broker fails with FATAL Shutdown on Windows, log or index renaming fail Key: KAFKA-8145 URL: https://issues.apache.org/jira/browse/KAFKA-8145 Project: Kafka Issue Type: Bug Components: core, log Affects Versions: 2.1.1, 2.0.1, 1.1.1, 1.1.0 Reporter: prehistoricpenguin When consumer offset log cleaning or Kafka log rolling triggered on Windows(The default rolling time is 168 hours, eg 1 week). Kafka broker will shut down due to file renaming fail, on Windows is't invalid to rename a file when it's opened, so this issue is Windows specific. A topical error log during log rolling: {quote}at kafka.log.Log$$anonfun$deleteSegments$1$$anonfun$apply$mcI$sp$1.apply(Log.scala:1170) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at kafka.log.Log$$anonfun$deleteSegments$1.apply$mcI$sp(Log.scala:1170) at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1161) at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1161) at kafka.log.Log.maybeHandleIOException(Log.scala:1678) at kafka.log.Log.deleteSegments(Log.scala:1161) at kafka.log.Log.deleteOldSegments(Log.scala:1156) at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1228) at kafka.log.Log.deleteOldSegments(Log.scala:1222) at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:854) at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:852) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) at scala.collection.immutable.List.foreach(List.scala:392) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) at kafka.log.LogManager.cleanupLogs(LogManager.scala:852) at kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:385) at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110) at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:62) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Suppressed: java.nio.file.FileSystemException: Z:\tmp\kafka-logs\test-0\.log -> Z:\tmp\kafka-logs\test-0\.log.deleted: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) at sun.nio.fs.WindowsFileCopy.move(Unknown Source) at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source) at java.nio.file.Files.move(Unknown Source) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694) ... 32 more{quote} And the log from consumer offset log cleaner: {quote}[2019-02-26 02:16:05,151] ERROR Failed to clean up log for __consumer_offsets-29 in dir C:\Program Files (x86)\...\tmp\kafka-logs due to IOException (kafka.server.LogDirFailureChannel) java.nio.file.FileSystemException: C:\Program Files (x86)\...\tmp\kafka-logs\__consumer_offsets-29\.log.cleaned -> C:\Program Files (x86)...\tmp\kafka-logs\__consumer_offsets-29\.log.swap: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:697) at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:212) at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:415) at kafka.log.Log.replaceSegments(Log.scala:1644) at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:535) at kafka.log.Cleaner$$anonfun$doClean$4.apply(LogCleaner.scala:462) at kafka.log.Cleaner$$anonfun$doClean$4.apply(LogCleaner.scala:461) at scala.collection.immutable.List.foreach(List.scala:392) at kafka.log.Cleaner.doClean(LogCleaner.scala:461) at kafka.log.Cleaner.clean(LogCleaner.scala:438) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:305) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:291) at kafka.utils.ShutdownableT
[jira] [Updated] (KAFKA-8145) Broker fails with FATAL Shutdown on Windows, log or index renaming fail
[ https://issues.apache.org/jira/browse/KAFKA-8145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] prehistoricpenguin updated KAFKA-8145: -- Description: When consumer offset log cleaning or Kafka log rolling triggered on Windows(The default rolling time is 168 hours, eg 1 week). Kafka broker will shut down due to file renaming fail, on Windows is't invalid to rename a file when it's opened, so this issue is Windows specific. A topical error log during log rolling: {quote}at kafka.log.Log$$anonfun$deleteSegments$1$$anonfun$apply$mcI$sp$1.apply(Log.scala:1170) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at kafka.log.Log$$anonfun$deleteSegments$1.apply$mcI$sp(Log.scala:1170) at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1161) at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1161) at kafka.log.Log.maybeHandleIOException(Log.scala:1678) at kafka.log.Log.deleteSegments(Log.scala:1161) at kafka.log.Log.deleteOldSegments(Log.scala:1156) at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1228) at kafka.log.Log.deleteOldSegments(Log.scala:1222) at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:854) at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:852) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) at scala.collection.immutable.List.foreach(List.scala:392) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) at kafka.log.LogManager.cleanupLogs(LogManager.scala:852) at kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:385) at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110) at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:62) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.runAndReset(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Suppressed: java.nio.file.FileSystemException: Z:\tmp\kafka-logs\test-0\.log -> Z:\tmp\kafka-logs\test-0\.log.deleted: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) at sun.nio.fs.WindowsFileCopy.move(Unknown Source) at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source) at java.nio.file.Files.move(Unknown Source) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694) ... 32 more {quote} And the log from consumer offset log cleaner: {quote}[2019-02-26 02:16:05,151] ERROR Failed to clean up log for __consumer_offsets-29 in dir C:\Program Files (x86)\...\tmp\kafka-logs due to IOException (kafka.server.LogDirFailureChannel) java.nio.file.FileSystemException: C:\Program Files (x86)\...\tmp\kafka-logs__consumer_offsets-29\.log.cleaned -> C:\Program Files (x86)...\tmp\kafka-logs__consumer_offsets-29\.log.swap: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:697) at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:212) at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:415) at kafka.log.Log.replaceSegments(Log.scala:1644) at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:535) at kafka.log.Cleaner$$anonfun$doClean$4.apply(LogCleaner.scala:462) at kafka.log.Cleaner$$anonfun$doClean$4.apply(LogCleaner.scala:461) at scala.collection.immutable.List.foreach(List.scala:392) at kafka.log.Cleaner.doClean(LogCleaner.scala:461) at kafka.log.Cleaner.clean(LogCleaner.scala:438) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:305) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:291) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82) Suppressed: java.nio.file.FileSystemException: C:\Program Files (x86)\...\tmp\kafka-logs__consumer_offsets-29\.log.clea
[jira] [Commented] (KAFKA-8145) Broker fails with FATAL Shutdown on Windows, log or index renaming fail
[ https://issues.apache.org/jira/browse/KAFKA-8145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798791#comment-16798791 ] prehistoricpenguin commented on KAFKA-8145: --- I have tried to fixed it via this [PR|https://github.com/apache/kafka/pull/6403], and our manual test have passed. Please review my patch. > Broker fails with FATAL Shutdown on Windows, log or index renaming fail > --- > > Key: KAFKA-8145 > URL: https://issues.apache.org/jira/browse/KAFKA-8145 > Project: Kafka > Issue Type: Bug > Components: core, log >Affects Versions: 1.1.0, 1.1.1, 2.0.1, 2.1.1 >Reporter: prehistoricpenguin >Priority: Major > Labels: windows > Original Estimate: 168h > Remaining Estimate: 168h > > When consumer offset log cleaning or Kafka log rolling triggered on > Windows(The default rolling time is 168 hours, eg 1 week). Kafka broker will > shut down due to file renaming fail, on Windows is't invalid to rename a file > when it's opened, so this issue is Windows specific. > A topical error log during log rolling: > {quote}at > kafka.log.Log$$anonfun$deleteSegments$1$$anonfun$apply$mcI$sp$1.apply(Log.scala:1170) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at kafka.log.Log$$anonfun$deleteSegments$1.apply$mcI$sp(Log.scala:1170) > at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1161) > at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1161) > at kafka.log.Log.maybeHandleIOException(Log.scala:1678) > at kafka.log.Log.deleteSegments(Log.scala:1161) > at kafka.log.Log.deleteOldSegments(Log.scala:1156) > at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1228) > at kafka.log.Log.deleteOldSegments(Log.scala:1222) > at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:854) > at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:852) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) > at scala.collection.immutable.List.foreach(List.scala:392) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) > at kafka.log.LogManager.cleanupLogs(LogManager.scala:852) > at kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:385) > at > kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110) > at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:62) > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) > at java.util.concurrent.FutureTask.runAndReset(Unknown Source) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown > Source) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Suppressed: java.nio.file.FileSystemException: > Z:\tmp\kafka-logs\test-0\.log -> > Z:\tmp\kafka-logs\test-0\.log.deleted: The process cannot > access the file because it is being used by another process. > at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) > at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) > at sun.nio.fs.WindowsFileCopy.move(Unknown Source) > at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source) > at java.nio.file.Files.move(Unknown Source) > at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694) > ... 32 more > {quote} > > And the log from consumer offset log cleaner: > {quote}[2019-02-26 02:16:05,151] ERROR Failed to clean up log for > __consumer_offsets-29 in dir C:\Program Files (x86)\...\tmp\kafka-logs due to > IOException (kafka.server.LogDirFailureChannel) > java.nio.file.FileSystemException: C:\Program Files > (x86)\...\tmp\kafka-logs__consumer_offsets-29\.log.cleaned > -> C:\Program Files > (x86)...\tmp\kafka-logs__consumer_offsets-29\.log.swap: > The process cannot access the file because it is being used by another > process. > at > sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) > at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) > at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387) > at > sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) > at java.nio.file.Files.move(Files.java:1395) > at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:697) > at org.apache.ka
[jira] [Created] (KAFKA-8549) Kafka Windows start up failed due to topic name conflict
prehistoricpenguin created KAFKA-8549: - Summary: Kafka Windows start up failed due to topic name conflict Key: KAFKA-8549 URL: https://issues.apache.org/jira/browse/KAFKA-8549 Project: Kafka Issue Type: Bug Components: core Affects Versions: 2.2.1 Reporter: prehistoricpenguin We are running Kafka server on windows, we got this exception during Kafka server start up: {code:java} 2019-06-11 14:50:48,537] ERROR Error while creating log for this_is_a_topic_name in dir C:\Program Files (x86)\dummy_path\tmp\kafka-logs (kafka.server.LogDirFailureChannel) java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section open at java.io.RandomAccessFile.setLength(Native Method) at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:188) at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) at kafka.log.AbstractIndex.resize(AbstractIndex.scala:175) at kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:238) at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:238) at kafka.log.LogSegment.recover(LogSegment.scala:377) at kafka.log.Log.recoverSegment(Log.scala:500) at kafka.log.Log.$anonfun$loadSegmentFiles$3(Log.scala:482) at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:792) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:791) at kafka.log.Log.loadSegmentFiles(Log.scala:454) at kafka.log.Log.$anonfun$loadSegments$1(Log.scala:565) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at kafka.log.Log.retryOnOffsetOverflow(Log.scala:2034) at kafka.log.Log.loadSegments(Log.scala:559) at kafka.log.Log.(Log.scala:292) at kafka.log.Log$.apply(Log.scala:2168) at kafka.log.LogManager.$anonfun$getOrCreateLog$1(LogManager.scala:716) at scala.Option.getOrElse(Option.scala:138) at kafka.log.LogManager.getOrCreateLog(LogManager.scala:674) at kafka.cluster.Partition.$anonfun$getOrCreateReplica$1(Partition.scala:202) at kafka.utils.Pool$$anon$1.apply(Pool.scala:61) at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) at kafka.utils.Pool.getAndMaybePut(Pool.scala:60) at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:198) at kafka.cluster.Partition.$anonfun$makeLeader$3(Partition.scala:376) at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) at scala.collection.Iterator.foreach(Iterator.scala:941) at scala.collection.Iterator.foreach$(Iterator.scala:941) at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) at scala.collection.IterableLike.foreach(IterableLike.scala:74) at scala.collection.IterableLike.foreach$(IterableLike.scala:73) at scala.collection.AbstractIterable.foreach(Iterable.scala:56) at scala.collection.TraversableLike.map(TraversableLike.scala:237) at scala.collection.TraversableLike.map$(TraversableLike.scala:230) at scala.collection.AbstractTraversable.map(Traversable.scala:108) at kafka.cluster.Partition.$anonfun$makeLeader$1(Partition.scala:376) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:259) at kafka.cluster.Partition.makeLeader(Partition.scala:370) at kafka.server.ReplicaManager.$anonfun$makeLeaders$5(ReplicaManager.scala:1188) at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149) at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237) at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44) at scala.collection.mutable.HashMap.foreach(HashMap.scala:149) at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:1186) at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:1098) at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:195) at kafka.server.KafkaApis.handle(KafkaApis.scala:112) at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69) at java.lang.Thread.run(Thread.java:748) [2019-06-11 14:50:48,542] INFO [ReplicaManager broker=0] Stopping serving replicas in dir C:\Program Files (x86)\dummy_path\tmp\kafka-logs (kafka.server.ReplicaManager) [2019-06-11 14:50:48,543] ERROR [ReplicaManager broker=0] Error while making broker the leader for partition Topic: this_is_a_topic_name; Partition: 0; Leader: None; AllReplicas: ; InSyncReplicas: in dir None (kafka.server
[jira] [Updated] (KAFKA-8549) Kafka Windows start up failed due to cannot be performed on a file with a user-mapped section open
[ https://issues.apache.org/jira/browse/KAFKA-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] prehistoricpenguin updated KAFKA-8549: -- Summary: Kafka Windows start up failed due to cannot be performed on a file with a user-mapped section open (was: Kafka Windows start up failed due to topic name conflict ) > Kafka Windows start up failed due to cannot be performed on a file with a > user-mapped section open > -- > > Key: KAFKA-8549 > URL: https://issues.apache.org/jira/browse/KAFKA-8549 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.2.1 >Reporter: prehistoricpenguin >Priority: Major > Labels: crash, windows > > We are running Kafka server on windows, we got this exception during Kafka > server start up: > {code:java} > 2019-06-11 14:50:48,537] ERROR Error while creating log for > this_is_a_topic_name in dir C:\Program Files (x86)\dummy_path\tmp\kafka-logs > (kafka.server.LogDirFailureChannel) > java.io.IOException: The requested operation cannot be performed on a file > with a user-mapped section open > at java.io.RandomAccessFile.setLength(Native Method) > at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:188) > at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23) > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) > at kafka.log.AbstractIndex.resize(AbstractIndex.scala:175) > at kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:238) > at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23) > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) > at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:238) > at kafka.log.LogSegment.recover(LogSegment.scala:377) > at kafka.log.Log.recoverSegment(Log.scala:500) > at kafka.log.Log.$anonfun$loadSegmentFiles$3(Log.scala:482) > at > scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:792) > at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) > at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:791) > at kafka.log.Log.loadSegmentFiles(Log.scala:454) > at kafka.log.Log.$anonfun$loadSegments$1(Log.scala:565) > at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) > at kafka.log.Log.retryOnOffsetOverflow(Log.scala:2034) > at kafka.log.Log.loadSegments(Log.scala:559) > at kafka.log.Log.(Log.scala:292) > at kafka.log.Log$.apply(Log.scala:2168) > at kafka.log.LogManager.$anonfun$getOrCreateLog$1(LogManager.scala:716) > at scala.Option.getOrElse(Option.scala:138) > at kafka.log.LogManager.getOrCreateLog(LogManager.scala:674) > at kafka.cluster.Partition.$anonfun$getOrCreateReplica$1(Partition.scala:202) > at kafka.utils.Pool$$anon$1.apply(Pool.scala:61) > at > java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) > at kafka.utils.Pool.getAndMaybePut(Pool.scala:60) > at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:198) > at kafka.cluster.Partition.$anonfun$makeLeader$3(Partition.scala:376) > at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) > at scala.collection.Iterator.foreach(Iterator.scala:941) > at scala.collection.Iterator.foreach$(Iterator.scala:941) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) > at scala.collection.IterableLike.foreach(IterableLike.scala:74) > at scala.collection.IterableLike.foreach$(IterableLike.scala:73) > at scala.collection.AbstractIterable.foreach(Iterable.scala:56) > at scala.collection.TraversableLike.map(TraversableLike.scala:237) > at scala.collection.TraversableLike.map$(TraversableLike.scala:230) > at scala.collection.AbstractTraversable.map(Traversable.scala:108) > at kafka.cluster.Partition.$anonfun$makeLeader$1(Partition.scala:376) > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) > at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:259) > at kafka.cluster.Partition.makeLeader(Partition.scala:370) > at > kafka.server.ReplicaManager.$anonfun$makeLeaders$5(ReplicaManager.scala:1188) > at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149) > at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237) > at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230) > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44) > at scala.collection.mutable.HashMap.foreach(HashMap.scala:149) > at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:1186) > at > kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.s
[jira] [Updated] (KAFKA-8549) Kafka Windows start up fail due to cannot be performed on a file with a user-mapped section open
[ https://issues.apache.org/jira/browse/KAFKA-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] prehistoricpenguin updated KAFKA-8549: -- Summary: Kafka Windows start up fail due to cannot be performed on a file with a user-mapped section open (was: Kafka Windows start up failed due to cannot be performed on a file with a user-mapped section open) > Kafka Windows start up fail due to cannot be performed on a file with a > user-mapped section open > > > Key: KAFKA-8549 > URL: https://issues.apache.org/jira/browse/KAFKA-8549 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.2.1 >Reporter: prehistoricpenguin >Priority: Major > Labels: crash, windows > > We are running Kafka server on windows, we got this exception during Kafka > server start up: > {code:java} > 2019-06-11 14:50:48,537] ERROR Error while creating log for > this_is_a_topic_name in dir C:\Program Files (x86)\dummy_path\tmp\kafka-logs > (kafka.server.LogDirFailureChannel) > java.io.IOException: The requested operation cannot be performed on a file > with a user-mapped section open > at java.io.RandomAccessFile.setLength(Native Method) > at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:188) > at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23) > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) > at kafka.log.AbstractIndex.resize(AbstractIndex.scala:175) > at kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:238) > at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23) > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) > at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:238) > at kafka.log.LogSegment.recover(LogSegment.scala:377) > at kafka.log.Log.recoverSegment(Log.scala:500) > at kafka.log.Log.$anonfun$loadSegmentFiles$3(Log.scala:482) > at > scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:792) > at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) > at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:791) > at kafka.log.Log.loadSegmentFiles(Log.scala:454) > at kafka.log.Log.$anonfun$loadSegments$1(Log.scala:565) > at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) > at kafka.log.Log.retryOnOffsetOverflow(Log.scala:2034) > at kafka.log.Log.loadSegments(Log.scala:559) > at kafka.log.Log.(Log.scala:292) > at kafka.log.Log$.apply(Log.scala:2168) > at kafka.log.LogManager.$anonfun$getOrCreateLog$1(LogManager.scala:716) > at scala.Option.getOrElse(Option.scala:138) > at kafka.log.LogManager.getOrCreateLog(LogManager.scala:674) > at kafka.cluster.Partition.$anonfun$getOrCreateReplica$1(Partition.scala:202) > at kafka.utils.Pool$$anon$1.apply(Pool.scala:61) > at > java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) > at kafka.utils.Pool.getAndMaybePut(Pool.scala:60) > at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:198) > at kafka.cluster.Partition.$anonfun$makeLeader$3(Partition.scala:376) > at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) > at scala.collection.Iterator.foreach(Iterator.scala:941) > at scala.collection.Iterator.foreach$(Iterator.scala:941) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) > at scala.collection.IterableLike.foreach(IterableLike.scala:74) > at scala.collection.IterableLike.foreach$(IterableLike.scala:73) > at scala.collection.AbstractIterable.foreach(Iterable.scala:56) > at scala.collection.TraversableLike.map(TraversableLike.scala:237) > at scala.collection.TraversableLike.map$(TraversableLike.scala:230) > at scala.collection.AbstractTraversable.map(Traversable.scala:108) > at kafka.cluster.Partition.$anonfun$makeLeader$1(Partition.scala:376) > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) > at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:259) > at kafka.cluster.Partition.makeLeader(Partition.scala:370) > at > kafka.server.ReplicaManager.$anonfun$makeLeaders$5(ReplicaManager.scala:1188) > at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149) > at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237) > at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230) > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44) > at scala.collection.mutable.HashMap.foreach(HashMap.scala:149) > at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:1186) > at > kafka.server.ReplicaManager.beco