Is the kafka for testing purposes? Although it is cross platform, the best is still under linux so that os buffer & cache can be efficient. If you can, you could try to run under WSL2. See a tech blog here: https://www.confluent.io/blog/set-up-and-run-kafka-on-windows-linux-wsl-2/
Lastly, if you have to run on windows as it is, you may try to find out which process is holding the file handle. https://serverfault.com/questions/1966/how-do-you-find-what-process-is-holding-a-file-open-in-windows There might be another process holding it open so it can't be renamed. (Unlike linux, windows doesn't allow that). Thanks Best Regards Shilin [image: Confluent] <https://www.confluent.io> Wu Shilin Solution Architect +6581007012 Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/company/confluent/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Kafka Summit] <https://www.kafka-summit.org/> On Tue, Jun 15, 2021 at 6:08 AM Peter Lee <[email protected]> wrote: > ---------- Forwarded message --------- > 보낸사람: hello <[email protected]> > Date: 2021년 6월 9일 (수) 오후 2:50 > Subject: Kafkabroker log swap writing error > To: users-sc.1623217635.lobeokoidbfekkpndnbk-leepeter2019= > [email protected] > <users-sc.1623217635.lobeokoidbfekkpndnbk-leepeter2019= > [email protected]> > > > Whenever I run Kafka server in windows 10, it failed to rename > controller.log to controller.log.time > > > > After several failed attempts, it stops running. > > > > *java.nio.file.FileSystemException: > > C:\Testing\.logskafka-logs\__consumer_offsets-31\00000000000000000000.index.cleaned > -> > > C:\Testing\.logskafka-logs\__consumer_offsets-31\00000000000000000000.index.swap:* > > *“The process cannot access, because the other process is in use”* > > > > *[2021-04-21 10:27:59,232] ERROR Shutdown broker because all log dirs in > C:\Testing\.logskafka-logs have failed (kafka.log.LogManager* > > > > Do you have any idea how to fix this? > > > > > > [2021-04-20 15:47:53,282] INFO [GroupCoordinator 0]: Stabilized group > monitoragent generation 9 (__consumer_offsets-31) > (kafka.coordinator.group.GroupCoordinator) > > [2021-04-20 15:47:59,474] INFO [GroupCoordinator 0]: Preparing to rebalance > group monitoragent in state PreparingRebalance with old generation 9 > (__consumer_offsets-31) (reason: Adding new member > node1-agent8-16cc2029-7ee0-4560-9890-028fea05b8b2 with group instance id > None) (kafka.coordinator.group.GroupCoordinator) > > [2021-04-20 15:52:53,286] INFO [GroupCoordinator 0]: Member > node1-agent8-4e7ee271-581a-40b2-9b1f-e5fb4f364704 in group monitoragent has > failed, removing it from the group > (kafka.coordinator.group.GroupCoordinator) > > [2021-04-20 15:52:53,287] INFO [GroupCoordinator 0]: Stabilized group > monitoragent generation 10 (__consumer_offsets-31) > (kafka.coordinator.group.GroupCoordinator) > > [2021-04-20 15:52:53,295] INFO [GroupCoordinator 0]: Assignment received > from leader for group monitoragent for generation 10 > (kafka.coordinator.group.GroupCoordinator) > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-20-15]. > > [2021-04-20 16:06:59,437] INFO [GroupCoordinator 0]: Member > node1-agent8-16cc2029-7ee0-4560-9890-028fea05b8b2 in group monitoragent has > failed, removing it from the group > (kafka.coordinator.group.GroupCoordinator) > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/server.log] to > [C:\kafka_2.13-2.7.0/logs/server.log.2021-04-20-15]. > > [2021-04-20 16:06:59,438] INFO [GroupCoordinator 0]: Preparing to rebalance > group monitoragent in state PreparingRebalance with old generation 10 > (__consumer_offsets-31) (reason: removing member > node1-agent8-16cc2029-7ee0-4560-9890-028fea05b8b2 on heartbeat expiration) > (kafka.coordinator.group.GroupCoordinator) > > [2021-04-20 16:06:59,439] INFO [GroupCoordinator 0]: Group monitoragent > with generation 11 is now empty (__consumer_offsets-31) > (kafka.coordinator.group.GroupCoordinator) > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-20-16]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-20-17]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-20-18]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-20-19]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-20-20]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-20-21]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-20-22]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-20-23]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-21-00]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-21-01]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-21-02]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-21-03]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-21-04]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-21-05]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-21-06]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-21-07]. > > [2021-04-21 08:11:56,092] INFO [GroupCoordinator 0]: Preparing to rebalance > group monitoragent in state PreparingRebalance with old generation 11 > (__consumer_offsets-31) (reason: Adding new member > node1-agent8-bd039ca4-dca7-43a9-bd31-48b6a9d86031 with group instance id > None) (kafka.coordinator.group.GroupCoordinator) > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/server.log] to > [C:\kafka_2.13-2.7.0/logs/server.log.2021-04-20-16]. > > [2021-04-21 08:11:56,095] INFO [GroupCoordinator 0]: Stabilized group > monitoragent generation 12 (__consumer_offsets-31) > (kafka.coordinator.group.GroupCoordinator) > > [2021-04-21 08:11:56,100] INFO [GroupCoordinator 0]: Assignment received > from leader for group monitoragent for generation 12 > (kafka.coordinator.group.GroupCoordinator) > > [2021-04-21 08:16:46,832] INFO [GroupCoordinator 0]: Preparing to rebalance > group monitoragent in state PreparingRebalance with old generation 12 > (__consumer_offsets-31) (reason: Adding new member > node1-agent8-4a3f8f6f-1091-4954-99c1-d1ff2c8ae12e with group instance id > None) (kafka.coordinator.group.GroupCoordinator) > > [2021-04-21 08:20:22,323] INFO [GroupCoordinator 0]: Member > node1-agent8-bd039ca4-dca7-43a9-bd31-48b6a9d86031 in group monitoragent has > failed, removing it from the group > (kafka.coordinator.group.GroupCoordinator) > > [2021-04-21 08:20:22,326] INFO [GroupCoordinator 0]: Stabilized group > monitoragent generation 13 (__consumer_offsets-31) > (kafka.coordinator.group.GroupCoordinator) > > [2021-04-21 08:20:22,383] INFO [GroupCoordinator 0]: Assignment received > from leader for group monitoragent for generation 13 > (kafka.coordinator.group.GroupCoordinator) > > [2021-04-21 08:23:38,927] INFO [GroupCoordinator 0]: Preparing to rebalance > group monitoragent in state PreparingRebalance with old generation 13 > (__consumer_offsets-31) (reason: Adding new member > node2-agent9-2210cbbe-9324-4d53-9b0e-a37997275127 with group instance id > None) (kafka.coordinator.group.GroupCoordinator) > > [2021-04-21 08:26:23,712] INFO [GroupCoordinator 0]: Stabilized group > monitoragent generation 14 (__consumer_offsets-31) > (kafka.coordinator.group.GroupCoordinator) > > [2021-04-21 08:26:23,722] INFO [GroupCoordinator 0]: Assignment received > from leader for group monitoragent for generation 14 > (kafka.coordinator.group.GroupCoordinator) > > [2021-04-21 09:01:02,878] INFO [GroupCoordinator 0]: Member[ > group.instance.id None, member.id > node1-agent8-4a3f8f6f-1091-4954-99c1-d1ff2c8ae12e] > in group monitoragent has left, removing it from the group > (kafka.coordinator.group.GroupCoordinator) > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/server.log] to > [C:\kafka_2.13-2.7.0/logs/server.log.2021-04-21-08]. > > [2021-04-21 09:01:02,880] INFO [GroupCoordinator 0]: Preparing to rebalance > group monitoragent in state PreparingRebalance with old generation 14 > (__consumer_offsets-31) (reason: removing member > node1-agent8-4a3f8f6f-1091-4954-99c1-d1ff2c8ae12e on LeaveGroup) > (kafka.coordinator.group.GroupCoordinator) > > [2021-04-21 09:02:27,729] INFO [GroupCoordinator 0]: Stabilized group > monitoragent generation 15 (__consumer_offsets-31) > (kafka.coordinator.group.GroupCoordinator) > > [2021-04-21 09:02:27,786] INFO [GroupCoordinator 0]: Assignment received > from leader for group monitoragent for generation 15 > (kafka.coordinator.group.GroupCoordinator) > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-21-08]. > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/controller.log] to > [C:\kafka_2.13-2.7.0/logs/controller.log.2021-04-21-09]. > > [2021-04-21 10:27:53,405] INFO [ProducerStateManager > partition=__consumer_offsets-31] Writing producer snapshot at offset 70374 > (kafka.log.ProducerStateManager) > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/server.log] to > [C:\kafka_2.13-2.7.0/logs/server.log.2021-04-21-09]. > > [2021-04-21 10:27:53,412] INFO [Log partition=__consumer_offsets-31, > dir=C:\Testing\.logskafka-logs] Rolled new log segment at offset 70374 in > 11 ms. (kafka.log.Log) > > log4j:ERROR Failed to rename [C:\kafka_2.13-2.7.0/logs/log-cleaner.log] to > [C:\kafka_2.13-2.7.0/logs/log-cleaner.log.2021-04-20-15]. > > [2021-04-21 10:27:59,102] ERROR Failed to clean up log for > __consumer_offsets-31 in dir C:\Testing\.logskafka-logs due to IOException > (kafka.server.LogDirFailureChannel) > > java.nio.file.FileSystemException: > > C:\Testing\.logskafka-logs\__consumer_offsets-31\00000000000000000000.index.cleaned > -> > > C:\Testing\.logskafka-logs\__consumer_offsets-31\00000000000000000000.index.swap: > “The process cannot access, because the other process is in use” > > > > at sun.nio.fs.WindowsException.translateToIOException(Unknown > Source) > > at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) > > at sun.nio.fs.WindowsFileCopy.move(Unknown Source) > > at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source) > > at java.nio.file.Files.move(Unknown Source) > > at > org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:917) > > at kafka.log.AbstractIndex.renameTo(AbstractIndex.scala:211) > > at kafka.log.LazyIndex$IndexValue.renameTo(LazyIndex.scala:155) > > at kafka.log.LazyIndex.$anonfun$renameTo$1(LazyIndex.scala:79) > > at kafka.log.LazyIndex.renameTo(LazyIndex.scala:79) > > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:496) > > at kafka.log.Log.$anonfun$replaceSegments$4(Log.scala:2338) > > at kafka.log.Log.$anonfun$replaceSegments$4$adapted(Log.scala:2338) > > at scala.collection.immutable.List.foreach(List.scala:333) > > at kafka.log.Log.replaceSegments(Log.scala:2338) > > at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:614) > > at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:539) > > at kafka.log.Cleaner.doClean(LogCleaner.scala:538) > > nsumer_offsets-31\00000000000000000000.index.swap: “The process cannot > access, because the other process is in use” > > > > at > sun.nio.fs.WindowsException.translateToIOException(Unknown Source) > > at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown > Source) > > at sun.nio.fs.WindowsFileCopy.move(Unknown Source) > > at sun.nio.fs.WindowsFileSystemProvider.move(Unknown > Source) > > at java.nio.file.Files.move(Unknown Source) > > at > org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:914) > > ... 18 more > > [2021-04-21 10:27:59,120] WARN [ReplicaManager broker=0] Stopping serving > replicas in dir C:\Testing\.logskafka-logs (kafka.server.ReplicaManager) > > [2021-04-21 10:27:59,124] INFO [ReplicaFetcherManager on broker 0] Removed > fetcher for partitions HashSet(fuzz_seed-36, __consumer_offsets-35, > __consumer_offsets-37, fuzz_seed-42, fuzz_seed-34, fuzz_seed-1, > __consumer_offsets-22, fuzz_seed-17, __consumer_offsets-30, fuzz_seed-12, > fuzz_seed-47, fuzz_seed-27, __consumer_offsets-8, __consumer_offsets-21, > __consumer_offsets-4, __consumer_offsets-27, __consumer_offsets-7, > __consumer_offsets-9, __consumer_offsets-46, fuzz_seed-33, fuzz_seed-2, > fuzz_seed-18, fuzz_seed-0, fuzz_seed-43, __consumer_offsets-25, > __consumer_offsets-41, __consumer_offsets-23, __consumer_offsets-49, > fuzz_seed-26, fuzz_seed-19, __consumer_offsets-33, fuzz_seed-30, > fuzz_seed-39, fuzz_seed-14, __consumer_offsets-47, __consumer_offsets-16, > fuzz_seed-3, fuzz_seed-48, __consumer_offsets-28, fuzz_seed-13, > __consumer_offsets-31, __consumer_offsets-36, __consumer_offsets-42, > __consumer_offsets-3, __consumer_offsets-18, fuzz_seed-46, fuzz_seed-31, > fuzz_seed-9, fuzz_seed-35, __consumer_offsets-15, __consumer_offsets-24, > fuzz_seed-29, fuzz_seed-4, fuzz_seed-20, fuzz_seed-23, > __consumer_offsets-38, __consumer_offsets-48, fuzz_seed-45, > __consumer_offsets-17, fuzz_seed-49, fuzz_seed-40, __consumer_offsets-19, > __consumer_offsets-11, fuzz_seed-21, fuzz_seed-16, fuzz_seed-8, > __consumer_offsets-13, fuzz_seed-10, __consumer_offsets-2, > __consumer_offsets-43, __consumer_offsets-6, fuzz_seed-38, > __consumer_offsets-14, fuzz_seed-25, fuzz_seed-22, fuzz_seed-6, > fuzz_seed-41, __consumer_offsets-20, __consumer_offsets-0, fuzz_seed-37, > fuzz_seed-32, fuzz_seed-11, __consumer_offsets-44, fuzz_seed-5, > __consumer_offsets-39, fuzz_seed-44, __consumer_offsets-12, fuzz_seed-28, > fuzz_seed-24, __consumer_offsets-45, __consumer_offsets-1, > __consumer_offsets-5, __consumer_offsets-26, __consumer_offsets-29, > __consumer_offsets-34, __consumer_offsets-10, __consumer_offsets-32, > fuzz_seed-7, __consumer_offsets-40, fuzz_seed-15) > (kafka.server.ReplicaFetcherManager) > > [2021-04-21 10:27:59,128] INFO [ReplicaAlterLogDirsManager on broker 0] > Removed fetcher for partitions HashSet(fuzz_seed-36, __consumer_offsets-35, > __consumer_offsets-37, fuzz_seed-42, fuzz_seed-34, fuzz_seed-1, > __consumer_offsets-22, fuzz_seed-17, __consumer_offsets-30, fuzz_seed-12, > fuzz_seed-47, fuzz_seed-27, __consumer_offsets-8, __consumer_offsets-21, > __consumer_offsets-4, __consumer_offsets-27, __consumer_offsets-7, > __consumer_offsets-9, __consumer_offsets-46, fuzz_seed-33, fuzz_seed-2, > fuzz_seed-18, fuzz_seed-0, fuzz_seed-43, __consumer_offsets-25, > __consumer_offsets-41, __consumer_offsets-23, __consumer_offsets-49, > fuzz_seed-26, fuzz_seed-19, __consumer_offsets-33, fuzz_seed-30, > fuzz_seed-39, fuzz_seed-14, __consumer_offsets-47, __consumer_offsets-16, > fuzz_seed-3, fuzz_seed-48, __consumer_offsets-28, fuzz_seed-13, > __consumer_offsets-31, __consumer_offsets-36, __consumer_offsets-42, > __consumer_offsets-3, __consumer_offsets-18, fuzz_seed-46, fuzz_seed-31, > fuzz_seed-9, fuzz_seed-35, __consumer_offsets-15, __consumer_offsets-24, > fuzz_seed-29, fuzz_seed-4, fuzz_seed-20, fuzz_seed-23, > __consumer_offsets-38, __consumer_offsets-48, fuzz_seed-45, > __consumer_offsets-17, fuzz_seed-49, fuzz_seed-40, __consumer_offsets-19, > __consumer_offsets-11, fuzz_seed-21, fuzz_seed-16, fuzz_seed-8, > __consumer_offsets-13, fuzz_seed-10, __consumer_offsets-2, > __consumer_offsets-43, __consumer_offsets-6, fuzz_seed-38, > __consumer_offsets-14, fuzz_seed-25, fuzz_seed-22, fuzz_seed-6, > fuzz_seed-41, __consumer_offsets-20, __consumer_offsets-0, fuzz_seed-37, > fuzz_seed-32, fuzz_seed-11, __consumer_offsets-44, fuzz_seed-5, > __consumer_offsets-39, fuzz_seed-44, __consumer_offsets-12, fuzz_seed-28, > fuzz_seed-24, __consumer_offsets-45, __consumer_offsets-1, > __consumer_offsets-5, __consumer_offsets-26, __consumer_offsets-29, > __consumer_offsets-34, __consumer_offsets-10, __consumer_offsets-32, > fuzz_seed-7, __consumer_offsets-40, fuzz_seed-15) > (kafka.server.ReplicaAlterLogDirsManager) > > [2021-04-21 10:27:59,220] WARN [ReplicaManager broker=0] Broker 0 stopped > fetcher for partitions > > fuzz_seed-36,__consumer_offsets-35,__consumer_offsets-37,fuzz_seed-42,fuzz_seed-34,fuzz_seed-1,__consumer_offsets-22,fuzz_seed-17,__consumer_offsets-30,fuzz_seed-12,fuzz_seed-47,fuzz_seed-27,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,fuzz_seed-33,fuzz_seed-2,fuzz_seed-18,fuzz_seed-0,fuzz_seed-43,__consumer_offsets-25,__consumer_offsets-41,__consumer_offsets-23,__consumer_offsets-49,fuzz_seed-26,fuzz_seed-19,__consumer_offsets-33,fuzz_seed-30,fuzz_seed-39,fuzz_seed-14,__consumer_offsets-47,__consumer_offsets-16,fuzz_seed-3,fuzz_seed-48,__consumer_offsets-28,fuzz_seed-13,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,__consumer_offsets-3,__consumer_offsets-18,fuzz_seed-46,fuzz_seed-31,fuzz_seed-9,fuzz_seed-35,__consumer_offsets-15,__consumer_offsets-24,fuzz_seed-29,fuzz_seed-4,fuzz_seed-20,fuzz_seed-23,__consumer_offsets-38,__consumer_offsets-48,fuzz_seed-45,__consumer_offsets-17,fuzz_seed-49,fuzz_seed-40,__consumer_offsets-19,__consumer_offsets-11,fuzz_seed-21,fuzz_seed-16,fuzz_seed-8,__consumer_offsets-13,fuzz_seed-10,__consumer_offsets-2,__consumer_offsets-43,__consumer_offsets-6,fuzz_seed-38,__consumer_offsets-14,fuzz_seed-25,fuzz_seed-22,fuzz_seed-6,fuzz_seed-41,__consumer_offsets-20,__consumer_offsets-0,fuzz_seed-37,fuzz_seed-32,fuzz_seed-11,__consumer_offsets-44,fuzz_seed-5,__consumer_offsets-39,fuzz_seed-44,__consumer_offsets-12,fuzz_seed-28,fuzz_seed-24,__consumer_offsets-45,__consumer_offsets-1,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,fuzz_seed-7,__consumer_offsets-40,fuzz_seed-15 > and stopped moving logs for partitions because they are in the failed log > directory C:\Testing\.logskafka-logs. (kafka.server.ReplicaManager) > > [2021-04-21 10:27:59,224] WARN Stopping serving logs in dir > C:\Testing\.logskafka-logs (kafka.log.LogManager) > > [2021-04-21 10:27:59,232] ERROR Shutdown broker because all log dirs in > C:\Testing\.logskafka-logs have failed (kafka.log.LogManager) >
