[jira] [Updated] (KAFKA-4502) Exception during startup, append offset to swap file
[ https://issues.apache.org/jira/browse/KAFKA-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harald Kirsch updated KAFKA-4502: - Labels: (was: log) > Exception during startup, append offset to swap file > > > Key: KAFKA-4502 > URL: https://issues.apache.org/jira/browse/KAFKA-4502 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.10.2.0 > Environment: Windows Server >Reporter: Harald Kirsch > > During startup, the Kafka server throws the exception shown below with a bit > of pre-context. > We are using the so-called SiphonRelease > (https://github.com/Microsoft/Kafka/tree/SiphonRelease, > https://issues.apache.org/jira/browse/KAFKA-1194?focusedCommentId=15702991) > which tries to circumvent problems of the logCleaner to rename and delete > segments still memory mapped by the broker. > The trouble seems to be as follows: since in the SiphonRelease the LogCleaner > still sometimes crashes, we have a monitoring script that detects this and > then restarts the Windows Service (apache procrun based) running Kafka. My > hunch is that the combination of restart-service/procrun does not allow Kafka > to shut down smoothly, since when it starts we get tons of messages like: > {noformat} > [2016-12-05 23:30:20,704] WARN Found a corrupted index file due to > requirement failed: Corrupt index found, index file > (d:\Search\kafka\fileshare-0\00084814.index) has non-zero size > but the last offset is 84814 which is no larger than the base offset 84814.}. > deleting d:\Search\kafka\fileshare-0\00084814.timeindex, > d:\Search\kafka\fileshare-0\00084814.index and rebuilding > index... (kafka.log.Log) > {noformat} > While this seems fixable by Kafka, my hunch is that a leftover .swap file > then breaks it as follows: > {noformat} > [2016-12-05 23:32:34,676] INFO Found log file > d:\Search\kafka\windream-4\.log.swap from interrupted > swap operation, repairing. (kafka.log.Log) > [2016-12-05 23:32:34,957] ERROR There was an error in one of the threads > during logs loading: kafka.common.InvalidOffsetException: Attempt to append > an offset (110460) to position 182 no larger than the last offset appended > (110735) to d:\Search\kafka\windream-4\.index.swap. > (kafka.log.LogManager) > [2016-12-05 23:32:34,957] FATAL Fatal error during KafkaServer startup. > Prepare to shutdown (kafka.server.KafkaServer) > kafka.common.InvalidOffsetException: Attempt to append an offset (110460) to > position 182 no larger than the last offset appended (110735) to > d:\Search\kafka\windream-4\.index.swap. > at > kafka.log.OffsetIndex$$anonfun$append$1.apply$mcV$sp(OffsetIndex.scala:132) > at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:122) > at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:122) > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234) > at kafka.log.OffsetIndex.append(OffsetIndex.scala:122) > at kafka.log.LogSegment.recover(LogSegment.scala:224) > at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:248) > at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:232) > at scala.collection.immutable.Set$Set1.foreach(Set.scala:74) > at kafka.log.Log.loadSegments(Log.scala:232) > at kafka.log.Log.(Log.scala:108) > at > kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:151) > at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:58) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-4502) Exception during startup, append offset to swap file
Harald Kirsch created KAFKA-4502: Summary: Exception during startup, append offset to swap file Key: KAFKA-4502 URL: https://issues.apache.org/jira/browse/KAFKA-4502 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.10.2.0 Environment: Windows Server Reporter: Harald Kirsch During startup, the Kafka server throws the exception shown below with a bit of pre-context. We are using the so-called SiphonRelease (https://github.com/Microsoft/Kafka/tree/SiphonRelease, https://issues.apache.org/jira/browse/KAFKA-1194?focusedCommentId=15702991) which tries to circumvent problems of the logCleaner to rename and delete segments still memory mapped by the broker. The trouble seems to be as follows: since in the SiphonRelease the LogCleaner still sometimes crashes, we have a monitoring script that detects this and then restarts the Windows Service (apache procrun based) running Kafka. My hunch is that the combination of restart-service/procrun does not allow Kafka to shut down smoothly, since when it starts we get tons of messages like: {noformat} [2016-12-05 23:30:20,704] WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (d:\Search\kafka\fileshare-0\00084814.index) has non-zero size but the last offset is 84814 which is no larger than the base offset 84814.}. deleting d:\Search\kafka\fileshare-0\00084814.timeindex, d:\Search\kafka\fileshare-0\00084814.index and rebuilding index... (kafka.log.Log) {noformat} While this seems fixable by Kafka, my hunch is that a leftover .swap file then breaks it as follows: {noformat} [2016-12-05 23:32:34,676] INFO Found log file d:\Search\kafka\windream-4\.log.swap from interrupted swap operation, repairing. (kafka.log.Log) [2016-12-05 23:32:34,957] ERROR There was an error in one of the threads during logs loading: kafka.common.InvalidOffsetException: Attempt to append an offset (110460) to position 182 no larger than the last offset appended (110735) to d:\Search\kafka\windream-4\.index.swap. (kafka.log.LogManager) [2016-12-05 23:32:34,957] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) kafka.common.InvalidOffsetException: Attempt to append an offset (110460) to position 182 no larger than the last offset appended (110735) to d:\Search\kafka\windream-4\.index.swap. at kafka.log.OffsetIndex$$anonfun$append$1.apply$mcV$sp(OffsetIndex.scala:132) at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:122) at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:122) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234) at kafka.log.OffsetIndex.append(OffsetIndex.scala:122) at kafka.log.LogSegment.recover(LogSegment.scala:224) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:248) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:232) at scala.collection.immutable.Set$Set1.foreach(Set.scala:74) at kafka.log.Log.loadSegments(Log.scala:232) at kafka.log.Log.(Log.scala:108) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:151) at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15712362#comment-15712362 ] Harald Kirsch commented on KAFKA-1194: -- [~soumyajitsahu] Here are the server configs as logged: {code} [2016-11-29 16:23:13,731] INFO KafkaConfig values: advertised.host.name = null advertised.listeners = null advertised.port = null authorizer.class.name = auto.create.topics.enable = false auto.leader.rebalance.enable = true background.threads = 10 broker.id = 0 broker.id.generation.enable = true broker.rack = null compression.type = producer connections.max.idle.ms = 60 controlled.shutdown.enable = true controlled.shutdown.max.retries = 3 controlled.shutdown.retry.backoff.ms = 5000 controller.socket.timeout.ms = 3 default.replication.factor = 1 delete.topic.enable = false fetch.purgatory.purge.interval.requests = 1000 group.max.session.timeout.ms = 30 group.min.session.timeout.ms = 6000 host.name = inter.broker.protocol.version = 0.10.1-IV2 leader.imbalance.check.interval.seconds = 300 leader.imbalance.per.broker.percentage = 10 listeners = PLAINTEXT://:9092 log.cleaner.backoff.ms = 15000 log.cleaner.dedupe.buffer.size = 134217728 log.cleaner.delete.retention.ms = 8640 log.cleaner.enable = true log.cleaner.io.buffer.load.factor = 0.9 log.cleaner.io.buffer.size = 524288 log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308 log.cleaner.min.cleanable.ratio = 0.1 log.cleaner.min.compaction.lag.ms = 0 log.cleaner.threads = 1 log.cleanup.policy = [compact] log.dir = /tmp/kafka-logs log.dirs = d:\Search\kafka log.flush.interval.messages = 9223372036854775807 log.flush.interval.ms = null log.flush.offset.checkpoint.interval.ms = 6 log.flush.scheduler.interval.ms = 9223372036854775807 log.index.interval.bytes = 4096 log.index.size.max.bytes = 10485760 log.message.format.version = 0.10.1-IV2 log.message.timestamp.difference.max.ms = 9223372036854775807 log.message.timestamp.type = CreateTime log.preallocate = false log.retention.bytes = -1 log.retention.check.interval.ms = 300100 log.retention.hours = 168 log.retention.minutes = null log.retention.ms = null log.roll.hours = 24 log.roll.jitter.hours = 0 log.roll.jitter.ms = null log.roll.ms = null log.segment.bytes = 200111000 log.segment.delete.delay.ms = 6 max.connections.per.ip = 2147483647 max.connections.per.ip.overrides = message.max.bytes = 20999000 metric.reporters = [] metrics.num.samples = 2 metrics.sample.window.ms = 3 min.insync.replicas = 1 num.io.threads = 36 num.network.threads = 36 num.partitions = 1 num.recovery.threads.per.data.dir = 1 num.replica.fetchers = 1 offset.metadata.max.bytes = 4096 offsets.commit.required.acks = -1 offsets.commit.timeout.ms = 5000 offsets.load.buffer.size = 5242880 offsets.retention.check.interval.ms = 60 offsets.retention.minutes = 2147483 offsets.topic.compression.codec = 0 offsets.topic.num.partitions = 50 offsets.topic.replication.factor = 3 offsets.topic.segment.bytes = 104857600 port = 9092 principal.builder.class = class org.apache.kafka.common.security.auth.DefaultPrincipalBuilder producer.purgatory.purge.interval.requests = 1000 queued.max.requests = 500 quota.consumer.default = 9223372036854775807 quota.producer.default = 9223372036854775807 quota.window.num = 11 quota.window.size.seconds = 1 replica.fetch.backoff.ms = 1000 replica.fetch.max.bytes = 20999000 replica.fetch.min.bytes = 1 replica.fetch.response.max.bytes = 10485760 replica.fetch.wait.max.ms = 500 replica.high.watermark.checkpoint.interval.ms = 5000 replica.lag.time.max.ms = 1 replica.socket.receive.buffer.bytes = 65536 replica.socket.timeout.ms = 3 replication.quota.throttled.rate = 9223372036854775807 replication.quota.window.num = 11 replication.quota.window.size.seconds = 1 request.timeout.ms = 32 reserved.broker.max.id = 1000 sasl.enabled.mechanisms = [GSSAPI] sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 6 sasl.kerberos.principal.to.local.rules = [DEFAULT] sasl.kerberos.service.name =
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708281#comment-15708281 ] Harald Kirsch commented on KAFKA-1194: -- The report of success was slightly exaggerted :-( After several hours of flawless operation, cleaner threads manage to trip over their own feed, it seems, with a variety of exceptions. I have seen this now with a single cleaner thread as well as with 4 threads. With 4 threads, I got variant 1 below 2 times then variant 2, then variant three but all within 10 seconds. I wonder in what kind of mood (mode) the system is that this all happens within a few seconds. Let me know if I can help with more information, details of the setup, configuration details or experiments, debug logging. Variant 1: {noformat} [2016-11-30 08:26:26,576] ERROR [kafka-log-cleaner-thread-1], Error due to (kafka.log.LogCleaner) kafka.common.KafkaStorageException: Failed to change the log file suffix from to .deleted for log segment 58972 at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:327) at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:329) at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:956) at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:1002) at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:997) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Log.replaceSegments(Log.scala:997) at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:425) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:364) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Cleaner.clean(LogCleaner.scala:363) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:239) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:218) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) Caused by: java.nio.file.FileAlreadyExistsException: d:\Search\kafka\fileshare-10\00058972.log -> d:\Search\kafka\fileshare-10\00058972.log.deleted at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:81) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670) at kafka.log.FileMessageSet.renameTo(FileMessageSet.scala:431) ... 14 more Suppressed: java.nio.file.AccessDeniedException: d:\Search\kafka\fileshare-10\00058972.log -> d:\Search\kafka\fileshare-10\00058972.log.deleted at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667) ... 15 more {noformat} Variant 2: {noformat} [2016-11-30 08:26:30,467] ERROR [kafka-log-cleaner-thread-3], Error due to (kafka.log.LogCleaner) kafka.common.InvalidOffsetException: Attempt to append an offset (59264) to position 61 no larger than the last offset appended (66994) to d:\Search\kafka\fileshare-10\.index.swap.cleaned. at kafka.log.OffsetIndex$$anonfun$append$1.apply$mcV$sp(OffsetIndex.scala:132) at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:122) at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:122) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234) at kafka.log.OffsetIndex.append(OffsetIndex.scala:122) at kafka.log.LogSegment.append(LogSegment.scala:105) at kafka.log.Cleaner.cleanInto(LogCleaner.scala:504) at kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:404) at kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:400) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:400) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:364) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Cleane
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705770#comment-15705770 ] Harald Kirsch commented on KAFKA-1194: -- Now I tried exactly the SiphonRelease mentioned three comments up and it works great. So great indeed that after testing on QA I could move it to our production where it is happily eating away outdated segments while the rest of the operations is going on just fine. Many thanks to [~soumyajitsahu] for providing this release. > The kafka broker cannot delete the old log files after the configured time > -- > > Key: KAFKA-1194 > URL: https://issues.apache.org/jira/browse/KAFKA-1194 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.1 > Environment: window >Reporter: Tao Qin >Assignee: Jay Kreps > Labels: features, patch > Fix For: 0.10.2.0 > > Attachments: KAFKA-1194.patch, Untitled.jpg, kafka-1194-v1.patch, > kafka-1194-v2.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > We tested it in windows environment, and set the log.retention.hours to 24 > hours. > # The minimum age of a log file to be eligible for deletion > log.retention.hours=24 > After several days, the kafka broker still cannot delete the old log file. > And we get the following exceptions: > [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task > 'kafka-log-retention' (kafka.utils.KafkaScheduler) > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 1516723 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249) > at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638) > at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) > at scala.collection.immutable.List.foreach(List.scala:76) > at kafka.log.Log.deleteOldSegments(Log.scala:418) > at > kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:284) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:316) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:314) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743) > at scala.collection.Iterator$class.foreach(Iterator.scala:772) > at > scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:73) > at > scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742) > at kafka.log.LogManager.cleanupLogs(LogManager.scala:314) > at > kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:143) > at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > I think this error happens because kafka tries to rename the log file when it > is still opened. So we should close the file first before rename. > The index file uses a special data structure, the MappedByteBuffer. Javadoc > describes it as: > A mapped byte buffer and the file mapping that it represents remain valid > until the buffer itself is garbage-collected. > Fortunately, I find a forceUnmap function in kafka code, and perhaps it can > be used to free the MappedByteBuffer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15701308#comment-15701308 ] Harald Kirsch commented on KAFKA-1194: -- See https://issues.apache.org/jira/browse/KAFKA-2170?focusedCommentId=15515792&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15515792 for a description of how I applied a patch (and which) and what I experienced. Maybe it works good enough to help you. But surely try this on a test system first. > The kafka broker cannot delete the old log files after the configured time > -- > > Key: KAFKA-1194 > URL: https://issues.apache.org/jira/browse/KAFKA-1194 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.1 > Environment: window >Reporter: Tao Qin >Assignee: Jay Kreps > Labels: features, patch > Fix For: 0.10.2.0 > > Attachments: KAFKA-1194.patch, kafka-1194-v1.patch, > kafka-1194-v2.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > We tested it in windows environment, and set the log.retention.hours to 24 > hours. > # The minimum age of a log file to be eligible for deletion > log.retention.hours=24 > After several days, the kafka broker still cannot delete the old log file. > And we get the following exceptions: > [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task > 'kafka-log-retention' (kafka.utils.KafkaScheduler) > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 1516723 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249) > at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638) > at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) > at scala.collection.immutable.List.foreach(List.scala:76) > at kafka.log.Log.deleteOldSegments(Log.scala:418) > at > kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:284) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:316) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:314) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743) > at scala.collection.Iterator$class.foreach(Iterator.scala:772) > at > scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:73) > at > scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742) > at kafka.log.LogManager.cleanupLogs(LogManager.scala:314) > at > kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:143) > at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > I think this error happens because kafka tries to rename the log file when it > is still opened. So we should close the file first before rename. > The index file uses a special data structure, the MappedByteBuffer. Javadoc > describes it as: > A mapped byte buffer and the file mapping that it represents remain valid > until the buffer itself is garbage-collected. > Fortunately, I find a forceUnmap function in kafka code, and perhaps it can > be used to free the MappedByteBuffer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15532092#comment-15532092 ] Harald Kirsch commented on KAFKA-1194: -- [~abhit011] The patch described in https://issues.apache.org/jira/browse/KAFKA-2170?focusedCommentId=15515023&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15515023 nearly worked for me, but not quite. Maybe it furthers the issue if you could try the patch as well on a test system and report your findings. > The kafka broker cannot delete the old log files after the configured time > -- > > Key: KAFKA-1194 > URL: https://issues.apache.org/jira/browse/KAFKA-1194 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.1 > Environment: window >Reporter: Tao Qin >Assignee: Jay Kreps > Labels: features, patch > Fix For: 0.10.2.0 > > Attachments: KAFKA-1194.patch, kafka-1194-v1.patch, > kafka-1194-v2.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > We tested it in windows environment, and set the log.retention.hours to 24 > hours. > # The minimum age of a log file to be eligible for deletion > log.retention.hours=24 > After several days, the kafka broker still cannot delete the old log file. > And we get the following exceptions: > [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task > 'kafka-log-retention' (kafka.utils.KafkaScheduler) > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 1516723 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249) > at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638) > at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) > at scala.collection.immutable.List.foreach(List.scala:76) > at kafka.log.Log.deleteOldSegments(Log.scala:418) > at > kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:284) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:316) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:314) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743) > at scala.collection.Iterator$class.foreach(Iterator.scala:772) > at > scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:73) > at > scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742) > at kafka.log.LogManager.cleanupLogs(LogManager.scala:314) > at > kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:143) > at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > I think this error happens because kafka tries to rename the log file when it > is still opened. So we should close the file first before rename. > The index file uses a special data structure, the MappedByteBuffer. Javadoc > describes it as: > A mapped byte buffer and the file mapping that it represents remain valid > until the buffer itself is garbage-collected. > Fortunately, I find a forceUnmap function in kafka code, and perhaps it can > be used to free the MappedByteBuffer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows
[ https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515792#comment-15515792 ] Harald Kirsch edited comment on KAFKA-2170 at 9/23/16 8:37 AM: --- Better, but not yet there, it seems. Here is what I did. I applied the .diff file of the 1757 pull request as downloaded from github with the patch command to 106a7456060750ab0604d290b8c1e33a57adbf20 from http://git-wip-us.apache.org/repos/asf/kafka.git. It ran in just fin. Build the tgz, put it on my test machine, ran kafka with these settings: {code} log.segment.bytes=6000111 log.cleaner.enable=true log.cleanup.policy=compact log.cleaner.min.cleanable.ratio=0.01 log.cleaner.backoff.ms=15000 log.segment.delete.delay.ms=600 auto.create.topics.enable=false {code} Ran in around 75 files as binary blobs between 10k and 6M in size twice. The cleanup triggered and worked just fine. Tried this a few times more, also with running the files in in quick succession and it worked just fine. Stopped Kafka with CTRL-C, it shut down nicely an restarted (mentioning just for completeness, not sure whether this is relevant). Tried this a few more times, running the files in roughly 15 times in quick succession and it bombed out as shown follows: {code} kafka.common.KafkaStorageException: Failed to change the log file suffix from to .deleted for log segment 695 at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:327) at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:329) at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:956) at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:1002) at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:997) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Log.replaceSegments(Log.scala:997) at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:425) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:364) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Cleaner.clean(LogCleaner.scala:363) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:239) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:218) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) Caused by: java.nio.file.FileAlreadyExistsException: C:\Users\hk\tmp\kafka-data\hktest-0\0695.log -> C:\Users\hk\tmp\kafka-data\hktest-0\0695.log.deleted at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:81) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670) at kafka.log.FileMessageSet.renameTo(FileMessageSet.scala:431) ... 14 more Suppressed: java.nio.file.AccessDeniedException: C:\Users\hk\tmp\kafka-data\hktest-0\0695.log -> C:\Users\hk\tmp\kafka-data\hktest-0\0695.log.deleted at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667) ... 15 more [2016-09-23 10:12:36,254] INFO [kafka-log-cleaner-thread-0], Stopped (kafka.log.LogCleaner) {code} Afterwards I restartet Kafka, which worked without complaints. I ran one round on the 75 files and the logcleaner cleaned up just fine, so it looks like you're pretty close to a working solution. Let me know if I need to provide more information or run a different experiment. was (Author: haraldk): Better, but not yet there, it seems. Here is what I did. I applied the .diff file of the 1757 pull request as downloaded from github with the patch command to 106a7456060750ab0604d290b8c1e33a57adbf20 from http://git-wip-us.apache.org/repos/asf/kafka.git. It ran in just fin. Build the tgz, put it on my test machine, ran kafka with these settings: {code} log.segment.bytes=6000111 log.cleaner.enable=true log.cleanup.policy=compact log.cleaner.min.cleanable.ratio=0.01 log.cleaner.backoff.ms=15000 log.segment.delete.delay.ms=600 auto.create.topics.
[jira] [Commented] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows
[ https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515792#comment-15515792 ] Harald Kirsch commented on KAFKA-2170: -- Better, but not yet there, it seems. Here is what I did. I applied the .diff file of the 1757 pull request as downloaded from github with the patch command to 106a7456060750ab0604d290b8c1e33a57adbf20 from http://git-wip-us.apache.org/repos/asf/kafka.git. It ran in just fin. Build the tgz, put it on my test machine, ran kafka with these settings: {code} log.segment.bytes=6000111 log.cleaner.enable=true log.cleanup.policy=compact log.cleaner.min.cleanable.ratio=0.01 log.cleaner.backoff.ms=15000 log.segment.delete.delay.ms=600 auto.create.topics.enable=false {code} Ran in around 75 files as binary blobs between 10k and 6M in size twice. The cleanup triggered and worked just fine. Tried this a few times more, also with running the files in in quick succession and it worked just fine. Stopped Kafka with CTRL-C, it shut down nicely an restarted (mentioning just for completeness, not sure whether this is relevant). Tried this a few more times, running the files in roughly 15 times in quick succession and it bombed out as shown follows: {code} kafka.common.KafkaStorageException: Failed to change the log file suffix from to .deleted for log segment 695 at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:327) at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:329) at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:956) at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:1002) at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:997) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Log.replaceSegments(Log.scala:997) at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:425) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:364) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Cleaner.clean(LogCleaner.scala:363) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:239) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:218) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) Caused by: java.nio.file.FileAlreadyExistsException: C:\Users\hk\tmp\kafka-data\hktest-0\0695.log -> C:\Users\hk\tmp\kafka-data\hktest-0\0695.log.deleted at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:81) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670) at kafka.log.FileMessageSet.renameTo(FileMessageSet.scala:431) ... 14 more Suppressed: java.nio.file.AccessDeniedException: C:\Users\hk\tmp\kafka-data\hktest-0\0695.log -> C:\Users\hk\tmp\kafka-data\hktest-0\0695.log.deleted at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667) ... 15 more [2016-09-23 10:12:36,254] INFO [kafka-log-cleaner-thread-0], Stopped (kafka.log.LogCleaner) {code} Let me know if I need to provide more information or run a different experiment. > 10 LogTest cases failed for file.renameTo failed under windows > --- > > Key: KAFKA-2170 > URL: https://issues.apache.org/jira/browse/KAFKA-2170 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.10.1.0 > Environment: Windows >Reporter: Honghai Chen >Assignee: Jay Kreps > > get latest code from trunk, then run test > gradlew -i core:test --tests kafka.log.LogTest > Got 10 cases failed for same reason: > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 0 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:259) > at kafka.log.Log.kafka$log$Log$$asyncD
[jira] [Commented] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows
[ https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515615#comment-15515615 ] Harald Kirsch commented on KAFKA-2170: -- [~soumyajitsahu] This is great news. I assume the pull request goes agains the most recent trunk. I will try this out no. > 10 LogTest cases failed for file.renameTo failed under windows > --- > > Key: KAFKA-2170 > URL: https://issues.apache.org/jira/browse/KAFKA-2170 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.10.1.0 > Environment: Windows >Reporter: Honghai Chen >Assignee: Jay Kreps > > get latest code from trunk, then run test > gradlew -i core:test --tests kafka.log.LogTest > Got 10 cases failed for same reason: > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 0 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:259) > at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:756) > at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:747) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514) > at scala.collection.immutable.List.foreach(List.scala:318) > at kafka.log.Log.deleteOldSegments(Log.scala:514) > at kafka.log.LogTest.testAsyncDelete(LogTest.scala:633) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41) > at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) > at org.junit.runners.ParentRunner.run(ParentRunner.java:220) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:48) > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) > at > org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) > at > org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) > at $Proxy2.processTestClass(Unknown Source) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:105) > at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) > at > org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gra
[jira] [Commented] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows
[ https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438668#comment-15438668 ] Harald Kirsch commented on KAFKA-2170: -- Tried to added the patches. Since I had the 1757 in already, I only added the 1716 (one liner). The 1718 was in my code already too (prevent null DirectBuffer use). The test results are exactly as in my last comment, namely that the .index.deleted files are not really deleted (with no error message). The next time the cleaner runs, it complains about not being able to rename to the already existing file (no surprise). So the remaining problem seems to that even with a low log.segment.delete.delay.ms the files are not deleted. And also their counterparts without .delete, despite being empty, are not deleted. > 10 LogTest cases failed for file.renameTo failed under windows > --- > > Key: KAFKA-2170 > URL: https://issues.apache.org/jira/browse/KAFKA-2170 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.10.1.0 > Environment: Windows >Reporter: Honghai Chen >Assignee: Jay Kreps > > get latest code from trunk, then run test > gradlew -i core:test --tests kafka.log.LogTest > Got 10 cases failed for same reason: > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 0 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:259) > at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:756) > at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:747) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514) > at scala.collection.immutable.List.foreach(List.scala:318) > at kafka.log.Log.deleteOldSegments(Log.scala:514) > at kafka.log.LogTest.testAsyncDelete(LogTest.scala:633) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41) > at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) > at org.junit.runners.ParentRunner.run(ParentRunner.java:220) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:48) > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) > at > org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) > at > org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) > at $Proxy2.processTestClass(Unknown Source) > at > org.gradle.api.internal.tasks.testing.worker.
[jira] [Commented] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows
[ https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15426278#comment-15426278 ] Harald Kirsch commented on KAFKA-2170: -- The patch does not seem to work completely for compaction. I applied the patch on 40b1dd3f495a59abef8a0cba5450526994c92c04, ran ./gradlew releaseTarGz and unpacked on a windows 8.1. I ran the server with the following configs: {noformat} broker.id=0 listeners=PLAINTEXT://:9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=C:\\Users\\hk\\tmp\\kafka-data num.partitions=1 num.recovery.threads.per.data.dir=1 log.retention.hours=168 log.segment.bytes=6000111 log.retention.check.interval.ms=30 zookeeper.connect=hal.intranet.raytion.com:2181/kafka zookeeper.connection.timeout.ms=6000 log.cleaner.enable=true log.cleanup.policy=compact log.cleaner.min.cleanable.ratio=0.01 log.cleaner.backoff.ms=15000 log.segment.delete.delay.ms=600 auto.create.topics.enable=false {noformat} Then I fed the same set of docs twice. The cleaner logged a successful activity, but the folder now contains all {{.index.deleted}} files like this: {noformat} ModeLastWriteTime Length Name - -- -a---18.08.2016 12:33 0 .index -18.08.2016 12:33176 .index.deleted -a---18.08.2016 12:33 0 .log -a---18.08.2016 12:33 0 0023.index -18.08.2016 12:33192 0023.index.deleted -a---18.08.2016 12:33 583752 0023.log -a---18.08.2016 12:33 48 0048.index -18.08.2016 12:33192 0048.index.deleted -a---18.08.2016 12:335844328 0048.log -a---18.08.2016 12:33 48 0073.index -18.08.2016 12:33200 0073.index.deleted -a---18.08.2016 12:335494169 0073.log -a---18.08.2016 12:33 10485760 0099.index -a---18.08.2016 12:331529831 0099.log {noformat} Then I fed the same set of messages a third time. As soon as the cleaner started working thereafter, it bombed out like this: {noformat} [2016-08-18 13:10:43,254] ERROR [kafka-log-cleaner-thread-0], Error due to (kafka.log.LogCleaner) kafka.common.KafkaStorageException: Failed to change the index file suffix from to .deleted for log segment 0 at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:263) at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:269) at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:832) at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:878) at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:873) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Log.replaceSegments(Log.scala:873) at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:395) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:343) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:342) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Cleaner.clean(LogCleaner.scala:342) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:237) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:215) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) Caused by: java.nio.file.FileAlreadyExistsException: C:\Users\hk\tmp\kafka-data\hktest-0\.index -> C:\Users\hk\tmp\kafka-data\hktest-0\.index.deleted at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:81) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670) at kafka.log.OffsetIndex.renameTo(OffsetIndex.scala:364) ... 14 more Suppressed: java.nio.file.AccessDeniedException: C:\Users\hk\tmp\kafka-data\hktest-0\.index -> C:\Users\hk\tmp\kafka-data\hktest-0\.index.deleted at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405810#comment-15405810 ] Harald Kirsch commented on KAFKA-1194: -- Just stumbled over yet another problem instance. During startup, Kafka notices a corrupt log/index file and tries to repair it. Here is the stack trace: {noformat} [2016-08-03 13:56:17,467] INFO Found log file d:\Search\kafka\fileshare-1\.log.swap from interrupted swap operation, repairing. (kafka.log.Log) [2016-08-03 13:56:18,436] ERROR There was an error in one of the threads during logs loading: kafka.common.KafkaStorageException: Failed to change the index file suffix from .swap to for log segment 0 (kafka.log.LogManager) [2016-08-03 13:56:18,436] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) kafka.common.KafkaStorageException: Failed to change the index file suffix from .swap to for log segment 0 at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:268) at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:274) at kafka.log.Log.replaceSegments(Log.scala:886) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:230) at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:214) at scala.collection.immutable.Set$Set1.foreach(Set.scala:74) at kafka.log.Log.loadSegments(Log.scala:214) at kafka.log.Log.(Log.scala:101) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:151) at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:56) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.nio.file.FileSystemException: d:\Search\kafka\fileshare-1\.index.swap -> d:\Search\kafka\fileshare-1\.index: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670) at kafka.log.OffsetIndex.renameTo(OffsetIndex.scala:365) ... 14 more Suppressed: java.nio.file.FileSystemException: d:\Search\kafka\fileshare-1\.index.swap -> d:\Search\kafka\fileshare-1\.index: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667) ... 15 more [2016-08-03 13:56:18,451] INFO shutting down (kafka.server.KafkaServer) [2016-08-03 13:56:18,467] INFO shut down completed (kafka.server.KafkaServer) {noformat} > The kafka broker cannot delete the old log files after the configured time > -- > > Key: KAFKA-1194 > URL: https://issues.apache.org/jira/browse/KAFKA-1194 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.1 > Environment: window >Reporter: Tao Qin >Assignee: Jay Kreps > Labels: features, patch > Fix For: 0.10.1.0 > > Attachments: KAFKA-1194.patch, kafka-1194-v1.patch, > kafka-1194-v2.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > We tested it in windows environment, and set the log.retention.hours to 24 > hours. > # The minimum age of a log file to be eligible for deletion > log.retention.hours=24 > After several days, the kafka broker still cannot delete the old log file. > And we get the following exceptions: > [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task > 'kafka-log-retention' (kafka.utils
[jira] [Comment Edited] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379380#comment-15379380 ] Harald Kirsch edited comment on KAFKA-1194 at 7/15/16 1:41 PM: --- It seems we are one step further but not yet there. I just cloned the master, pulled in 1624.patch, installed and ran kafka on an existing log. The message has changed from not being able to rename the log file to not being able to rename the index file. Here is the full stack trace from the LogCleaner. {noformat} [2016-07-15 15:33:09,622] ERROR [kafka-log-cleaner-thread-0], Error due to (kafka.log.LogCleaner) kafka.common.KafkaStorageException: Failed to change the index file suffix from to .deleted for log segment 0 at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:268) at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:274) at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:837) at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:883) at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:878) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Log.replaceSegments(Log.scala:878) at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:395) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:343) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:342) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Cleaner.clean(LogCleaner.scala:342) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:237) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:215) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) Caused by: java.nio.file.FileSystemException: d:\Search\kafka\windream-9\.index -> d:\Search\kafka\windream-9\.index.deleted: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670) at kafka.log.OffsetIndex.renameTo(OffsetIndex.scala:365) ... 14 more Suppressed: java.nio.file.FileSystemException: d:\Search\kafka\windream-9\.index -> d:\Search\kafka\windream-9\.index.deleted: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667) ... 15 more [2016-07-15 15:33:09,622] INFO [kafka-log-cleaner-thread-0], Stopped (kafka.log.LogCleaner) {noformat} Indeed the overly long {{.log}} file is now {{.log.deleted}}. There are {{.log.swap}} and {{.index.swap}} and still just {{.index}}. For reference, the server.log lists: {noformat} memorymapped.file.updates.enabled = false {noformat} was (Author: haraldk): It seems we are one step further but not yet there. I just cloned the master, pulled in 1624.patch, installed and ran kafka on an existing log. The message has changed from not being able to rename the log file to not being able to rename the index file. Here is the full stack trace from the LogCleaner. {noformat} [2016-07-15 15:33:09,622] ERROR [kafka-log-cleaner-thread-0], Error due to (kafka.log.LogCleaner) kafka.common.KafkaStorageException: Failed to change the index file suffix from to .deleted for log segment 0 at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:268) at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:274) at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:837) at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:883) at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:878) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Log.replaceSegments(Log.scala:878) at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:395) at kafka.log.Cl
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379380#comment-15379380 ] Harald Kirsch commented on KAFKA-1194: -- It seems we are one step further but not yet there. I just cloned the master, pulled in 1624.patch, installed and ran kafka on an existing log. The message has changed from not being able to rename the log file to not being able to rename the index file. Here is the full stack trace from the LogCleaner. {noformat} [2016-07-15 15:33:09,622] ERROR [kafka-log-cleaner-thread-0], Error due to (kafka.log.LogCleaner) kafka.common.KafkaStorageException: Failed to change the index file suffix from to .deleted for log segment 0 at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:268) at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:274) at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:837) at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:883) at kafka.log.Log$$anonfun$replaceSegments$1.apply(Log.scala:878) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Log.replaceSegments(Log.scala:878) at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:395) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:343) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:342) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Cleaner.clean(LogCleaner.scala:342) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:237) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:215) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) Caused by: java.nio.file.FileSystemException: d:\Search\kafka\windream-9\.index -> d:\Search\kafka\windream-9\.index.deleted: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670) at kafka.log.OffsetIndex.renameTo(OffsetIndex.scala:365) ... 14 more Suppressed: java.nio.file.FileSystemException: d:\Search\kafka\windream-9\.index -> d:\Search\kafka\windream-9\.index.deleted: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667) ... 15 more [2016-07-15 15:33:09,622] INFO [kafka-log-cleaner-thread-0], Stopped (kafka.log.LogCleaner) {noformat} Indeed the overly long {{.log}} file is now {{.log.deleted}}. There are {{.log.swap}} and {{.index.swap}} and still just {{.index}}. > The kafka broker cannot delete the old log files after the configured time > -- > > Key: KAFKA-1194 > URL: https://issues.apache.org/jira/browse/KAFKA-1194 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.1 > Environment: window >Reporter: Tao Qin >Assignee: Jay Kreps > Labels: features, patch > Fix For: 0.10.1.0 > > Attachments: KAFKA-1194.patch, kafka-1194-v1.patch, > kafka-1194-v2.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > We tested it in windows environment, and set the log.retention.hours to 24 > hours. > # The minimum age of a log file to be eligible for deletion > log.retention.hours=24 > After several days, the kafka broker still cannot delete the old log file. > And we get the following exceptions: > [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task > 'kafka-log-retention' (kafka.utils.KafkaScheduler) > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 1516723 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15377023#comment-15377023 ] Harald Kirsch commented on KAFKA-1194: -- Having a very similar problem I would hope that the fix fixes this one too. The message and stacktrace is slightly different. We are using the logcleaner with compaction and get the below stack trace. This is on Windows. The claim of the error message that another process has the file open is misleading. I verified with procexp and handle search that only the Kafka process has the file open, so it is likely blocking itself on this. Any chance that the patch will fix this one too? {noformat} [2016-07-14 16:09:20,568] ERROR [kafka-log-cleaner-thread-0], Error due to (kafka.log.LogCleaner) kafka.common.KafkaStorageException: Failed to change the log file suffix from .cleaned to .swap for log segment 0 at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:263) at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:265) at kafka.log.Log.replaceSegments(Log.scala:869) at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:395) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:343) at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:342) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.log.Cleaner.clean(LogCleaner.scala:342) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:237) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:215) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) Caused by: java.nio.file.FileSystemException: d:\Search\kafka\__consumer_offsets-40\.log.cleaned -> d:\Search\kafka\__consumer_offsets-40\.log.swap: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670) at kafka.log.FileMessageSet.renameTo(FileMessageSet.scala:364) ... 10 more Suppressed: java.nio.file.FileSystemException: d:\Search\kafka\__consumer_offsets-40\.log.cleaned -> d:\Search\kafka\__consumer_offsets-40\.log.swap: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667) ... 11 more {noformat} > The kafka broker cannot delete the old log files after the configured time > -- > > Key: KAFKA-1194 > URL: https://issues.apache.org/jira/browse/KAFKA-1194 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.8.1 > Environment: window >Reporter: Tao Qin >Assignee: Jay Kreps > Labels: features, patch > Fix For: 0.10.1.0 > > Attachments: KAFKA-1194.patch, kafka-1194-v1.patch, > kafka-1194-v2.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > We tested it in windows environment, and set the log.retention.hours to 24 > hours. > # The minimum age of a log file to be eligible for deletion > log.retention.hours=24 > After several days, the kafka broker still cannot delete the old log file. > And we get the following exceptions: > [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task > 'kafka-log-retention' (kafka.utils.KafkaScheduler) > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 1516723 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249) > at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638) > at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at kafka.log.Log$$anonfun$deleteO
[jira] [Created] (KAFKA-3457) KafkaConsumer.committed(...) hangs forever if port number is wrong
Harald Kirsch created KAFKA-3457: Summary: KafkaConsumer.committed(...) hangs forever if port number is wrong Key: KAFKA-3457 URL: https://issues.apache.org/jira/browse/KAFKA-3457 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.9.0.1 Reporter: Harald Kirsch Create a KafkaConsumer with default settings but with a wrong host:port setting for bootstrap.servers. Have it in some consumer group, do not subscribe or assign partitions. Then call .committed(...) for a topic/partition combination a few times. It will hang on the 2nd or third call forever. In the debug log you will see that it repeats connections all over again. I waited many minutes and it never came back to throw an Exception. The connections problems should at least pop out on the WARNING log level. Likely the connection problems should throw an exception eventually. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete
[ https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182722#comment-15182722 ] Harald Kirsch commented on KAFKA-3339: -- [~singhashish] A bit too much for a pull request. For both methods I would just add something along the lines of: If no {@code TopicPartition} is provided, all topic/partition pairs returned by {@link #assignment} are repositioned. > org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, > seekToEnd incomplete > - > > Key: KAFKA-3339 > URL: https://issues.apache.org/jira/browse/KAFKA-3339 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 0.9.0.1 >Reporter: Harald Kirsch > > The api documentation for seekToBeginning and seekToEnd in > org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark > that all subscribed partitions are seeked if no TopicPartition is provided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete
[ https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harald Kirsch updated KAFKA-3339: - Description: The api documentation for seekToBeginning and seekToEnd in org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided. (was: The api documentation for seekToBeginning, seekToEnd in these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided.) > org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, > seekToEnd incomplete > - > > Key: KAFKA-3339 > URL: https://issues.apache.org/jira/browse/KAFKA-3339 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 0.9.0.1 >Reporter: Harald Kirsch > > The api documentation for seekToBeginning and seekToEnd in > org.apache.kafka.clients.consumer.KafkaConsumer these methods should remark > that all subscribed partitions are seeked if no TopicPartition is provided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete
[ https://issues.apache.org/jira/browse/KAFKA-3339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harald Kirsch updated KAFKA-3339: - Description: The api documentation for seekToBeginning, seekToEnd in these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided. (was: The api documentation for these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided.) > org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, > seekToEnd incomplete > - > > Key: KAFKA-3339 > URL: https://issues.apache.org/jira/browse/KAFKA-3339 > Project: Kafka > Issue Type: Improvement > Components: clients >Affects Versions: 0.9.0.1 >Reporter: Harald Kirsch > > The api documentation for seekToBeginning, seekToEnd in these methods should > remark that all subscribed partitions are seeked if no TopicPartition is > provided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-3339) org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete
Harald Kirsch created KAFKA-3339: Summary: org.apache.kafka.clients.consumer.KafkaConsumer javadoc for seekToBeginning, seekToEnd incomplete Key: KAFKA-3339 URL: https://issues.apache.org/jira/browse/KAFKA-3339 Project: Kafka Issue Type: Improvement Components: clients Affects Versions: 0.9.0.1 Reporter: Harald Kirsch The api documentation for these methods should remark that all subscribed partitions are seeked if no TopicPartition is provided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)