[jira] [Commented] (KAFKA-6322) Error deleting log for topic, all log dirs failed.
[
https://issues.apache.org/jira/browse/KAFKA-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16281270#comment-16281270
]
Ted Yu commented on KAFKA-6322:
---
Seems to be same problem as KAFKA-6075
> Error deleting log for topic, all log dirs failed.
> --
>
> Key: KAFKA-6322
> URL: https://issues.apache.org/jira/browse/KAFKA-6322
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: RancherOS with NFS mounted for log directory
>Reporter: dongyan li
>
> Hello,
> I encountered a error when I try to delete a topic with kafka version 1.0.0,
> the error is not present on version 0.10.2.1 which is the version I upgraded
> from.
> I suspect that some other thread is still using that file while the Kafka is
> trying to delete that.
> {code:java}
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,846] ERROR Exception while deleting
> Log(/opt/kafka/logs/topicname-0.112ff11872e4411ca7470ba7e3026ab0-delete) in
> dir /opt/kafka/logs. (kafka.log.LogManager)
> 12/6/2017 3:37:32 PMorg.apache.kafka.common.errors.KafkaStorageException:
> Error while deleting log for topicname-0 in dir /opt/kafka/logs
> 12/6/2017 3:37:32 PMCaused by: java.nio.file.FileSystemException:
> /opt/kafka/logs/topicname-0.112ff11872e4411ca7470ba7e3026ab0-delete/.nfs00e609f200ce:
> Device or resource busy
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.delete(Files.java:1126)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:630)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:619)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.walkFileTree(Files.java:2670)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.walkFileTree(Files.java:2742)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils.delete(Utils.java:619)
> 12/6/2017 3:37:32 PM at kafka.log.Log.$anonfun$delete$2(Log.scala:1432)
> 12/6/2017 3:37:32 PM at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
> 12/6/2017 3:37:32 PM at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
> 12/6/2017 3:37:32 PM at kafka.log.Log.delete(Log.scala:1427)
> 12/6/2017 3:37:32 PM at kafka.log.LogManager.deleteLogs(LogManager.scala:626)
> 12/6/2017 3:37:32 PM at
> kafka.log.LogManager.$anonfun$startup$7(LogManager.scala:362)
> 12/6/2017 3:37:32 PM at
> kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:110)
> 12/6/2017 3:37:32 PM at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 12/6/2017 3:37:32 PM at java.lang.Thread.run(Thread.java:748)
> {code}
> Then
> {code:java}
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,882] INFO Stopping serving logs in
> dir /opt/kafka/logs (kafka.log.LogManager)
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,883] FATAL Shutdown broker because
> all log dirs in /opt/kafka/logs have failed (kafka.log.LogManager)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6322) Error deleting log for topic, all log dirs failed.
[
https://issues.apache.org/jira/browse/KAFKA-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282064#comment-16282064
]
dongyan li commented on KAFKA-6322:
---
Yes, I think they attribute to the same problem. However, instead of changing
the methods for deleting, we may want to find whom is actually holding the file
handler when the system is try to delete the folder. I know NFS filesystem is
not a "regular" filesystem and that is the root problem.
> Error deleting log for topic, all log dirs failed.
> --
>
> Key: KAFKA-6322
> URL: https://issues.apache.org/jira/browse/KAFKA-6322
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: RancherOS with NFS mounted for log directory
>Reporter: dongyan li
>
> Hello,
> I encountered a error when I try to delete a topic with kafka version 1.0.0,
> the error is not present on version 0.10.2.1 which is the version I upgraded
> from.
> I suspect that some other thread is still using that file while the Kafka is
> trying to delete that.
> {code:java}
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,846] ERROR Exception while deleting
> Log(/opt/kafka/logs/topicname-0.112ff11872e4411ca7470ba7e3026ab0-delete) in
> dir /opt/kafka/logs. (kafka.log.LogManager)
> 12/6/2017 3:37:32 PMorg.apache.kafka.common.errors.KafkaStorageException:
> Error while deleting log for topicname-0 in dir /opt/kafka/logs
> 12/6/2017 3:37:32 PMCaused by: java.nio.file.FileSystemException:
> /opt/kafka/logs/topicname-0.112ff11872e4411ca7470ba7e3026ab0-delete/.nfs00e609f200ce:
> Device or resource busy
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.delete(Files.java:1126)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:630)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:619)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.walkFileTree(Files.java:2670)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.walkFileTree(Files.java:2742)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils.delete(Utils.java:619)
> 12/6/2017 3:37:32 PM at kafka.log.Log.$anonfun$delete$2(Log.scala:1432)
> 12/6/2017 3:37:32 PM at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
> 12/6/2017 3:37:32 PM at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
> 12/6/2017 3:37:32 PM at kafka.log.Log.delete(Log.scala:1427)
> 12/6/2017 3:37:32 PM at kafka.log.LogManager.deleteLogs(LogManager.scala:626)
> 12/6/2017 3:37:32 PM at
> kafka.log.LogManager.$anonfun$startup$7(LogManager.scala:362)
> 12/6/2017 3:37:32 PM at
> kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:110)
> 12/6/2017 3:37:32 PM at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 12/6/2017 3:37:32 PM at java.lang.Thread.run(Thread.java:748)
> {code}
> Then
> {code:java}
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,882] INFO Stopping serving logs in
> dir /opt/kafka/logs (kafka.log.LogManager)
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,883] FATAL Shutdown broker because
> all log dirs in /opt/kafka/logs have failed (kafka.log.LogManager)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6322) Error deleting log for topic, all log dirs failed.
[
https://issues.apache.org/jira/browse/KAFKA-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282105#comment-16282105
]
Ismael Juma commented on KAFKA-6322:
Probably a duplicate of KAFKA-6324, that is nearly ready to be merged.
> Error deleting log for topic, all log dirs failed.
> --
>
> Key: KAFKA-6322
> URL: https://issues.apache.org/jira/browse/KAFKA-6322
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: RancherOS with NFS mounted for log directory
>Reporter: dongyan li
>
> Hello,
> I encountered a error when I try to delete a topic with kafka version 1.0.0,
> the error is not present on version 0.10.2.1 which is the version I upgraded
> from.
> I suspect that some other thread is still using that file while the Kafka is
> trying to delete that.
> {code:java}
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,846] ERROR Exception while deleting
> Log(/opt/kafka/logs/topicname-0.112ff11872e4411ca7470ba7e3026ab0-delete) in
> dir /opt/kafka/logs. (kafka.log.LogManager)
> 12/6/2017 3:37:32 PMorg.apache.kafka.common.errors.KafkaStorageException:
> Error while deleting log for topicname-0 in dir /opt/kafka/logs
> 12/6/2017 3:37:32 PMCaused by: java.nio.file.FileSystemException:
> /opt/kafka/logs/topicname-0.112ff11872e4411ca7470ba7e3026ab0-delete/.nfs00e609f200ce:
> Device or resource busy
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.delete(Files.java:1126)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:630)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:619)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.walkFileTree(Files.java:2670)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.walkFileTree(Files.java:2742)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils.delete(Utils.java:619)
> 12/6/2017 3:37:32 PM at kafka.log.Log.$anonfun$delete$2(Log.scala:1432)
> 12/6/2017 3:37:32 PM at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
> 12/6/2017 3:37:32 PM at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
> 12/6/2017 3:37:32 PM at kafka.log.Log.delete(Log.scala:1427)
> 12/6/2017 3:37:32 PM at kafka.log.LogManager.deleteLogs(LogManager.scala:626)
> 12/6/2017 3:37:32 PM at
> kafka.log.LogManager.$anonfun$startup$7(LogManager.scala:362)
> 12/6/2017 3:37:32 PM at
> kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:110)
> 12/6/2017 3:37:32 PM at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 12/6/2017 3:37:32 PM at java.lang.Thread.run(Thread.java:748)
> {code}
> Then
> {code:java}
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,882] INFO Stopping serving logs in
> dir /opt/kafka/logs (kafka.log.LogManager)
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,883] FATAL Shutdown broker because
> all log dirs in /opt/kafka/logs have failed (kafka.log.LogManager)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6322) Error deleting log for topic, all log dirs failed.
[
https://issues.apache.org/jira/browse/KAFKA-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282120#comment-16282120
]
Ted Yu commented on KAFKA-6322:
---
FileSystemException was encountered in Dongyan's case (which is not among the
exceptions thrown by
https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#deleteIfExists(java.nio.file.Path)).
> Error deleting log for topic, all log dirs failed.
> --
>
> Key: KAFKA-6322
> URL: https://issues.apache.org/jira/browse/KAFKA-6322
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: RancherOS with NFS mounted for log directory
>Reporter: dongyan li
>
> Hello,
> I encountered a error when I try to delete a topic with kafka version 1.0.0,
> the error is not present on version 0.10.2.1 which is the version I upgraded
> from.
> I suspect that some other thread is still using that file while the Kafka is
> trying to delete that.
> {code:java}
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,846] ERROR Exception while deleting
> Log(/opt/kafka/logs/topicname-0.112ff11872e4411ca7470ba7e3026ab0-delete) in
> dir /opt/kafka/logs. (kafka.log.LogManager)
> 12/6/2017 3:37:32 PMorg.apache.kafka.common.errors.KafkaStorageException:
> Error while deleting log for topicname-0 in dir /opt/kafka/logs
> 12/6/2017 3:37:32 PMCaused by: java.nio.file.FileSystemException:
> /opt/kafka/logs/topicname-0.112ff11872e4411ca7470ba7e3026ab0-delete/.nfs00e609f200ce:
> Device or resource busy
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.delete(Files.java:1126)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:630)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:619)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.walkFileTree(Files.java:2670)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.walkFileTree(Files.java:2742)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils.delete(Utils.java:619)
> 12/6/2017 3:37:32 PM at kafka.log.Log.$anonfun$delete$2(Log.scala:1432)
> 12/6/2017 3:37:32 PM at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
> 12/6/2017 3:37:32 PM at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
> 12/6/2017 3:37:32 PM at kafka.log.Log.delete(Log.scala:1427)
> 12/6/2017 3:37:32 PM at kafka.log.LogManager.deleteLogs(LogManager.scala:626)
> 12/6/2017 3:37:32 PM at
> kafka.log.LogManager.$anonfun$startup$7(LogManager.scala:362)
> 12/6/2017 3:37:32 PM at
> kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:110)
> 12/6/2017 3:37:32 PM at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 12/6/2017 3:37:32 PM at java.lang.Thread.run(Thread.java:748)
> {code}
> Then
> {code:java}
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,882] INFO Stopping serving logs in
> dir /opt/kafka/logs (kafka.log.LogManager)
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,883] FATAL Shutdown broker because
> all log dirs in /opt/kafka/logs have failed (kafka.log.LogManager)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6322) Error deleting log for topic, all log dirs failed.
[
https://issues.apache.org/jira/browse/KAFKA-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282156#comment-16282156
]
Ismael Juma commented on KAFKA-6322:
It was thrown by Files.delete as shown by the stacktrace. Not sure how the
javadoc helps here.
> Error deleting log for topic, all log dirs failed.
> --
>
> Key: KAFKA-6322
> URL: https://issues.apache.org/jira/browse/KAFKA-6322
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: RancherOS with NFS mounted for log directory
>Reporter: dongyan li
>
> Hello,
> I encountered a error when I try to delete a topic with kafka version 1.0.0,
> the error is not present on version 0.10.2.1 which is the version I upgraded
> from.
> I suspect that some other thread is still using that file while the Kafka is
> trying to delete that.
> {code:java}
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,846] ERROR Exception while deleting
> Log(/opt/kafka/logs/topicname-0.112ff11872e4411ca7470ba7e3026ab0-delete) in
> dir /opt/kafka/logs. (kafka.log.LogManager)
> 12/6/2017 3:37:32 PMorg.apache.kafka.common.errors.KafkaStorageException:
> Error while deleting log for topicname-0 in dir /opt/kafka/logs
> 12/6/2017 3:37:32 PMCaused by: java.nio.file.FileSystemException:
> /opt/kafka/logs/topicname-0.112ff11872e4411ca7470ba7e3026ab0-delete/.nfs00e609f200ce:
> Device or resource busy
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.delete(Files.java:1126)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:630)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:619)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.walkFileTree(Files.java:2670)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.walkFileTree(Files.java:2742)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils.delete(Utils.java:619)
> 12/6/2017 3:37:32 PM at kafka.log.Log.$anonfun$delete$2(Log.scala:1432)
> 12/6/2017 3:37:32 PM at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
> 12/6/2017 3:37:32 PM at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
> 12/6/2017 3:37:32 PM at kafka.log.Log.delete(Log.scala:1427)
> 12/6/2017 3:37:32 PM at kafka.log.LogManager.deleteLogs(LogManager.scala:626)
> 12/6/2017 3:37:32 PM at
> kafka.log.LogManager.$anonfun$startup$7(LogManager.scala:362)
> 12/6/2017 3:37:32 PM at
> kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:110)
> 12/6/2017 3:37:32 PM at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 12/6/2017 3:37:32 PM at java.lang.Thread.run(Thread.java:748)
> {code}
> Then
> {code:java}
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,882] INFO Stopping serving logs in
> dir /opt/kafka/logs (kafka.log.LogManager)
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,883] FATAL Shutdown broker because
> all log dirs in /opt/kafka/logs have failed (kafka.log.LogManager)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6322) Error deleting log for topic, all log dirs failed.
[
https://issues.apache.org/jira/browse/KAFKA-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282186#comment-16282186
]
Ted Yu commented on KAFKA-6322:
---
Edited previous comment.
Since Dongyan's case was not for file not existing at time of deletion, we
should look out for possible cause of file being used.
> Error deleting log for topic, all log dirs failed.
> --
>
> Key: KAFKA-6322
> URL: https://issues.apache.org/jira/browse/KAFKA-6322
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: RancherOS with NFS mounted for log directory
>Reporter: dongyan li
>
> Hello,
> I encountered a error when I try to delete a topic with kafka version 1.0.0,
> the error is not present on version 0.10.2.1 which is the version I upgraded
> from.
> I suspect that some other thread is still using that file while the Kafka is
> trying to delete that.
> {code:java}
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,846] ERROR Exception while deleting
> Log(/opt/kafka/logs/topicname-0.112ff11872e4411ca7470ba7e3026ab0-delete) in
> dir /opt/kafka/logs. (kafka.log.LogManager)
> 12/6/2017 3:37:32 PMorg.apache.kafka.common.errors.KafkaStorageException:
> Error while deleting log for topicname-0 in dir /opt/kafka/logs
> 12/6/2017 3:37:32 PMCaused by: java.nio.file.FileSystemException:
> /opt/kafka/logs/topicname-0.112ff11872e4411ca7470ba7e3026ab0-delete/.nfs00e609f200ce:
> Device or resource busy
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.delete(Files.java:1126)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:630)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:619)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.walkFileTree(Files.java:2670)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.walkFileTree(Files.java:2742)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils.delete(Utils.java:619)
> 12/6/2017 3:37:32 PM at kafka.log.Log.$anonfun$delete$2(Log.scala:1432)
> 12/6/2017 3:37:32 PM at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
> 12/6/2017 3:37:32 PM at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
> 12/6/2017 3:37:32 PM at kafka.log.Log.delete(Log.scala:1427)
> 12/6/2017 3:37:32 PM at kafka.log.LogManager.deleteLogs(LogManager.scala:626)
> 12/6/2017 3:37:32 PM at
> kafka.log.LogManager.$anonfun$startup$7(LogManager.scala:362)
> 12/6/2017 3:37:32 PM at
> kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:110)
> 12/6/2017 3:37:32 PM at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 12/6/2017 3:37:32 PM at java.lang.Thread.run(Thread.java:748)
> {code}
> Then
> {code:java}
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,882] INFO Stopping serving logs in
> dir /opt/kafka/logs (kafka.log.LogManager)
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,883] FATAL Shutdown broker because
> all log dirs in /opt/kafka/logs have failed (kafka.log.LogManager)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6322) Error deleting log for topic, all log dirs failed.
[
https://issues.apache.org/jira/browse/KAFKA-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282200#comment-16282200
]
Ismael Juma commented on KAFKA-6322:
The issue was always related to holding a file handle while trying to delete
the file. KAFKA-6324 fixes one instance of that.
> Error deleting log for topic, all log dirs failed.
> --
>
> Key: KAFKA-6322
> URL: https://issues.apache.org/jira/browse/KAFKA-6322
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: RancherOS with NFS mounted for log directory
>Reporter: dongyan li
>
> Hello,
> I encountered a error when I try to delete a topic with kafka version 1.0.0,
> the error is not present on version 0.10.2.1 which is the version I upgraded
> from.
> I suspect that some other thread is still using that file while the Kafka is
> trying to delete that.
> {code:java}
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,846] ERROR Exception while deleting
> Log(/opt/kafka/logs/topicname-0.112ff11872e4411ca7470ba7e3026ab0-delete) in
> dir /opt/kafka/logs. (kafka.log.LogManager)
> 12/6/2017 3:37:32 PMorg.apache.kafka.common.errors.KafkaStorageException:
> Error while deleting log for topicname-0 in dir /opt/kafka/logs
> 12/6/2017 3:37:32 PMCaused by: java.nio.file.FileSystemException:
> /opt/kafka/logs/topicname-0.112ff11872e4411ca7470ba7e3026ab0-delete/.nfs00e609f200ce:
> Device or resource busy
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> 12/6/2017 3:37:32 PM at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.delete(Files.java:1126)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:630)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils$1.visitFile(Utils.java:619)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.walkFileTree(Files.java:2670)
> 12/6/2017 3:37:32 PM at java.nio.file.Files.walkFileTree(Files.java:2742)
> 12/6/2017 3:37:32 PM at
> org.apache.kafka.common.utils.Utils.delete(Utils.java:619)
> 12/6/2017 3:37:32 PM at kafka.log.Log.$anonfun$delete$2(Log.scala:1432)
> 12/6/2017 3:37:32 PM at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
> 12/6/2017 3:37:32 PM at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
> 12/6/2017 3:37:32 PM at kafka.log.Log.delete(Log.scala:1427)
> 12/6/2017 3:37:32 PM at kafka.log.LogManager.deleteLogs(LogManager.scala:626)
> 12/6/2017 3:37:32 PM at
> kafka.log.LogManager.$anonfun$startup$7(LogManager.scala:362)
> 12/6/2017 3:37:32 PM at
> kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:110)
> 12/6/2017 3:37:32 PM at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 12/6/2017 3:37:32 PM at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 12/6/2017 3:37:32 PM at java.lang.Thread.run(Thread.java:748)
> {code}
> Then
> {code:java}
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,882] INFO Stopping serving logs in
> dir /opt/kafka/logs (kafka.log.LogManager)
> 12/6/2017 3:37:32 PM[2017-12-06 21:37:32,883] FATAL Shutdown broker because
> all log dirs in /opt/kafka/logs have failed (kafka.log.LogManager)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
