[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16517050#comment-16517050 ] Grégory R. commented on KAFKA-4392: --- [~mjsax] Not tried but will do asap. However folder structure is created fine by the stream so it does not seem to be a problem of right access. [~guozhang] Yes my mistake. I try to set a different state directory in the config and will declare an other issue if necessary. > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang >Priority: Major > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_12 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_18 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_19 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_24 > drwxr-xr-x 3 root root 64 Nov 2 15:43
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16507578#comment-16507578 ] Matthias J. Sax commented on KAFKA-4392: This Jira describes as `NoSuchFileException` while your exception says `AccessDeniedException` – not sure if it is related to KAFKA-6655 though, because there the issues is a `DirectoryNotEmptyException`. [~boushi] Have you tried to set a different state directory in the config? Maybe you are just not allowed to write to the directory. Also note, that the default state directory config is not recommended for production. > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang >Priority: Major > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_12 > drwxr-xr-x 3 root root
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503841#comment-16503841 ] Guozhang Wang commented on KAFKA-4392: -- [~boushi] Is this more related to https://issues.apache.org/jira/browse/KAFKA-6655? If yes could you paste your comments on that JIRA? > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang >Priority: Major > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_12 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_18 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_19 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_24 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_25 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_26 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_4 > drwxr-xr-x 3 root root 64
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502979#comment-16502979 ] Grégory R. commented on KAFKA-4392: --- Same issue here on windows and kafka 1.1.0. 2018-06-06 10:06:43 ERROR StateDirectory:251 - stream-thread [bridgeWorkerData-e125bca4-5e2f-4b08-b0bc-b9f6eb5135ba-CleanupThread] Failed to lock the state directory due to an unexpected exception java.nio.file.AccessDeniedException: \tmp\kafka-streams\myapp1\0_0\.lock at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) 2018-06-06 10:07:00 ERROR StateDirectory:251 - stream-thread [bridgeWorkerOutput-abfe4c31-cfa3-43a4-8e09-8590df4d2369-CleanupThread] Failed to lock the state directory due to an unexpected exception java.nio.file.AccessDeniedException: \tmp\kafka-streams\myapp2\0_0\.lock at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115) at java.nio.channels.FileChannel.open(FileChannel.java:287) at java.nio.channels.FileChannel.open(FileChannel.java:335) at org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:294) at org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:148) at org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:240) at org.apache.kafka.streams.KafkaStreams$3.run(KafkaStreams.java:761) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-06-06 10:07:03 ERROR StateDirectory:251 - stream-thread [bridgeWorkerEvent-33463024-bcf5-4989-93fb-6c94034ea4ae-CleanupThread] Failed t o lock the state directory due to an unexpected exception java.nio.file.AccessDeniedException: \tmp\kafka-streams\myapp3\0_0\.lock at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115) at java.nio.channels.FileChannel.open(FileChannel.java:287) at java.nio.channels.FileChannel.open(FileChannel.java:335) at org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:294) at org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:148) at org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:240) at org.apache.kafka.streams.KafkaStreams$3.run(KafkaStreams.java:761) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3 streams started, 3 different paths and the issue is raised esxactly at the same time, the first time the exception is raised. Next, each 10 minutes, this exception is raised. > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara E
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400735#comment-16400735 ] Matthias J. Sax commented on KAFKA-4392: https://issues.apache.org/jira/browse/KAFKA-6647 Seems to be a related issues. I also did a PR that includes some improvements. Atm, we are not able to reproduce the issues though. It seems to be related to different behavior of Linux and Windows (unfortunately, all our testing is done using Linux machines atm) so it's hard to detect if Linux/Windows behaves differently. In order to keep concerns separated, I'll reopen KAFKA-6655 and point out that this tickets relates to Windows. > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang >Priority: Major > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400336#comment-16400336 ] Srini commented on KAFKA-4392: -- [~mjsax] Yes, I did set the path and can view the contents. In fact, I created /tmp as well. The path is not the issue since java respects unix style paths on Windows. My guess is org.apache.kafka.common.utils.Utils.delete is unable to delete the ".lock" or ".checkpoint" file(s). @[~habdank] Thanks, yes I am aware. Unfortunately, that will not work since we are implementing a stateful store, 24/7 four 9s service and will need the app to recover gracefully in case of a crash. > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang >Priority: Major > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3 root root 64 Nov 2
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400055#comment-16400055 ] Seweryn Habdank-Wojewodzki commented on KAFKA-4392: --- I see it on my Windows workstation as well. The workaround is before kafka start to clear all files, which are stored in this folder like _/data/1/kafka-streams/myapp-streams/_ . This is only workaround, because if the service is really working like 24/7 then it will loos complete state and messages might be lost :-(. > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang >Priority: Major > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_12 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_18 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_19 > drwxr-xr-x
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399461#comment-16399461 ] Matthias J. Sax commented on KAFKA-4392: I just saw that you create KAFKA-6655 that duplicate this JIRA. Close the other one. One thing I am wondering about is the error message. If you run on windows, and the error message is java.nio.file.DirectoryNotEmptyException: /data/myapp/tmp/kafka-state/streamer/18_4 (or similar), the path looks like a Linux path. I am wondering if it's related to this... > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang >Priority: Major > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_12 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_18 > drwxr-xr-x 3 root root 64 Nov 2 15:
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399430#comment-16399430 ] Srini commented on KAFKA-4392: -- @matthias I am running on Windows > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang >Priority: Major > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_12 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_18 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_19 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_24 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_25 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_26 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_4 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_9 > drwxr-xr-x 3 root root 58 Nov 2 15:43 5_1 > drwxr-xr-x 3 root root 58 Nov 2 15:43 5_10 > drw
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399364#comment-16399364 ] Matthias J. Sax commented on KAFKA-4392: [~hana] Are you running on Linux or Windows? > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang >Priority: Major > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_12 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_18 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_19 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_24 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_25 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_26 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_4 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_9 > drwxr-xr-x 3 root root 58 Nov 2 15:43 5_1 > drwxr-xr-x 3 root roo
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398662#comment-16398662 ] Srini commented on KAFKA-4392: -- I am using 1.0.1. Getting this error consistently. Failed to lock the state directory due to an unexpected exception > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang >Priority: Major > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_12 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_18 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_19 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_24 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_25 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_26 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_4 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_9 > drwxr-xr-x 3 roo
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166378#comment-16166378 ] Seweryn Habdank-Wojewodzki commented on KAFKA-4392: --- Sure, but next week :-). > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_12 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_18 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_19 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_24 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_25 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_26 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_4 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_9 > drwxr-xr-x 3 root root 58 Nov 2 15:43 5_1 > drwxr-xr-x 3 root root 58 Nov 2 15:43 5_10 >
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165226#comment-16165226 ] Guozhang Wang commented on KAFKA-4392: -- Could you try out 0.11.0.1, which is just released today? > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_12 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_18 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_19 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_24 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_25 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_26 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_4 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_9 > drwxr-xr-x 3 root root 58 Nov 2 15:43 5_1 > drwxr-xr-x 3 root root 58 Nov 2 15:43
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164190#comment-16164190 ] Seweryn Habdank-Wojewodzki commented on KAFKA-4392: --- I am using 0.11.0.0. > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_12 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_18 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_19 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_24 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_25 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_26 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_4 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_9 > drwxr-xr-x 3 root root 58 Nov 2 15:43 5_1 > drwxr-xr-x 3 root root 58 Nov 2 15:43 5_10 > drwx
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16163966#comment-16163966 ] Guozhang Wang commented on KAFKA-4392: -- Which version are you using? I'm pretty confident that this issue has been resolved in 0.11.0 / 1.0.0. > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_10 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_15 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_17 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_18 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 1_5 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_1 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_10 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_12 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_20 > drwxr-xr-x 3 root root 60 Nov 2 15:43 2_24 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_10 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_11 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_19 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_20 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_25 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_26 > drwxr-xr-x 3 root root 61 Nov 2 15:43 3_3 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_11 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_12 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_18 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_19 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_24 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_25 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_26 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_4 > drwxr-xr-x 3 root root 64 Nov 2 15:43 4_9 > drwxr-xr-x 3 root root 58 Nov 2 15:43 5_
[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception
[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161405#comment-16161405 ] Seweryn Habdank-Wojewodzki commented on KAFKA-4392: --- I have the same problem. num.stream.threads = 4 in number partitions = 10. At some moment KStream based app ends with: ERROR StateDirectory:227 - stream-thread [...] Failed to lock the state directory due to an unexpected exception java.nio.file.DirectoryNotEmptyException: /data/myapp/tmp/kafka-state/streamer/18_4 at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) ~[?:1.8.0_102] at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) ~[?:1.8.0_102] at java.nio.file.Files.delete(Files.java:1126) ~[?:1.8.0_102] at org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:597) ~[myapp-streamer.jar:?] at org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:580) ~[myapp-streamer.jar:?] at java.nio.file.Files.walkFileTree(Files.java:2688) ~[?:1.8.0_102] at java.nio.file.Files.walkFileTree(Files.java:2742) ~[?:1.8.0_102] at org.apache.kafka.common.utils.Utils.delete(Utils.java:580) ~[myapp-streamer.jar:?] at org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:221) [myapp-streamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:878) [myapp-streamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:569) [myapp-streamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:527) [myapp-streamer.jar:?] > Failed to lock the state directory due to an unexpected exception > - > > Key: KAFKA-4392 > URL: https://issues.apache.org/jira/browse/KAFKA-4392 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.10.1.0 >Reporter: Ara Ebrahimi >Assignee: Guozhang Wang > Fix For: 0.10.2.0 > > > This happened on streaming startup, on a clean installation, no existing > folder. Here I was starting 4 instances of our streaming app on 4 machines > and one threw this exception. Seems to me there’s a race condition somewhere > when instances discover others, or something like that. > 2016-11-02 15:43:47 INFO StreamRunner:59 - Started http server successfully. > 2016-11-02 15:44:50 ERROR StateDirectory:147 - Failed to lock the state > directory due to an unexpected exception > java.nio.file.NoSuchFileException: > /data/1/kafka-streams/myapp-streams/7_21/.lock > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.kafka.streams.processor.internals.StateDirectory.getOrCreateFileChannel(StateDirectory.java:176) > at > org.apache.kafka.streams.processor.internals.StateDirectory.lock(StateDirectory.java:90) > at > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:140) > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeClean(StreamThread.java:552) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:459) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242) > ^C > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/7_21/ > ls: cannot access /data/1/kafka-streams/myapp-streams/7_21/: No such file or > directory > [arae@a4 ~]$ ls -al /data/1/kafka-streams/myapp-streams/ > total 4 > drwxr-xr-x 74 root root 4096 Nov 2 15:44 . > drwxr-xr-x 3 root root 27 Nov 2 15:43 .. > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_1 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_13 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_14 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_16 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_2 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_22 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_28 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_3 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_31 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_5 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_7 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_8 > drwxr-xr-x 3 root root 32 Nov 2 15:43 0_9
