[ https://issues.apache.org/jira/browse/KAFKA-5998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745716#comment-16745716 ]
Dmitry Minkovsky edited comment on KAFKA-5998 at 1/18/19 2:26 AM: ------------------------------------------------------------------ I'm getting this as well: {{[2019-01-18 02:10:07,709] WARN (org.apache.kafka.streams.processor.internals.ProcessorStateManager:295) task [2_1] Failed to write offset checkpoint file to /opt/streams-state/user-service/2_1/.checkpoint: java.io.FileNotFoundException: /opt/streams-state/user-service/2_1/.checkpoint.tmp (No such file or directory) }} {{ [2019-01-18 02:10:07,790] WARN (org.apache.kafka.streams.processor.internals.ProcessorStateManager:295) task [2_1] Failed to write offset checkpoint file to /opt/streams-state/user-service/2_1/.checkpoint: java.io.FileNotFoundException: /opt/streams-state/user-service/2_1/.checkpoint.tmp (No such file or directory)}} When I look in the state dir: {quote}root@user-service-659fcb455-2fkxp:/opt/streams-state/user-service# ls -alh total 48K drwxr-xr-x 12 root root 4.0K Jan 18 01:50 . drwxrwxrwx 3 root root 4.0K Jan 18 01:30 .. drwxr-xr-x 4 root root 4.0K Jan 18 02:10 0_0 drwxr-xr-x 4 root root 4.0K Jan 18 01:30 0_1 drwxr-xr-x 4 root root 4.0K Jan 18 02:10 0_2 drwxr-xr-x 4 root root 4.0K Jan 18 01:30 0_3 drwxr-xr-x 4 root root 4.0K Jan 18 01:30 0_4 drwxr-xr-x 3 root root 4.0K Jan 18 01:30 1_0 drwxr-xr-x 3 root root 4.0K Jan 18 02:10 1_1 drwxr-xr-x 3 root root 4.0K Jan 18 01:30 1_2 drwxr-xr-x 3 root root 4.0K Jan 18 01:30 1_3 drwxr-xr-x 3 root root 4.0K Jan 18 01:30 1_4 root@user-service-659fcb455-2fkxp:/opt/streams-state/user-service# {quote} Not sure what that task 2_1. Will have to add some logging to check. was (Author: dminkovsky): I'm getting this as well: {{[2019-01-18 02:10:07,709] WARN (org.apache.kafka.streams.processor.internals.ProcessorStateManager:295) task [2_1] Failed to write offset checkpoint file to /opt/streams-state/user-service/2_1/.checkpoint: java.io.FileNotFoundException: /opt/streams-state/user-service/2_1/.checkpoint.tmp (No such file or directory) }} {{ [2019-01-18 02:10:07,790] WARN (org.apache.kafka.streams.processor.internals.ProcessorStateManager:295) task [2_1] Failed to write offset checkpoint file to /opt/streams-state/user-service/2_1/.checkpoint: java.io.FileNotFoundException: /opt/streams-state/user-service/2_1/.checkpoint.tmp (No such file or directory)}} When I look in the state dir: root@user-service-659fcb455-2fkxp:/opt/streams-state/user-service# ls -alh total 48K drwxr-xr-x 12 root root 4.0K Jan 18 01:50 . drwxrwxrwx 3 root root 4.0K Jan 18 01:30 .. drwxr-xr-x 4 root root 4.0K Jan 18 02:10 0_0 drwxr-xr-x 4 root root 4.0K Jan 18 01:30 0_1 drwxr-xr-x 4 root root 4.0K Jan 18 02:10 0_2 drwxr-xr-x 4 root root 4.0K Jan 18 01:30 0_3 drwxr-xr-x 4 root root 4.0K Jan 18 01:30 0_4 drwxr-xr-x 3 root root 4.0K Jan 18 01:30 1_0 drwxr-xr-x 3 root root 4.0K Jan 18 02:10 1_1 drwxr-xr-x 3 root root 4.0K Jan 18 01:30 1_2 drwxr-xr-x 3 root root 4.0K Jan 18 01:30 1_3 drwxr-xr-x 3 root root 4.0K Jan 18 01:30 1_4 root@user-service-659fcb455-2fkxp:/opt/streams-state/user-service# > /.checkpoint.tmp Not found exception > ------------------------------------ > > Key: KAFKA-5998 > URL: https://issues.apache.org/jira/browse/KAFKA-5998 > Project: Kafka > Issue Type: Bug > Components: streams > Affects Versions: 0.11.0.0, 0.11.0.1 > Reporter: Yogesh BG > Priority: Major > Attachments: 5998.v1.txt, 5998.v2.txt, Topology.txt, exc.txt, > props.txt, streams.txt > > > I have one kafka broker and one kafka stream running... I am running its > since two days under load of around 2500 msgs per second.. On third day am > getting below exception for some of the partitions, I have 16 partitions only > 0_0 and 0_1 gives this error > {{09:43:25.955 [ks_0_inst-StreamThread-6] WARN > o.a.k.s.p.i.ProcessorStateManager - Failed to write checkpoint file to > /data/kstreams/rtp-kafkastreams/0_1/.checkpoint: > java.io.FileNotFoundException: > /data/kstreams/rtp-kafkastreams/0_1/.checkpoint.tmp (No such file or > directory) > at java.io.FileOutputStream.open(Native Method) ~[na:1.7.0_111] > at java.io.FileOutputStream.<init>(FileOutputStream.java:221) > ~[na:1.7.0_111] > at java.io.FileOutputStream.<init>(FileOutputStream.java:171) > ~[na:1.7.0_111] > at > org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:73) > ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.ProcessorStateManager.checkpoint(ProcessorStateManager.java:324) > ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:267) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:260) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:254) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.AssignedTasks$1.apply(AssignedTasks.java:322) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:415) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.AssignedTasks.commit(AssignedTasks.java:314) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.commitAll(StreamThread.java:700) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:683) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:523) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:480) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:457) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > 09:43:25.974 [ks_0_inst-StreamThread-15] WARN > o.a.k.s.p.i.ProcessorStateManager - Failed to write checkpoint file to > /data/kstreams/rtp-kafkastreams/0_0/.checkpoint: > java.io.FileNotFoundException: > /data/kstreams/rtp-kafkastreams/0_0/.checkpoint.tmp (No such file or > directory) > at java.io.FileOutputStream.open(Native Method) ~[na:1.7.0_111] > at java.io.FileOutputStream.<init>(FileOutputStream.java:221) > ~[na:1.7.0_111] > at java.io.FileOutputStream.<init>(FileOutputStream.java:171) > ~[na:1.7.0_111] > at > org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:73) > ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.ProcessorStateManager.checkpoint(ProcessorStateManager.java:324) > ~[rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:267) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:260) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:254) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.AssignedTasks$1.apply(AssignedTasks.java:322) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:415) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.AssignedTasks.commit(AssignedTasks.java:314) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.commitAll(StreamThread.java:700) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:683) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:523) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:480) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:457) > [rtp-kafkastreams-1.0-SNAPSHOT-jar-with-dependencies.jar:na] > }} -- This message was sent by Atlassian JIRA (v7.6.3#76005)