Hi Bhavesh, I've taken a look at what you've posted, but I'll need the logs *before *the exception occurred. Can you attach the entire log?
In the meantime, if this causing issues, you can effectively disable *state.cleanup.delay.ms <http://state.cleanup.delay.ms> *to a large value (Long.MAX_VALUE ?) Thanks, Bill On Thu, Sep 20, 2018 at 4:56 PM Bhavesh Patel <bhavesh3...@gmail.com> wrote: > Hi Bill, > > Yes and thanks for your response. > > Many thanks, > Bhavesh Patel > > On Thu, Sep 20, 2018, 21:31 Bill Bejeck <b...@confluent.io> wrote: > > > Hi Bhavesh, > > > > I haven't forgotten about you, just got sidetracked with some other > > issues. > > > > Just to be clear you are using Kafka Streams version 2.0 correct? > > > > I'll take a look in the next day or so. > > > > Thanks, > > Bill > > > > On Thu, Sep 20, 2018 at 2:31 PM Bhavesh Patel <bhavesh3...@gmail.com> > > wrote: > > > > > Hi, > > > Is it possible to get h lp with below issue? > > > > > > Many Thanks, > > > Bhavesh Patel > > > > > > On Wed, Sep 19, 2018, 09:13 Bhavesh Patel <bhavesh3...@gmail.com> > wrote: > > > > > > > Hi Bill, > > > > Apologies, I was trying to attached image. > > > > Please find below the error log, > > > > > > > > 2018-09-18 09:26:09.112 INFO 1 --- [5-CleanupThread] > > > > o.a.k.s.p.internals.StateDirectory : stream-thread > > > > [ApplicationName-1ae22d38-32d3-451a-b039-372c79b2e6a5-CleanupThread] > > > > Deleting obsolete state directory 2_1 for task 2_1 as 601112ms has > > > elapsed > > > > (cleanup delay is 600000ms). > > > > > > > > 2018-09-18 09:26:09.116 ERROR 1 --- [5-CleanupThread] > > > > o.a.k.s.p.internals.StateDirectory : stream-thread > > > > [ApplicationName-1ae22d38-32d3-451a-b039-372c79b2e6a5-CleanupThread] > > > Failed > > > > to delete the state directory. > > > > > > > > > > > > > > > > java.nio.file.DirectoryNotEmptyException: > > > > /tmp/ApplicationName/ApplicationName/2_1 > > > > > > > > at > > > > > > > > > > sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) > > > > ~[na:1.8.0_161] > > > > > > > > at > > > > > > > > > > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) > > > > ~[na:1.8.0_161] > > > > > > > > at java.nio.file.Files.delete(Files.java:1126) > > > > ~[na:1.8.0_161] > > > > > > > > at > > > > > > org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:740) > > > > ~[kafka-clients-2.0.0.jar!/:na] > > > > > > > > at > > > > > > org.apache.kafka.common.utils.Utils$2.postVisitDirectory(Utils.java:723) > > > > ~[kafka-clients-2.0.0.jar!/:na] > > > > > > > > at java.nio.file.Files.walkFileTree(Files.java:2688) > > > > ~[na:1.8.0_161] > > > > > > > > at java.nio.file.Files.walkFileTree(Files.java:2742) > > > > ~[na:1.8.0_161] > > > > > > > > at > > > > org.apache.kafka.common.utils.Utils.delete(Utils.java:723) > > > > ~[kafka-clients-2.0.0.jar!/:na] > > > > > > > > at > > > > > > > > > > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:287) > > > > [kafka-streams-2.0.0.jar!/:na] > > > > > > > > at > > > > > > > > > > org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:250) > > > > [kafka-streams-2.0.0.jar!/:na] > > > > > > > > at > > > > org.apache.kafka.streams.KafkaStreams$2.run(KafkaStreams.java:800) > > > > [kafka-streams-2.0.0.jar!/:na] > > > > > > > > at > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > > [na:1.8.0_161] > > > > > > > > at > > > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > > > [na:1.8.0_161] > > > > > > > > at > > > > > > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > > > > [na:1.8.0_161] > > > > > > > > at > > > > > > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > > > > [na:1.8.0_161] > > > > > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > > > [na:1.8.0_161] > > > > > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > > > [na:1.8.0_161] > > > > > > > > at java.lang.Thread.run(Thread.java:748) > [na:1.8.0_161] > > > > > > > > Many Thanks, > > > > Bhavesh Patel > > > > > > > > > > > > On Wed, Sep 19, 2018, 02:02 Bill Bejeck <b...@confluent.io> wrote: > > > > > > > >> Hi Bhavesh, > > > >> > > > >> Sorry, but I'm still not seeing any log file attachment, maybe it's > > > being > > > >> filtered out. > > > >> > > > >> Can you copy the relevant section in the body of the email? > > > >> > > > >> Thanks, > > > >> Bill > > > >> > > > >> On Tue, Sep 18, 2018 at 3:21 PM Bhavesh Patel < > bhavesh3...@gmail.com> > > > >> wrote: > > > >> > > > >> > Hi Bill, > > > >> > > > > >> > Apologies, Please find the error stack attached. > > > >> > > > > >> > We are using kafka 2.0. > > > >> > > > > >> > Thanks, > > > >> > Bhavesh Patel > > > >> > > > > >> > On Tue, Sep 18, 2018 at 8:04 PM Bill Bejeck <b...@confluent.io> > > > wrote: > > > >> > > > > >> >> Hi Bhavesh, > > > >> >> > > > >> >> I don't see the log file attachment, can you resend it? > > > >> >> > > > >> >> Also, what version of Kafka Streams are you running? > > > >> >> > > > >> >> Thanks, > > > >> >> Bill > > > >> >> > > > >> >> On Tue, Sep 18, 2018 at 12:31 PM Bhavesh Patel < > > > bhavesh3...@gmail.com> > > > >> >> wrote: > > > >> >> > > > >> >> > Hi, > > > >> >> > > > > >> >> > We have a stream application where we are facing a little > strange > > > >> >> problem > > > >> >> > with deleting state directory. > > > >> >> > > > > >> >> > Our application is deployed into in-house container platform > and > > > >> >> storage is > > > >> >> > backed by nfs. > > > >> >> > > > > >> >> > When app is running for a while and when it hits the cleanup > > delay > > > of > > > >> >> > 600000ms as defined dafault in Kafka settings we are getting > > > >> exception > > > >> >> of > > > >> >> > DIRECTORYNOTEMPTYEXCEPTION. > > > >> >> > > > > >> >> > I am attaching the log below, thoughts on this issue will be > much > > > >> >> > appreciated. > > > >> >> > > > > >> >> > Many Thanks, > > > >> >> > Bhavesh Patel > > > >> >> > > > > >> >> > > > >> > > > > >> > > > > > > > > > >
