Re: kafka-streams app(s) stopped consuming new events

Thanks Damian ! That's was it, after fixing number compaction threads to be higher than 1, it finally continue to consume stream. On Fri, Jun 30, 2017 at 7:48 PM, Dmitriy Vsekhvalnov wrote: > Yeah, can confirm there is only 1 vCPU. > > Okay, will try that configuration and get back to you guys.

Re: kafka-streams app(s) stopped consuming new events

Yeah, can confirm there is only 1 vCPU. Okay, will try that configuration and get back to you guys. Thanks so far ! On Fri, Jun 30, 2017 at 7:35 PM, Damian Guy wrote: > I'm wondering if it is related to this: > http://docs.confluent.io/current/streams/faq.html# > rocksdb-behavior-in-1-core-env

Re: kafka-streams app(s) stopped consuming new events

I'm wondering if it is related to this: http://docs.confluent.io/current/streams/faq.html#rocksdb-behavior-in-1-core-environments It seems like the same issue. Thanks, Damian On Fri, 30 Jun 2017 at 17:16 Dmitriy Vsekhvalnov wrote: > Yes, StreamThread-1 #93 daemon is still at at org.rocksdb.Rock

Re: kafka-streams app(s) stopped consuming new events

Yes, StreamThread-1 #93 daemon is still at at org.rocksdb.RocksDB.put. No, AWS machine. kafka-streams v0.10.2.1 May be some option for RockDB that can unlock it? Also i can try to run app locally against same env to see if it make difference (though it will be different OS). On Fri, Jun 30, 201

Re: kafka-streams app(s) stopped consuming new events

Yep, if you take another thread dump is it in the same spot? Which version of streams are you running? Are you using docker? Thanks, Damian On Fri, 30 Jun 2017 at 16:22 Dmitriy Vsekhvalnov wrote: > Is this one looking suspicious? > > > https://gist.github.com/dvsekhvalnov/146ba41c8e783169410989

Re: kafka-streams app(s) stopped consuming new events

Is this one looking suspicious? https://gist.github.com/dvsekhvalnov/146ba41c8e78316941098997c9d2f18a#file-thread-dump On Fri, Jun 30, 2017 at 6:10 PM, Damian Guy wrote: > It looks like Task [4_2] is stuck restoring state, though it doesn't look > like there is much state to restore. > It might

Re: kafka-streams app(s) stopped consuming new events

It looks like Task [4_2] is stuck restoring state, though it doesn't look like there is much state to restore. It might be helpful if you take some thread dumps to see where it is blocked. Thanks, Damian On Fri, 30 Jun 2017 at 16:04 Dmitriy Vsekhvalnov wrote: > Set org.apache.kafka.streams to

Re: kafka-streams app(s) stopped consuming new events

Set org.apache.kafka.streams to DEBUG. Here is gist: https://gist.github.com/dvsekhvalnov/b84b72349837f6c6394f1adfe18cdb61#file-debug-logs On Fri, Jun 30, 2017 at 12:37 PM, Dmitriy Vsekhvalnov < dvsekhval...@gmail.com> wrote: > Sure, how to enable debug logs? Just adjust logback logger? > > > O

Re: kafka-streams app(s) stopped consuming new events

Sure, how to enable debug logs? Just adjust logback logger? On Fri, Jun 30, 2017 at 11:24 Eno Thereska wrote: > It’s hard to tell, the logs do not contain much, I agree. It could be a > number of things. > > If it’s happening as you say on restart as well (so it’s reproducible), > any chance yo

Re: kafka-streams app(s) stopped consuming new events

It’s hard to tell, the logs do not contain much, I agree. It could be a number of things. If it’s happening as you say on restart as well (so it’s reproducible), any chance you could start streaming with DEBUG logs on and collect those logs? I’m hoping something shows up there. Thanks, Eno >

Re: kafka-streams app(s) stopped consuming new events

Nothing for stat-change.log for giving time window. Last line logged 4 hours before app stopped. Any ideas so far? Personally i don't see anything relevant in logs. On Wed, Jun 28, 2017 at 6:33 PM, Bill Bejeck wrote: > Sure, couldn't hurt. > > Thanks, > Bill > > On Wed, Jun 28, 2017 at 9:51 AM,

Re: kafka-streams app(s) stopped consuming new events

Sure, couldn't hurt. Thanks, Bill On Wed, Jun 28, 2017 at 9:51 AM, Dmitriy Vsekhvalnov wrote: > Here are logs: > > app: > https://gist.github.com/dvsekhvalnov/f98afc3463f0c63b1722417e3710a8 > e7#file-kafka-streams-log > brokers: > https://gist.github.com/dvsekhvalnov/8e870f7347394e8d004c282880e

Re: kafka-streams app(s) stopped consuming new events

Here are logs: app: https://gist.github.com/dvsekhvalnov/f98afc3463f0c63b1722417e3710a8e7#file-kafka-streams-log brokers: https://gist.github.com/dvsekhvalnov/8e870f7347394e8d004c282880ef385a#file-kafka-broker-1-2-3-log All broker logs are same, so single gist. There are also state-change.log fi

Re: kafka-streams app(s) stopped consuming new events

Hi Dmitry, At the moment I don't have anything specific to look for, just trying to get more context around the issue. As for the logs maybe broker and streams logs for the last 30 minutes up to the time the application stopped processing records. Thanks, Bill On Wed, Jun 28, 2017 at 9:04 AM, D

Re: kafka-streams app(s) stopped consuming new events

Hi Bill, 1. sure, can extract some logs, what exactly to look for? There are 11 hours of logs and most of them looks like: [2017-06-27 03:30:50,553] [] [INFO ] [StreamThread-1] [org.apache.kafka.streams.processor.internals.StreamThread] [stream-thread [StreamThread-1] Committing all tasks because

Re: kafka-streams app(s) stopped consuming new events

Hi Dimitry, I'm happy to help, but I could use more information. Can you share the streams logs and broker logs? Have you confirmed messages are still being delivered to topics (via console consumer)? Thanks, Bill On Wed, Jun 28, 2017 at 8:24 AM, Dmitriy Vsekhvalnov wrote: > Hi all, > > look

kafka-streams app(s) stopped consuming new events

Hi all, looking for some assistance in debugging kafka-streams application. Kafka broker 0.10.2.1 - x3 Node cluster kafka-streams 0.10.2.1 - x2 application nodes x 1 stream thread each. In streams configuration only: - SSL transport - kafka.streams.commitIntervalMs set to 5000 (instead of d