Re: Group Coordinator stuck on PrepareRebalance state.

2019-07-15 Thread Aravind Dongara
Hi Boyang, Our initial assessment was not accurate, but after scanning through all the stream threads on all machines we found that while most of the threads are participating in the joinRequest call some are stuck at TransactionalRequestResult.await() while doing StreamTask.commit(). We think

Re: Group Coordinator stuck on PrepareRebalance state.

2019-07-11 Thread Thameem Ansari
In addition to the session timeout try increasing the request timeout as well. We had similar issue and resolved it by increasing the timeouts. As per my understanding, If you have complex topology then it will take some time for kafka brokers to create the tasks and assign them to consumers. In

Re: Group Coordinator stuck on PrepareRebalance state.

2019-07-11 Thread Aravind Dongara
Hi Boyang, Thanks for the quick response. We are on version 2.2.0. We are using the following properties on KStreams/consumer: session.timeout.ms=15000 heartbeat.interval.ms=3000 I was wondering if a member might leak if it satisfies “shouldKeepAlive” condition in "onExpireHeartbeat" and the co

Re: Group Coordinator stuck on PrepareRebalance state.

2019-07-10 Thread Boyang Chen
Hey Aravind, If your client/broker just upgraded to 2.3, Jason has filed a blocker for 2.3: https://issues.apache.org/jira/browse/KAFKA-8653 and a fix is on its way: https://github.com/apache/kafka/pull/7072/files Let me know if you are actually on a different version. Boyang On Wed, Jul 10, 2

Group Coordinator stuck on PrepareRebalance state.

2019-07-10 Thread Aravind Dongara
Our kafka streams application is stuck and continuously emits "(Re-)joining group” log message every 5 minutes without making any progress. Kafka-consumer-groups cmd line tool with “—members” option shows lots of stale members, in addition to expected member-ids shown on log msgs on kafka-stream