Hi All, I am trying to upgrade my Kafka streams application to 2.7.1 version of Kafka. The brokers are upgraded to 2.7.1 and kafka dependencies are also on 2.7.1. But when I start the application, rebalance is failing with the following message
Rebalance failed. org.apache.kafka.common.errors.DisconnectException I am also seeing Group coordinator broker102:9092 (id: 2147483645 rack: null) is unavailable or invalid due to cause: coordinator unavailable.isDisconnected: false. Rediscovery will be attempted. The full set of logs (which gets printed every 30 seconds) is given below INFO 2021-07-09 09:33:20.229 | internals.AbstractCoordinator [Consumer clientId=consumer-my-app-v1-task-coordinator-consumer-app-node100-9, groupId=my-app-v1-task-coordinator-consumer-app-node100] Group coordinator broker102:9092 (id: 2147483645 rack: null) is unavailable or invalid due to cause: null.isDisconnected: true. Rediscovery will be attempted. INFO 2021-07-09 09:33:20.230 | internals.AbstractCoordinator [Consumer clientId=consumer-my-app-v1-task-coordinator-consumer-app-node100-9, groupId=my-app-v1-task-coordinator-consumer-app-node100] Discovered group coordinator broker102:9092 (id: 2147483645 rack: null) INFO 2021-07-09 09:33:20.230 | internals.AbstractCoordinator [Consumer clientId=consumer-my-app-v1-task-coordinator-consumer-app-node100-9, groupId=my-app-v1-task-coordinator-consumer-app-node100] Group coordinator broker102:9092 (id: 2147483645 rack: null) is unavailable or invalid due to cause: coordinator unavailable.isDisconnected: false. Rediscovery will be attempted. INFO 2021-07-09 09:33:20.330 | internals.AbstractCoordinator [Consumer clientId=consumer-my-app-v1-task-coordinator-consumer-app-node100-9, groupId=my-app-v1-task-coordinator-consumer-app-node100] Discovered group coordinator broker102:9092 (id: 2147483645 rack: null) INFO 2021-07-09 09:33:20.331 | internals.AbstractCoordinator [Consumer clientId=consumer-my-app-v1-task-coordinator-consumer-app-node100-9, groupId=my-app-v1-task-coordinator-consumer-app-node100] Rebalance failed. org.apache.kafka.common.errors.DisconnectException INFO 2021-07-09 09:33:20.331 | internals.AbstractCoordinator [Consumer clientId=consumer-my-app-v1-task-coordinator-consumer-app-node100-9, groupId=my-app-v1-task-coordinator-consumer-app-node100] (Re-)joining group INFO 2021-07-09 09:33:20.333 | internals.AbstractCoordinator [Consumer clientId=consumer-my-app-v1-task-coordinator-consumer-app-node100-9, groupId=my-app-v1-task-coordinator-consumer-app-node100] (Re-)joining group INFO 2021-07-09 09:33:20.419 | internals.AbstractCoordinator [Consumer clientId=consumer-my-app-v1-master-coordinator-consumer-8, groupId=my-app-v1-master-coordinator-consumer] Group coordinator broker101:9092 (id: 2147483646 rack: null) is unavailable or invalid due to cause: null.isDisconnected: true. Rediscovery will be attempted. INFO 2021-07-09 09:33:20.420 | internals.AbstractCoordinator [Consumer clientId=consumer-my-app-v1-master-coordinator-consumer-8, groupId=my-app-v1-master-coordinator-consumer] Discovered group coordinator broker101:9092 (id: 2147483646 rack: null) INFO 2021-07-09 09:33:20.420 | internals.AbstractCoordinator [Consumer clientId=consumer-my-app-v1-master-coordinator-consumer-8, groupId=my-app-v1-master-coordinator-consumer] Group coordinator broker101:9092 (id: 2147483646 rack: null) is unavailable or invalid due to cause: coordinator unavailable.isDisconnected: false. Rediscovery will be attempted. INFO 2021-07-09 09:33:20.521 | internals.AbstractCoordinator [Consumer clientId=consumer-my-app-v1-master-coordinator-consumer-8, groupId=my-app-v1-master-coordinator-consumer] Discovered group coordinator broker101:9092 (id: 2147483646 rack: null) INFO 2021-07-09 09:33:20.522 | internals.AbstractCoordinator [Consumer clientId=consumer-my-app-v1-master-coordinator-consumer-8, groupId=my-app-v1-master-coordinator-consumer] Rebalance failed. org.apache.kafka.common.errors.DisconnectException INFO 2021-07-09 09:33:20.523 | internals.AbstractCoordinator [Consumer clientId=consumer-my-app-v1-master-coordinator-consumer-8, groupId=my-app-v1-master-coordinator-consumer] (Re-)joining group INFO 2021-07-09 09:33:20.524 | internals.AbstractCoordinator [Consumer clientId=consumer-my-app-v1-master-coordinator-consumer-8, groupId=my-app-v1-master-coordinator-consumer] (Re-)joining group The application was working fine on 2.5.1. Please note with 2.5.1 the build used was kafka_*2.12*-2.5.1, but with 2.7.1 I used kafka_*2.13*-2.7.1 *The broker config is * broker.id=2 listeners=PLAINTEXT://broker102:9092 advertised.listeners=PLAINTEXT://broker102:9092 num.network.threads=4 num.io.threads=4 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/mnt/store/kafka/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=4 offsets.topic.replication.factor=2 transaction.state.log.replication.factor=2 transaction.state.log.min.isr=2 log.retention.hours=1 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=broker100:2181,broker101:2181,broker102:2181,broker103:2181,broker104:2181 zookeeper.connection.timeout.ms=30000 group.initial.rebalance.delay.ms=120000 offsets.retention.minutes=10080 message.max.bytes=31457280 replica.fetch.max.bytes=31457280 group.max.session.timeout.ms=1200000 request.timeout.ms=600000 connections.max.idle.ms=1080000 What could be wrong? Should I switch to kafka_*2.12*-2.7.1 ? Thanks, Tony
