Hi,
We have a Kafka cluster (version 1.1.1) where one node unexpectedly failed.
After that consumers from a couple of consumers are stuck in the poll() API
call. Looking at the thread dump, it looks like the consumer is stuck in
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady()
call. The heartbeat thread is also blocked waiting for the
ConsumerCoordinator. Any idea what is the cause and how to resolve this issue?
Thanks.
"BusinessEventRecordsDispatcherThread" #43 prio=5 os_prio=0
tid=0x00007f71764fb800 nid=0x241e sleeping[0x00007f70d24e9000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.kafka.common.utils.SystemTime.sleep(SystemTime.java:45)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:235)
- locked <0x00000004fc30e628> (a
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:205)
- locked <0x00000004fc30e628> (a
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:351)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:290)
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1149)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
...
at java.lang.Thread.run(Thread.java:748)
"kafka-coordinator-heartbeat-thread | prod-mkt-datahub-loader" #45 daemon
prio=5 os_prio=0 tid=0x00007f711a286800 nid=0x2422 in Object.wait()
[0x00007f7120125000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000004fc30e628> (a
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
at java.lang.Object.wait(Object.java:502)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:937)
- locked <0x00000004fc30e628> (a
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
