zhangtongr created KAFKA-19558:
----------------------------------
Summary: kafka-consumer-groups.sh --describe --all-groups command
times out on large clusters with many consumer groups
Key: KAFKA-19558
URL: https://issues.apache.org/jira/browse/KAFKA-19558
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 2.7.1
Reporter: zhangtongr
Description:
When running the following command in a Kafka cluster with a large number of
consumer groups (over 380) and topics (over 500), the kafka-consumer-groups.sh
--describe --all-groups operation consistently times out and fails to return
results.
Command used:
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe
--all-groups
Observed behavior:
The command fails with a TimeoutException, and no consumer group information is
returned. The following stack trace is observed:
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.TimeoutException:
Call(callName=describeConsumerGroups, deadlineMs=1753170317381, tries=1,
nextAllowedTryMs=1753170317482) timed out at 1753170317382 after 1 attempt(s)
at
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
...
Caused by: org.apache.kafka.common.errors.TimeoutException:
Call(callName=describeConsumerGroups, deadlineMs=..., tries=1, ...) timed out
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting
to send the call. Call: describeConsumerGroups
Expected behavior:
The command should be able to return the description of all consumer groups, or
at least fail more gracefully. Ideally, there should be:
A way to paginate or batch the describe operation;
Or configuration options to increase internal timeout thresholds;
Or better recommendations for dealing with large clusters.
Additional context:
Manually describing individual consumer groups via --group performs as expected
and returns data quickly.
The issue appears to scale linearly with the number of consumer groups.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
