Nicholas Telford created KAFKA-15178:
----------------------------------------
Summary: Poor performance of ConsumerCoordinator with many
TopicPartitions
Key: KAFKA-15178
URL: https://issues.apache.org/jira/browse/KAFKA-15178
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 3.5.0
Reporter: Nicholas Telford
Assignee: Nicholas Telford
Attachments: pollPhase.png
Doing some profiling of my Kafka Streams application, I noticed that the
{{pollPhase}} suffers from a minor performance issue.
See flame graph below. !pollPhase.png|width=1028,height=308!
{{ConsumerCoordinator.poll}} calls {{{}rejoinNeededOrPending{}}}, which checks
the current {{metadataSnapshot}} against the {{{}assignmentSnapshot{}}}. This
comparison is a deep-equality check, and if there's a large number of
topic-partitions being consumed by the application, then this comparison can
perform poorly.
I suspect this can be trivially addressed with a {{boolean}} flag that
indicates when the {{metadataSnapshot}} has been updated (or is "dirty"), and
actually needs to be checked, since most of the time it should be identical to
{{{}assignmentSnapshot{}}}.
I plan to raise a PR with this optimization to address this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
