Sean Quah created KAFKA-19160:
---------------------------------
Summary: Fetching stable offsets is very slow with a large number
of partition and open transactions
Key: KAFKA-19160
URL: https://issues.apache.org/jira/browse/KAFKA-19160
Project: Kafka
Issue Type: Bug
Components: group-coordinator
Reporter: Sean Quah
Assignee: Sean Quah
When fetching stable offsets in the group coordinator, we iterate over all
requested partitions and for each partition, check if any ongoing transactions
have updated the group's offset for that partition. This is O(#partitions *
#open_transactions).
For a large topic with 4,000 partitions and a correspondingly large group, we
might have up to 4,000 open transactions. Fetching stable offsets for all
partitions in this case can take around 350 ms.
It looks like the old coordinator also has this issue but it's being removed
soon anyway.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
