Lincong Li created KAFKA-12854:
----------------------------------
Summary: Add a config to allow skipping metadata cache update when
topic partition is unassigned
Key: KAFKA-12854
URL: https://issues.apache.org/jira/browse/KAFKA-12854
Project: Kafka
Issue Type: Improvement
Components: clients
Reporter: Lincong Li
The "assign" method in the consumer triggers a metadata cache update if the new
partition assignment is different from the current assignment. It makes sense
to update the MD cache if the new assignment contains partitions that do not
exist in the current assignment.
However, I wonder why is updating its MD cache necessary if the new partition
assignment is a subset of the current assignment. For example, the new
assignment is tp0, tp1, and the current assignment is tp0, tp1, tp2.
The current behavior does too many drawbacks in most cases. However, if the
number of consumer instances is large and each consumer instance is constantly
getting some topic partitions unassigned, the QPS of the MD request sent out to
update the MD cache becomes high as a result.
Proposed changes:
Add a config to allow skipping metadata cache update when topic partition(s) is
unassigned
Existing PR to a forked repo:
https://github.com/linkedin/kafka/pull/166/files
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
