FrankYang0529 commented on code in PR #17444:
URL: https://github.com/apache/kafka/pull/17444#discussion_r1817877601
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/TopicMetadata.java:
##########
@@ -41,10 +41,25 @@ public class TopicMetadata {
*/
private final int numPartitions;
+ /**
+ * Map of every partition Id to a set of its rack Ids, if they exist.
+ * If rack information is unavailable for all partitions, this is an empty
map.
+ */
+ private final Map<Integer, Set<String>> partitionRacks;
Review Comment:
Hi @chia7712, thanks for the comments. Ideally, assignors should generate
deterministic results, so it doesn't trigger useless rebalance.
I have another approach to determine when do we need to generate a new
target assignment if there is new metadata image.
In `GroupMetadataManager#onNewMetadataImage`, it gets new metadata image and
metadata delta, so we can know which topics are updated in the new metadata.
Also, we have a `GroupMetadataManager#groupsByTopics` data, so we know which
group needs to rebalance when there is topic update.
For metadata change, there are two cases can trigger group rebalance:
1. There is new partition in a topic.
2. Rack information is changed. For `broker.rack` change, it can only happen
when we restart a broker.
In `TopicDelta`, we can get new partition by `TopicDelta#newPartitions`, so
this solves the first case.
In `PartitionRegistration`, we can get `removingReplicas` and
`addingReplicas`. The partition racks is calculated by partition replicas. The
`broker.rack` can only be changed when broker restarts. That means when broker
stop/start, it's in `removingReplicas`/`addingReplicas`.
Finally, we can set a new value in `ModernGroup`. When above cases are
detected in `GroupMetadataManager#onNewMetadataImage`, we set the value to
true. In new consumer group heartbeat, we don't need to calculate subscription
metadata, we can check the value and calculate a new target assignment
directly. After we gets new target assignment, we set the value back to false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
