Andrew Olson created KAFKA-3297:
-----------------------------------
Summary: More optimally balanced partition assignment strategy
(new consumer)
Key: KAFKA-3297
URL: https://issues.apache.org/jira/browse/KAFKA-3297
Project: Kafka
Issue Type: Improvement
Reporter: Andrew Olson
Assignee: Andrew Olson
While the roundrobin partition assignment strategy is an improvement over the
range strategy, when the consumer topic subscriptions are not identical
(previously disallowed but will be possible as of KAFKA-2172) it can produce
heavily skewed assignments. As suggested
[here|https://issues.apache.org/jira/browse/KAFKA-2172?focusedCommentId=14530767&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14530767]
it would be nice to have a strategy that attempts to assign an equal number of
partitions to each consumer in a group, regardless of how similar their
individual topic subscriptions are. We can accomplish this by tracking the
number of partitions assigned to each consumer, and having the partition
assignment loop assign each partition to a consumer interested in that topic
with the least number of partitions assigned.
Additionally, we can optimize the distribution fairness by adjusting the
partition assignment order:
* Topics with fewer consumers are assigned first.
* In the event of a tie for least consumers, the topic with more partitions is
assigned first.
The general idea behind these two rules is to keep the most flexible assignment
choices available as long as possible by starting with the most constrained
partitions/consumers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
