[ https://issues.apache.org/jira/browse/KAFKA-9352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006244#comment-17006244 ]
Ning Zhang commented on KAFKA-9352: ----------------------------------- pr: https://github.com/apache/kafka/pull/7880 > unbalanced assignment of topic-partition to tasks > ------------------------------------------------- > > Key: KAFKA-9352 > URL: https://issues.apache.org/jira/browse/KAFKA-9352 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker > Affects Versions: 2.4.0 > Reporter: Ning Zhang > Priority: Major > Fix For: 2.5.0 > > Attachments: Screen Shot 2019-12-19 at 12.16.02 PM.png, Screen Shot > 2019-12-19 at 8.22.17 AM.png > > > originally, when mirrormaker replicates a group of topics, the assignment > between topic-partition and tasks are pretty static. E.g. partitions from the > same topic tend to be grouped together as much as possible on the same task. > For example, 3 tasks to mirror 3 topics with 8, 2 and 2 > partitions respectively. 't1' denotes 'task 1', 't0p5' denotes 'topic 0, > partition 5' > The original assignment will look like: > t1 -> [t0p0, t0p1, t0p2, t0p3] > t2 -> [t0p4, t0p5, t0p6, t0p7] > t3 -> [t1p0, t1p2, t2p0, t2p1] > The potential issue of above assignment is: if topic 0 has more traffic than > other topics (topic 1, topic 2), t1 and t2 will be loaded more traffic than > t3. When the tasks are mapped to the mirrormaker instances (workers) and > launched, it will create unbalanced load on the workers. Please see the > picture below as an unbalanced example of 2 mirrormaker instances: > !Screen Shot 2019-12-19 at 12.16.02 PM.png! > Given each mirrored topic has different traffic and number of partitions, to > balance the load > across all mirrormaker instances (workers), 'roundrobin' helps to evenly > assign all > topic-partition to the tasks, then the tasks are further distributed to > workers by calling > 'ConnectorUtils.groupPartitions()'. For example, 3 tasks to mirror 3 topics > with 8, 2 and 2 > partitions respectively. 't1' denotes 'task 1', 't0p5' denotes 'topic 0, > partition 5' > t1 -> [t0p0, t0p3, t0p6, t1p1] > t2 -> [t0p1, t0p4, t0p7, t2p0] > t3 -> [t0p2, t0p5, t1p0, t2p1] > The improvement of this new above assignment over the original assignment is: > the partitions of topic 0, topic 1 and topic 2 are all spread over all tasks, > which creates a relatively even load on all workers, after the tasks are > mapped to the workers and launched. > Please see the picture below as a balanced example of 4 mirrormaker instances: > !Screen Shot 2019-12-19 at 8.22.17 AM.png! > PR link is: https://github.com/apache/kafka/pull/7880 > -- This message was sent by Atlassian Jira (v8.3.4#803005)