[ https://issues.apache.org/jira/browse/KAFKA-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324358#comment-17324358 ]
Travis Bischel commented on KAFKA-12675: ---------------------------------------- Great insight on getting rid of partition2AllPotentialConsumers, as well as keeping some more things sorted! I was able to translate that into my own code and dropped the large imbalance from 9.5s to 0.5s, as well as from 8.5G memory util to 0.5G :) I'll take a look at the code more in depth soon. > Improve sticky general assignor scalability and performance > ----------------------------------------------------------- > > Key: KAFKA-12675 > URL: https://issues.apache.org/jira/browse/KAFKA-12675 > Project: Kafka > Issue Type: Improvement > Reporter: Luke Chen > Assignee: Luke Chen > Priority: Major > > Currently, we have "general assignor" for non-equal subscription case and > "constrained assignor" for all equal subscription case. There's a performance > test for constrained assignor with: > topicCount = {color:#0000ff}500{color}; > partitionCount = {color:#0000ff}2000{color}; > consumerCount = {color:#0000ff}2000{color}; > in _testLargeAssignmentAndGroupWithUniformSubscription,_ total 1 million > partitions and we can complete the assignment within 2 second in my machine. > However, if we let 1 of the consumer subscribe to only 1 topic, it'll use > "general assignor", and the result with the same setting as above is: > *OutOfMemory,* > Even we down the count to: > topicCount = {color:#0000ff}50{color}; > partitionCount = 1{color:#0000ff}000{color}; > consumerCount = 1{color:#0000ff}000{color}; > We still got *OutOfMemory*. > With this setting: > topicCount = {color:#0000ff}50{color}; > partitionCount = 8{color:#0000ff}00{color}; > consumerCount = 8{color:#0000ff}00{color}; > We can complete in 10 seconds in my machine, which is still slow. > > Since we are going to set default assignment strategy to > "CooperativeStickyAssignor" soon, we should improve the scalability and > performance for sticky general assignor. -- This message was sent by Atlassian Jira (v8.3.4#803005)