[ https://issues.apache.org/jira/browse/KAFKA-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17598108#comment-17598108 ]
Grant Neale commented on KAFKA-5337: ------------------------------------ There is no plan to implement this as a core feature. If you would find it useful, you could always implement your own partition assignment strategy based on the prototype at [https://github.com/grantneale/kafka-lag-based-assignor] > Partition assignment strategy that distributes lag evenly across consumers in > each group > ---------------------------------------------------------------------------------------- > > Key: KAFKA-5337 > URL: https://issues.apache.org/jira/browse/KAFKA-5337 > Project: Kafka > Issue Type: New Feature > Components: consumer > Affects Versions: 0.10.2.1 > Reporter: Grant Neale > Priority: Minor > > Existing partition assignment strategies (RangeAssignor and > RoundRobinAssignor) do not account for the current consumer group lag on each > partition. This can result in sub-optimal assignments when the distribution > of lags for a given topic and consumer group is skewed. > The LagBasedAssignor operates on a per-topic basis, and attempts to assign > partitions such that lag is distributed as evenly across a consumer group. > h4. Algorithm: > For each topic, we first obtain the lag on all partitions. Lag on a given > partition is the difference between the end offset and the last offset > committed by the consumer group. If no offsets have been committed for a > partition we determine the lag based on the code auto.offset.reset property. > If auto.offset.reset=latest, we assign a lag of 0. If > auto.offset.reset=earliest (or any other value) we assume assign lag equal to > the total number of message currently available in that partition. > We then create a map storing the current total lag of all partitions assigned > to each member of the consumer group. Partitions are assigned in decreasing > order of lag, with each partition assigned to the consumer with least total > number of assigned partitions, breaking ties by assigning to the consumer > with the least total assigned lag. > Distributing partitions evenly across consumers (by count) ensures that the > partition assignment is balanced when all partitions have a current lag of 0 > or if the distribution of lags is heavily skewed. It also gives the consumer > group the best possible chance of remaining balanced if the assignment is > retained for a long period. -- This message was sent by Atlassian Jira (v8.20.10#820010)