Thanks Ben, this makes sense to me. As discussed in the PR[1] and the JIRA you filed[2], it probably makes sense to also split the throttle limit replication.quota.throttled.rate into separate leader and follower limits.
Ismael [1] *https://github.com/apache/kafka/pull/1906#issuecomment-249733772 <https://github.com/apache/kafka/pull/1906#issuecomment-249733772>* [2] https://issues.apache.org/jira/browse/KAFKA-4225 On Sun, Sep 25, 2016 at 3:35 PM, Ben Stopford <b...@confluent.io> wrote: > Hi All > > We’ve made an adjustment to KIP-73: Replication Quotas which I’d like to > open up to the community for approval. > > Previously the admin passed a list of replicas to be throttled: > > quota.replication.throttled.replicas = > [partId]:[replica],[partId]:[replica],[partId]:[replica] etc > > The change is to split this into two properties. One that corresponds to > the leader-side throttle, and the other that corresponds to the > follower-side throttle. > > quota.leader.replication.throttled.replicas = > [partId]:[replica],[partId]:[replica],[partId]:[replica] > quota.follower.replication.throttled.replicas = > [partId]:[replica],[partId]:[replica],[partId]:[replica] > > This provides more control over the throttling process. It also helps us > with a common use case which I’ve described below, for those interested. > > Please reply if you have any comments / issues / suggestions. > > Thanks as ever. > > Ben > > > Sample Use Case: > > Say we are moving partition 0. It has replicas [104,107,109] moving to > [105,107,113] > > So the leader could be any of [104,107,109] and we know we will be > creating new replicas on 105 & 113. > > In the current mechanism, all we can do is apply both leader and follower > throttles to all 5: [104,107,109,105,113] which will mean the regular > replication traffic between (say 107 is leader) 107->104 and 107->109 will > be throttled also. This is potentially problematic. > > What we really want to do is apply: > > LeaderThrottle [104,107,109] > FollowerThrottle [105,113] > > This way, during a rebalance, we that standard replication traffic will > not be throttled, but the rebalance will perform correctly if leaders > move. One subtlety is that, should the leader move to the “move > destination” node, it would no longer be throttled. But this is actually to > our benefit in the normal case. > >
