Makes sense to me. Thanks Ben! On Mon, Sep 26, 2016 at 9:39 AM Jun Rao <j...@confluent.io> wrote:
> Yes, this change makes sense to me since it gives the admin better control. > > Thanks, > > Jun > > On Sun, Sep 25, 2016 at 7:35 AM, Ben Stopford <b...@confluent.io> wrote: > > > Hi All > > > > We’ve made an adjustment to KIP-73: Replication Quotas which I’d like to > > open up to the community for approval. > > > > Previously the admin passed a list of replicas to be throttled: > > > > quota.replication.throttled.replicas = > > [partId]:[replica],[partId]:[replica],[partId]:[replica] etc > > > > The change is to split this into two properties. One that corresponds to > > the leader-side throttle, and the other that corresponds to the > > follower-side throttle. > > > > quota.leader.replication.throttled.replicas = > > [partId]:[replica],[partId]:[replica],[partId]:[replica] > > quota.follower.replication.throttled.replicas = > > [partId]:[replica],[partId]:[replica],[partId]:[replica] > > > > This provides more control over the throttling process. It also helps us > > with a common use case which I’ve described below, for those interested. > > > > Please reply if you have any comments / issues / suggestions. > > > > Thanks as ever. > > > > Ben > > > > > > Sample Use Case: > > > > Say we are moving partition 0. It has replicas [104,107,109] moving to > > [105,107,113] > > > > So the leader could be any of [104,107,109] and we know we will be > > creating new replicas on 105 & 113. > > > > In the current mechanism, all we can do is apply both leader and follower > > throttles to all 5: [104,107,109,105,113] which will mean the regular > > replication traffic between (say 107 is leader) 107->104 and 107->109 > will > > be throttled also. This is potentially problematic. > > > > What we really want to do is apply: > > > > LeaderThrottle [104,107,109] > > FollowerThrottle [105,113] > > > > This way, during a rebalance, we that standard replication traffic will > > not be throttled, but the rebalance will perform correctly if leaders > > move. One subtlety is that, should the leader move to the “move > > destination” node, it would no longer be throttled. But this is actually > to > > our benefit in the normal case. > > > > > -- Thanks, Neha
