Hi All

We’ve made an adjustment to KIP-73: Replication Quotas which I’d like to open 
up to the community for approval. 

Previously the admin passed a list of replicas to be throttled:

        quota.replication.throttled.replicas = 
[partId]:[replica],[partId]:[replica],[partId]:[replica] etc

The change is to split this into two properties. One that corresponds to the 
leader-side throttle, and the other that corresponds to the follower-side 
throttle.

        quota.leader.replication.throttled.replicas = 
[partId]:[replica],[partId]:[replica],[partId]:[replica] 
        quota.follower.replication.throttled.replicas = 
[partId]:[replica],[partId]:[replica],[partId]:[replica] 

This provides more control over the throttling process. It also helps us with a 
common use case which I’ve described below, for those interested. 

Please reply if you have any comments / issues / suggestions. 

Thanks as ever.

Ben


Sample Use Case:

Say we are moving partition 0. It has replicas [104,107,109] moving to 
[105,107,113]

So the leader could be any of [104,107,109] and we know we will be creating new 
replicas on 105 & 113.

In the current mechanism, all we can do is apply both leader and follower 
throttles to all 5:  [104,107,109,105,113] which will mean the regular 
replication traffic between (say 107 is leader) 107->104 and 107->109 will be 
throttled also. This is potentially problematic. 

What we really want to do is apply:

LeaderThrottle [104,107,109]
FollowerThrottle [105,113]

This way, during a rebalance, we that standard replication traffic will not be  
throttled, but the rebalance will perform correctly if leaders move. One 
subtlety is that, should the leader move to the “move destination” node, it 
would no longer be throttled. But this is actually to our benefit in the normal 
case. 

Reply via email to