Re: rate-limiting on rebalancing, or sync from non-leaders?

Hi Charity There will be a KIP for this coming out shortly. All the best B > On 4 Jul 2016, at 13:14, Alexis Midon wrote: > > Same here at Airbnb. Moving data is the biggest operational challenge > because of the network bandwidth cannibalization. > I was

Re: rate-limiting on rebalancing, or sync from non-leaders?

Same here at Airbnb. Moving data is the biggest operational challenge because of the network bandwidth cannibalization. I was hoping that rate limiting would apply to replica fetchers too. On Sun, Jul 3, 2016 at 15:38 Tom Crayford wrote: > Hi Charity, > > I'm not sure

Re: rate-limiting on rebalancing, or sync from non-leaders?

Hi Charity, I'm not sure about the roadmap. The way we (and linkedin/dropbox/netflix) handle rebalances right now is to do a small handful of partitions at a time (LinkedIn does 10 partitions at a time the last I heard), not a big bang rebalance of all the partitions in the cluster. That's not

rate-limiting on rebalancing, or sync from non-leaders?

Hi there, I'm curious if there's anything on the Kafka roadmap for adding rate-limiting or max-throughput for rebalancing processes. Alternately, if you have RF>2, maybe a setting to instruct followers to sync from other followers? I'm super impressed with how fast and efficient the kafka data