Hi Brian, thanks for the recommendation. I do in fact plan to use autothrottle (although I need to make it work with our prometheus metrics first).
However if I understand correctly, I would need to apply a throttle to essentially every partition of every topic, since in the use case I described I don't directly control which topic/partition is replicated. Is that correct ? On Thu, Feb 20, 2020, at 22:09, Brian Sang wrote: > Hey Vincent, I think you need to set both the configs for the brokers and > the individual partitions you are moving themselves. > > For an automated system that can make this easier, check out > https://github.com/DataDog/kafka-kit/tree/master/cmd/autothrottle, though > it requires DataDog, you can use a similar algorithm for applying throttles > on reassignment. > > We have something at Yelp that is similar built in-house. > > Best, > Brian > > > > On Thu, Feb 20, 2020 at 5:46 AM Vincent Rischmann <vinc...@rischmann.fr> > wrote: > > > Hello, > > > > We have a cluster of 10 brokers. > > > > recently we replaced some broken HDDs on a single broker (id 2 for future > > reference), all data on this broker was erased. > > We have a replication factor of 3 minimum on all our topics so no data was > > lost. > > > > To add the broker to the cluster again I configured it with the same > > broker id than before, assuming the replication will take care of > > everything. This worked and after a couple hours the cluster was fully up > > to date with no under replicated partitions. > > > > Although this worked, this replication had a visible inpact on the load of > > the cluster and on some consumers especially. > > During replication broker 2 almost always capped the 1Gbps link, there was > > around 3TB to transfer. > > > > Before bootstrapping the node I applied the following configurations > > thinking it would work: > > > > for i in (seq 1 11) ; kafka-configs.sh --zookeeper 10.40.1.1:2181 > > --alter --entity-type brokers --add-config > > leader.replication.throttled.rate=524288,follower.replication.throttled.rate=524288 > > --entity-name $i ; end > > > > but it didn't seem to do anything. After reading > > https://kafka.apache.org/11/documentation.html#rep-throttle I think these > > config may not be do what I want, it looks like the throttling must be > > configured with leader.replication.throttled.replicas and > > follower.replication.throttled.replicas for each topic ? > > > > Basically, I'm looking for a way to tell a broker to throttle itself > > globally at for example 100 Mbps. Is there an easy way for me do to this ? > > >
