You can get the list of partitions that was on the broker that you replaced. That way you don't need to apply throttle to every topic partition in the cluster.
On Fri, Feb 21, 2020 at 4:48 AM Vincent Rischmann <vinc...@rischmann.fr> wrote: > Hi Brian, > > thanks for the recommendation. I do in fact plan to use autothrottle > (although I need to make it work with our prometheus metrics first). > > However if I understand correctly, I would need to apply a throttle to > essentially every partition of every topic, since in the use case I > described I don't directly control which topic/partition is replicated. > > Is that correct ? > > On Thu, Feb 20, 2020, at 22:09, Brian Sang wrote: > > Hey Vincent, I think you need to set both the configs for the brokers and > > the individual partitions you are moving themselves. > > > > For an automated system that can make this easier, check out > > https://github.com/DataDog/kafka-kit/tree/master/cmd/autothrottle, > though > > it requires DataDog, you can use a similar algorithm for applying > throttles > > on reassignment. > > > > We have something at Yelp that is similar built in-house. > > > > Best, > > Brian > > > > > > > > On Thu, Feb 20, 2020 at 5:46 AM Vincent Rischmann <vinc...@rischmann.fr> > > wrote: > > > > > Hello, > > > > > > We have a cluster of 10 brokers. > > > > > > recently we replaced some broken HDDs on a single broker (id 2 for > future > > > reference), all data on this broker was erased. > > > We have a replication factor of 3 minimum on all our topics so no data > was > > > lost. > > > > > > To add the broker to the cluster again I configured it with the same > > > broker id than before, assuming the replication will take care of > > > everything. This worked and after a couple hours the cluster was fully > up > > > to date with no under replicated partitions. > > > > > > Although this worked, this replication had a visible inpact on the > load of > > > the cluster and on some consumers especially. > > > During replication broker 2 almost always capped the 1Gbps link, there > was > > > around 3TB to transfer. > > > > > > Before bootstrapping the node I applied the following configurations > > > thinking it would work: > > > > > > for i in (seq 1 11) ; kafka-configs.sh --zookeeper 10.40.1.1:2181 > > > --alter --entity-type brokers --add-config > > > > leader.replication.throttled.rate=524288,follower.replication.throttled.rate=524288 > > > --entity-name $i ; end > > > > > > but it didn't seem to do anything. After reading > > > https://kafka.apache.org/11/documentation.html#rep-throttle I think > these > > > config may not be do what I want, it looks like the throttling must be > > > configured with leader.replication.throttled.replicas and > > > follower.replication.throttled.replicas for each topic ? > > > > > > Basically, I'm looking for a way to tell a broker to throttle itself > > > globally at for example 100 Mbps. Is there an easy way for me do to > this ? > > > > > >
