yeah that's not a bad idea. So to recap, it's enough to set follower.replication.throttled.replicas for every partition assigned to that broker ? I'm assuming that during bootstrap the broker is a follower for everything.
On Fri, Feb 21, 2020, at 22:09, Brian Sang wrote: > You can get the list of partitions that was on the broker that you > replaced. That way you don't need to apply throttle to every topic > partition in the cluster. > > On Fri, Feb 21, 2020 at 4:48 AM Vincent Rischmann <vinc...@rischmann.fr> > wrote: > > > Hi Brian, > > > > thanks for the recommendation. I do in fact plan to use autothrottle > > (although I need to make it work with our prometheus metrics first). > > > > However if I understand correctly, I would need to apply a throttle to > > essentially every partition of every topic, since in the use case I > > described I don't directly control which topic/partition is replicated. > > > > Is that correct ? > > > > On Thu, Feb 20, 2020, at 22:09, Brian Sang wrote: > > > Hey Vincent, I think you need to set both the configs for the brokers and > > > the individual partitions you are moving themselves. > > > > > > For an automated system that can make this easier, check out > > > https://github.com/DataDog/kafka-kit/tree/master/cmd/autothrottle, > > though > > > it requires DataDog, you can use a similar algorithm for applying > > throttles > > > on reassignment. > > > > > > We have something at Yelp that is similar built in-house. > > > > > > Best, > > > Brian > > > > > > > > > > > > On Thu, Feb 20, 2020 at 5:46 AM Vincent Rischmann <vinc...@rischmann.fr> > > > wrote: > > > > > > > Hello, > > > > > > > > We have a cluster of 10 brokers. > > > > > > > > recently we replaced some broken HDDs on a single broker (id 2 for > > future > > > > reference), all data on this broker was erased. > > > > We have a replication factor of 3 minimum on all our topics so no data > > was > > > > lost. > > > > > > > > To add the broker to the cluster again I configured it with the same > > > > broker id than before, assuming the replication will take care of > > > > everything. This worked and after a couple hours the cluster was fully > > up > > > > to date with no under replicated partitions. > > > > > > > > Although this worked, this replication had a visible inpact on the > > load of > > > > the cluster and on some consumers especially. > > > > During replication broker 2 almost always capped the 1Gbps link, there > > was > > > > around 3TB to transfer. > > > > > > > > Before bootstrapping the node I applied the following configurations > > > > thinking it would work: > > > > > > > > for i in (seq 1 11) ; kafka-configs.sh --zookeeper 10.40.1.1:2181 > > > > --alter --entity-type brokers --add-config > > > > > > leader.replication.throttled.rate=524288,follower.replication.throttled.rate=524288 > > > > --entity-name $i ; end > > > > > > > > but it didn't seem to do anything. After reading > > > > https://kafka.apache.org/11/documentation.html#rep-throttle I think > > these > > > > config may not be do what I want, it looks like the throttling must be > > > > configured with leader.replication.throttled.replicas and > > > > follower.replication.throttled.replicas for each topic ? > > > > > > > > Basically, I'm looking for a way to tell a broker to throttle itself > > > > globally at for example 100 Mbps. Is there an easy way for me do to > > this ? > > > > > > > > > >
