Thanks for your response Jun. JIRA has been filed (see link below). Please let me know if I should add more details/context:
https://issues.apache.org/jira/browse/KAFKA-1464 Thanks, Marcos Juarez On Wed, May 21, 2014 at 8:40 AM, Jun Rao <jun...@gmail.com> wrote: > We don't have such throttling right now. Could you file a jira? > > Thanks, > > Jun > > > On Tue, May 20, 2014 at 10:29 PM, Marcos Juarez Lopez <mjua...@gmail.com > >wrote: > > > Hi, > > > > We have several Kafka clusters in production, and we've had to reassign > > replication a few times now in production. Some of our topic/partitions > > are pretty large, up to 32 partitions per topic, and 16GB per partition, > so > > adding a new broker and/or repairing a broker that had been down for some > > time turns out to be a major undertaking. > > > > Today, when we attempt to replicate a single partition, it pegs the disk > > IO, and uses a significant chunk of the 10Gbps interface for a good ~5 > > minutes. This is causing problems for our downstream consumers, which > rely > > on having a consistent stream of realtime data being sent to them. > > > > Is there a way to throttle Kafka replication between nodes, so that > instead > > of it going full blast, it will replicate at a fixed rate in megabytes or > > activities/batches per second? Or maybe is this planned for a future > > release, maybe 0.9? > > > > Thanks, > > > > Marcos Juarez > > >