What kind of network do you have? gigabit? if so 90 MB/s would make sense.... Also since you have one partition what's your raw transfer speed to the disk? 90 MB/s makes sense here as well...
If I were looking for rapid replica catch up I'd have at least 2x Gbit and partitioned topics spread out over several physical nodes. That would make it possible to catch up at ~200 MB/s without totally blocking normal operation of the cluster. For a single partition you would face lower catch up rate since all data is coming from one physical disk /svante 2014-12-22 17:26 GMT+01:00 Jun Rao <j...@confluent.io>: > Increasing replica.fetch.max.bytes will help, but will only get diminishing > returns at some point. > > Increasing num.replica.fetchers will also help. You need to make sure that > the leaders are balanced in the cluster. See > http://kafka.apache.org/documentation.html#basic_ops_leader_balancing > > Thanks, > > Jun > > On Wed, Dec 17, 2014 at 8:08 AM, nitin sharma <kumarsharma.ni...@gmail.com > > > wrote: > > > Hi All, > > > > I am trying to figure out best configuration for my Kafka brokers so that > > in case of restarted, the new node catch up with Leader at quick pace. > > > > My test environment has 2 kafka brokers and 1 Topic with one Partition. > > > > I first ran the test (Test#1) with default setting, i.e. > > num.replica.fetchers =1 and replica.fetch.max.bytes = 1048576 Bytes (1 > MB). > > it took 11min and 40 sec to copy the 37.9 GB @ the rate 55.5MB/sec ( > > 37.9*1024/700) > > > > Later I increased the num.replica.fetchers = 5 and > > replica.fetch.max.bytes=1MB and ran another test (Test #2). I got the > > replica @ 89 MB/sec. Which is good but i was expecting 4*55 = 221 > MB/sec. > > > > i ran two more test and results got much worse: > > Test#3 : replica thread = 5 and replica.fetch.max.bytes = 5MB. > > replication rate = 92.7 MB/sec > > > > Test #4: replication thread = 20 and replica.fetch.max.bytes = 5 MB. > > replication rate = 93.54 MB/sec > > > > Any reason why increasing the replica fetcher thread or increase in fetch > > max bytes not increasing my replication rate linearly. > > > > note: in all the test CPU utilization was not more than 45% > > > > > > Regards, > > Nitin Kumar Sharma. > > >
