Re: Increase in Kafka replication fetcher thread not reducing log replication
Increasing replica.fetch.max.bytes will help, but will only get diminishing returns at some point. Increasing num.replica.fetchers will also help. You need to make sure that the leaders are balanced in the cluster. See http://kafka.apache.org/documentation.html#basic_ops_leader_balancing Thanks, Jun On Wed, Dec 17, 2014 at 8:08 AM, nitin sharma kumarsharma.ni...@gmail.com wrote: Hi All, I am trying to figure out best configuration for my Kafka brokers so that in case of restarted, the new node catch up with Leader at quick pace. My test environment has 2 kafka brokers and 1 Topic with one Partition. I first ran the test (Test#1) with default setting, i.e. num.replica.fetchers =1 and replica.fetch.max.bytes = 1048576 Bytes (1 MB). it took 11min and 40 sec to copy the 37.9 GB @ the rate 55.5MB/sec ( 37.9*1024/700) Later I increased the num.replica.fetchers = 5 and replica.fetch.max.bytes=1MB and ran another test (Test #2). I got the replica @ 89 MB/sec. Which is good but i was expecting 4*55 = 221 MB/sec. i ran two more test and results got much worse: Test#3 : replica thread = 5 and replica.fetch.max.bytes = 5MB. replication rate = 92.7 MB/sec Test #4: replication thread = 20 and replica.fetch.max.bytes = 5 MB. replication rate = 93.54 MB/sec Any reason why increasing the replica fetcher thread or increase in fetch max bytes not increasing my replication rate linearly. note: in all the test CPU utilization was not more than 45% Regards, Nitin Kumar Sharma.
Re: Increase in Kafka replication fetcher thread not reducing log replication
What kind of network do you have? gigabit? if so 90 MB/s would make sense Also since you have one partition what's your raw transfer speed to the disk? 90 MB/s makes sense here as well... If I were looking for rapid replica catch up I'd have at least 2x Gbit and partitioned topics spread out over several physical nodes. That would make it possible to catch up at ~200 MB/s without totally blocking normal operation of the cluster. For a single partition you would face lower catch up rate since all data is coming from one physical disk /svante 2014-12-22 17:26 GMT+01:00 Jun Rao j...@confluent.io: Increasing replica.fetch.max.bytes will help, but will only get diminishing returns at some point. Increasing num.replica.fetchers will also help. You need to make sure that the leaders are balanced in the cluster. See http://kafka.apache.org/documentation.html#basic_ops_leader_balancing Thanks, Jun On Wed, Dec 17, 2014 at 8:08 AM, nitin sharma kumarsharma.ni...@gmail.com wrote: Hi All, I am trying to figure out best configuration for my Kafka brokers so that in case of restarted, the new node catch up with Leader at quick pace. My test environment has 2 kafka brokers and 1 Topic with one Partition. I first ran the test (Test#1) with default setting, i.e. num.replica.fetchers =1 and replica.fetch.max.bytes = 1048576 Bytes (1 MB). it took 11min and 40 sec to copy the 37.9 GB @ the rate 55.5MB/sec ( 37.9*1024/700) Later I increased the num.replica.fetchers = 5 and replica.fetch.max.bytes=1MB and ran another test (Test #2). I got the replica @ 89 MB/sec. Which is good but i was expecting 4*55 = 221 MB/sec. i ran two more test and results got much worse: Test#3 : replica thread = 5 and replica.fetch.max.bytes = 5MB. replication rate = 92.7 MB/sec Test #4: replication thread = 20 and replica.fetch.max.bytes = 5 MB. replication rate = 93.54 MB/sec Any reason why increasing the replica fetcher thread or increase in fetch max bytes not increasing my replication rate linearly. note: in all the test CPU utilization was not more than 45% Regards, Nitin Kumar Sharma.
Increase in Kafka replication fetcher thread not reducing log replication
Hi All, I am trying to figure out best configuration for my Kafka brokers so that in case of restarted, the new node catch up with Leader at quick pace. My test environment has 2 kafka brokers and 1 Topic with one Partition. I first ran the test (Test#1) with default setting, i.e. num.replica.fetchers =1 and replica.fetch.max.bytes = 1048576 Bytes (1 MB). it took 11min and 40 sec to copy the 37.9 GB @ the rate 55.5MB/sec ( 37.9*1024/700) Later I increased the num.replica.fetchers = 5 and replica.fetch.max.bytes=1MB and ran another test (Test #2). I got the replica @ 89 MB/sec. Which is good but i was expecting 4*55 = 221 MB/sec. i ran two more test and results got much worse: Test#3 : replica thread = 5 and replica.fetch.max.bytes = 5MB. replication rate = 92.7 MB/sec Test #4: replication thread = 20 and replica.fetch.max.bytes = 5 MB. replication rate = 93.54 MB/sec Any reason why increasing the replica fetcher thread or increase in fetch max bytes not increasing my replication rate linearly. note: in all the test CPU utilization was not more than 45% Regards, Nitin Kumar Sharma.
