One partition is one file on disk. This is why your throughput per
partition will not go over 150MB/s.
I can not explain why you do not get 300MB/s for two partition files on
different disks. It could be related to your hardware. How fast are your
writes using 2 dd processes?
To get more throughput you would normally use more brokers on more
machines with a lot of partitions.
Am 17.05.2017 um 11:13 schrieb 陈 建平Chen Jianping:
Hi Group,
Recently I am trying to turn Kafka write performance to improve throughput. On
my Kafka broker, there are 3 disks (7200 RPM).
For one disk, the Kafka write throughput can reach 150MB/s. In my opinion, if I
send message to topic test_p3 (which has 3 partitions located on different disk
in the same server), the whole write throughput can reach 450 MB/s due to
parallel writing disk. However the test result is still 150MB/s. Is there any
reason that multiple disk doesn’t multiply the write throughput? And is there
any bottleneck for the Kafka write throughput or I need some configuration to
update?
I also try to test sending message to two different topic (whose partition on
different disk of that server), and the total throughput only reach 200 MB/s
instead of 300 MB/s as I expect. Below is my Kafka configuration and setting.
Thanks for any idea or advice on it:)
##Kafka producer setting
./kafka-run-class org.apache.kafka.tools.ProducerPerformance --topic test_p3
--num-records 50000000 --record-size 100 --throughput -1 --producer-props
acks=0 bootstrap.servers=localhost:9092 buffer.memory=33554432 batch.size=16384
##OS tuning setting
net.core.rmem_default = 124928
net.core.rmem_max = 2048000
net.core.wmem_default = 124928
net.core.wmem_max = 2048000
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 87380 4194304
net.ipv4.tcp_max_tw_buckets = 262144
net.ipv4.tcp_max_syn_backlog = 1024
vm.oom_kill_allocating_task = 1
vm.max_map_count = 200000
vm.swappiness = 1
vm.dirty_writeback_centisecs = 500
vm.dirty_expire_centisecs = 500
vm.dirty_ratio = 60
vm.dirty_background_ratio = 5
Thanks,
Eric
--
Alexander Binzberger
System Designer - WINGcon AG
Tel. +49 7543 966-119
Sitz der Gesellschaft: Langenargen
Registergericht: ULM, HRB 734260
USt-Id.: DE232931635, WEEE-Id.: DE74015979
Vorstand: thomasThomas Ehrle (Vorsitz), Fritz R. Paul (Stellvertreter), Tobias
Treß
Aufsichtsrat: Jürgen Maucher (Vorsitz), Andreas Paul (Stellvertreter), Martin
Sauter
