Hello users. I test max throughput to a single kafka broker but results do not match speed limits disk or network.
/kafka-producer-perf-test.sh --topic perf-test --num-records 3000 --record-size 1000000 --throughput -1 --producer-props bootstrap.servers=localhost:9092 3000 records sent, 98.872850 records/sec (94.29 MB/sec), 13.51 ms avg latency, 528.00 ms max latency, 9 ms 50th, 15 ms 95th, 162 ms 99th, 350 ms 99.9th. Smaller record size gets lower throughput. Is upper limit throughput for broker or something another limit throughput? What is bottleneck for this? There are detailed information: hardware: CPU: 6230R 2.1GHz 2x26core, RAM: 192 GB, disks SATA SSD – 20 x1,92T (test run on single disk) Java version: java-11-openjdk-11.0.17.0.8-2.el8_6.x86_64 Kafka version: ./kafka-topics.sh --version 3.3.1 (Commit:e23c59d00e687ff5) kafka_2.13-3.3.1.jar Network speed tested by iperf3: 5,46 GB/s (testing kafka both localhost and second node) Disk speed: hdparm -tT /dev/mapper/vg1-kafka /dev/mapper/vg1-kafka: Timing cached reads: 19176 MB in 2.00 seconds = 9600.40 MB/sec Timing buffered disk reads: 10338 MB in 3.00 seconds = 3445.37 MB/sec writes: dd if=/dev/zero of=/kafka/test1.img bs=1G count=1 oflag=dsync 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.13398 s, 947 MB/s Testing both on default jvm settings and with 6GB RAM: -Xmx6g -Xms6g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:+ExplicitGCInvokesConcurrent Kafka installed and ran from instructions https://kafka.apache.org/quickstart on a single node. Pozdrawiam/Regards Tomasz Sąciński -- <https://linuxpolska.pl/> Linux Polska Sp. z o.o. Al. Jerozolimskie 100, 00-807 Warszawa <https://www.google.com/maps/place/Linux+Polska+Sp.+z+o.+o./@52.221986,20.9744718,17z/data=!3m1!4b1!4m5!3m4!1s0x0:0x5756147639e870d6!8m2!3d52.221986!4d20.9766605?authuser=2> tel. +48 22 213 95 71, fax +48 22 213 96 71 KRS 00000326158, Sąd Rejonowy dla M. St. Warszawy w Warszawie, XII Wydział Gospodarczy KRS Kapitał zakładowy 1 000 500 PLN wpłacony w całości, NIP 7010181018, REGON 141791601 www.linuxpolska.pl <https://linuxpolska.pl/> | <https://www.linkedin.com/company/linux-polska/> <https://www.facebook.com/linuxpolskapl> <https://twitter.com/linuxpolska> _________________________________________________________________________ This message may contain confidential information that is covered by legal privilege. If you are not the intended recipient or if you have received this message by mistake, please notify the sender immediately and delete this e-mail and its attachments from your system. Any unauthorized copying, disclosure or distribution of the material in this e-mail and its attachments is strictly forbidden. -- <https://linuxpolska.pl/baner-lp/>
