[jira] [Updated] (KAFKA-15264) Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
[
https://issues.apache.org/jira/browse/KAFKA-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jianbin.chen updated KAFKA-15264:
-
Attachment: server.3.log
> Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
> ---
>
> Key: KAFKA-15264
> URL: https://issues.apache.org/jira/browse/KAFKA-15264
> Project: Kafka
> Issue Type: Bug
>Reporter: jianbin.chen
>Priority: Major
> Attachments: image-2023-07-28-09-51-01-662.png,
> image-2023-07-28-09-52-38-941.png, image-2023-07-31-18-04-54-112.png,
> image-2023-07-31-18-05-21-772.png, image-2023-08-01-10-24-10-227.png,
> image-2023-08-01-10-26-07-146.png, server.1.log, server.2.log, server.3.log
>
>
> I was preparing to upgrade from 1.1.0 to 3.5.1 kraft mode (new cluster
> deployment), and when I recently compared and tested, I found that when using
> the following stress test command, the throughput gap is obvious
>
> {code:java}
> ./kafka-producer-perf-test.sh --topic test321 --num-records 3000
> --record-size 1024 --throughput -1 --producer-props
> bootstrap.servers=xxx: acks=1
> 419813 records sent, 83962.6 records/sec (81.99 MB/sec), 241.1 ms avg
> latency, 588.0 ms max latency.
> 555300 records sent, 111015.6 records/sec (108.41 MB/sec), 275.1 ms avg
> latency, 460.0 ms max latency.
> 552795 records sent, 110536.9 records/sec (107.95 MB/sec), 265.9 ms avg
> latency, 1120.0 ms max latency.
> 552600 records sent, 110520.0 records/sec (107.93 MB/sec), 284.5 ms avg
> latency, 1097.0 ms max latency.
> 538500 records sent, 107656.9 records/sec (105.13 MB/sec), 277.5 ms avg
> latency, 610.0 ms max latency.
> 511545 records sent, 102309.0 records/sec (99.91 MB/sec), 304.1 ms avg
> latency, 1892.0 ms max latency.
> 511890 records sent, 102337.1 records/sec (99.94 MB/sec), 288.4 ms avg
> latency, 3000.0 ms max latency.
> 519165 records sent, 103812.2 records/sec (101.38 MB/sec), 262.1 ms avg
> latency, 1781.0 ms max latency.
> 513555 records sent, 102669.9 records/sec (100.26 MB/sec), 338.2 ms avg
> latency, 2590.0 ms max latency.
> 463329 records sent, 92665.8 records/sec (90.49 MB/sec), 276.8 ms avg
> latency, 1463.0 ms max latency.
> 494248 records sent, 98849.6 records/sec (96.53 MB/sec), 327.2 ms avg
> latency, 2362.0 ms max latency.
> 506272 records sent, 101254.4 records/sec (98.88 MB/sec), 322.1 ms avg
> latency, 2986.0 ms max latency.
> 393758 records sent, 78735.9 records/sec (76.89 MB/sec), 387.0 ms avg
> latency, 2958.0 ms max latency.
> 426435 records sent, 85252.9 records/sec (83.25 MB/sec), 363.3 ms avg
> latency, 1959.0 ms max latency.
> 412560 records sent, 82298.0 records/sec (80.37 MB/sec), 374.1 ms avg
> latency, 1995.0 ms max latency.
> 370137 records sent, 73997.8 records/sec (72.26 MB/sec), 396.8 ms avg
> latency, 1496.0 ms max latency.
> 391781 records sent, 78340.5 records/sec (76.50 MB/sec), 410.7 ms avg
> latency, 2446.0 ms max latency.
> 355901 records sent, 71166.0 records/sec (69.50 MB/sec), 397.5 ms avg
> latency, 2715.0 ms max latency.
> 385410 records sent, 77082.0 records/sec (75.28 MB/sec), 417.5 ms avg
> latency, 2702.0 ms max latency.
> 381160 records sent, 76232.0 records/sec (74.45 MB/sec), 407.7 ms avg
> latency, 1846.0 ms max latency.
> 67 records sent, 0.1 records/sec (65.10 MB/sec), 456.2 ms avg
> latency, 1414.0 ms max latency.
> 376251 records sent, 75175.0 records/sec (73.41 MB/sec), 401.9 ms avg
> latency, 1897.0 ms max latency.
> 354434 records sent, 70886.8 records/sec (69.23 MB/sec), 425.8 ms avg
> latency, 1601.0 ms max latency.
> 353795 records sent, 70744.9 records/sec (69.09 MB/sec), 411.7 ms avg
> latency, 1563.0 ms max latency.
> 321993 records sent, 64360.0 records/sec (62.85 MB/sec), 447.3 ms avg
> latency, 1975.0 ms max latency.
> 404075 records sent, 80750.4 records/sec (78.86 MB/sec), 408.4 ms avg
> latency, 1753.0 ms max latency.
> 384526 records sent, 76905.2 records/sec (75.10 MB/sec), 406.0 ms avg
> latency, 1833.0 ms max latency.
> 387652 records sent, 77483.9 records/sec (75.67 MB/sec), 397.3 ms avg
> latency, 1927.0 ms max latency.
> 343286 records sent, 68629.7 records/sec (67.02 MB/sec), 455.6 ms avg
> latency, 1685.0 ms max latency.
> 00 records sent, 66646.7 records/sec (65.08 MB/sec), 456.6 ms avg
> latency, 2146.0 ms max latency.
> 361191 records sent, 72238.2 records/sec (70.55 MB/sec), 409.4 ms avg
> latency, 2125.0 ms max latency.
> 357525 records sent, 71490.7 records/sec (69.82 MB/sec), 436.0 ms avg
> latency, 1502.0 ms max latency.
> 340238 records sent, 68047.6 records/sec (66.45 MB/sec), 427.9 ms avg
> latency, 1932.0 ms max latency.
> 390016 records sent, 77956.4 records/sec (76.13 MB/sec), 418.5 ms avg
> latency, 1807.0 ms max latency.
> 352830 records sent, 70523.7 records/sec (68.87
[jira] [Updated] (KAFKA-15264) Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
[
https://issues.apache.org/jira/browse/KAFKA-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jianbin.chen updated KAFKA-15264:
-
Attachment: server.1.log
> Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
> ---
>
> Key: KAFKA-15264
> URL: https://issues.apache.org/jira/browse/KAFKA-15264
> Project: Kafka
> Issue Type: Bug
>Reporter: jianbin.chen
>Priority: Major
> Attachments: image-2023-07-28-09-51-01-662.png,
> image-2023-07-28-09-52-38-941.png, image-2023-07-31-18-04-54-112.png,
> image-2023-07-31-18-05-21-772.png, image-2023-08-01-10-24-10-227.png,
> image-2023-08-01-10-26-07-146.png, server.1.log, server.2.log, server.3.log
>
>
> I was preparing to upgrade from 1.1.0 to 3.5.1 kraft mode (new cluster
> deployment), and when I recently compared and tested, I found that when using
> the following stress test command, the throughput gap is obvious
>
> {code:java}
> ./kafka-producer-perf-test.sh --topic test321 --num-records 3000
> --record-size 1024 --throughput -1 --producer-props
> bootstrap.servers=xxx: acks=1
> 419813 records sent, 83962.6 records/sec (81.99 MB/sec), 241.1 ms avg
> latency, 588.0 ms max latency.
> 555300 records sent, 111015.6 records/sec (108.41 MB/sec), 275.1 ms avg
> latency, 460.0 ms max latency.
> 552795 records sent, 110536.9 records/sec (107.95 MB/sec), 265.9 ms avg
> latency, 1120.0 ms max latency.
> 552600 records sent, 110520.0 records/sec (107.93 MB/sec), 284.5 ms avg
> latency, 1097.0 ms max latency.
> 538500 records sent, 107656.9 records/sec (105.13 MB/sec), 277.5 ms avg
> latency, 610.0 ms max latency.
> 511545 records sent, 102309.0 records/sec (99.91 MB/sec), 304.1 ms avg
> latency, 1892.0 ms max latency.
> 511890 records sent, 102337.1 records/sec (99.94 MB/sec), 288.4 ms avg
> latency, 3000.0 ms max latency.
> 519165 records sent, 103812.2 records/sec (101.38 MB/sec), 262.1 ms avg
> latency, 1781.0 ms max latency.
> 513555 records sent, 102669.9 records/sec (100.26 MB/sec), 338.2 ms avg
> latency, 2590.0 ms max latency.
> 463329 records sent, 92665.8 records/sec (90.49 MB/sec), 276.8 ms avg
> latency, 1463.0 ms max latency.
> 494248 records sent, 98849.6 records/sec (96.53 MB/sec), 327.2 ms avg
> latency, 2362.0 ms max latency.
> 506272 records sent, 101254.4 records/sec (98.88 MB/sec), 322.1 ms avg
> latency, 2986.0 ms max latency.
> 393758 records sent, 78735.9 records/sec (76.89 MB/sec), 387.0 ms avg
> latency, 2958.0 ms max latency.
> 426435 records sent, 85252.9 records/sec (83.25 MB/sec), 363.3 ms avg
> latency, 1959.0 ms max latency.
> 412560 records sent, 82298.0 records/sec (80.37 MB/sec), 374.1 ms avg
> latency, 1995.0 ms max latency.
> 370137 records sent, 73997.8 records/sec (72.26 MB/sec), 396.8 ms avg
> latency, 1496.0 ms max latency.
> 391781 records sent, 78340.5 records/sec (76.50 MB/sec), 410.7 ms avg
> latency, 2446.0 ms max latency.
> 355901 records sent, 71166.0 records/sec (69.50 MB/sec), 397.5 ms avg
> latency, 2715.0 ms max latency.
> 385410 records sent, 77082.0 records/sec (75.28 MB/sec), 417.5 ms avg
> latency, 2702.0 ms max latency.
> 381160 records sent, 76232.0 records/sec (74.45 MB/sec), 407.7 ms avg
> latency, 1846.0 ms max latency.
> 67 records sent, 0.1 records/sec (65.10 MB/sec), 456.2 ms avg
> latency, 1414.0 ms max latency.
> 376251 records sent, 75175.0 records/sec (73.41 MB/sec), 401.9 ms avg
> latency, 1897.0 ms max latency.
> 354434 records sent, 70886.8 records/sec (69.23 MB/sec), 425.8 ms avg
> latency, 1601.0 ms max latency.
> 353795 records sent, 70744.9 records/sec (69.09 MB/sec), 411.7 ms avg
> latency, 1563.0 ms max latency.
> 321993 records sent, 64360.0 records/sec (62.85 MB/sec), 447.3 ms avg
> latency, 1975.0 ms max latency.
> 404075 records sent, 80750.4 records/sec (78.86 MB/sec), 408.4 ms avg
> latency, 1753.0 ms max latency.
> 384526 records sent, 76905.2 records/sec (75.10 MB/sec), 406.0 ms avg
> latency, 1833.0 ms max latency.
> 387652 records sent, 77483.9 records/sec (75.67 MB/sec), 397.3 ms avg
> latency, 1927.0 ms max latency.
> 343286 records sent, 68629.7 records/sec (67.02 MB/sec), 455.6 ms avg
> latency, 1685.0 ms max latency.
> 00 records sent, 66646.7 records/sec (65.08 MB/sec), 456.6 ms avg
> latency, 2146.0 ms max latency.
> 361191 records sent, 72238.2 records/sec (70.55 MB/sec), 409.4 ms avg
> latency, 2125.0 ms max latency.
> 357525 records sent, 71490.7 records/sec (69.82 MB/sec), 436.0 ms avg
> latency, 1502.0 ms max latency.
> 340238 records sent, 68047.6 records/sec (66.45 MB/sec), 427.9 ms avg
> latency, 1932.0 ms max latency.
> 390016 records sent, 77956.4 records/sec (76.13 MB/sec), 418.5 ms avg
> latency, 1807.0 ms max latency.
> 352830 records sent, 70523.7 records/sec (68.87
[jira] [Updated] (KAFKA-15264) Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
[
https://issues.apache.org/jira/browse/KAFKA-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jianbin.chen updated KAFKA-15264:
-
Attachment: server.2.log
> Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
> ---
>
> Key: KAFKA-15264
> URL: https://issues.apache.org/jira/browse/KAFKA-15264
> Project: Kafka
> Issue Type: Bug
>Reporter: jianbin.chen
>Priority: Major
> Attachments: image-2023-07-28-09-51-01-662.png,
> image-2023-07-28-09-52-38-941.png, image-2023-07-31-18-04-54-112.png,
> image-2023-07-31-18-05-21-772.png, image-2023-08-01-10-24-10-227.png,
> image-2023-08-01-10-26-07-146.png, server.1.log, server.2.log, server.3.log
>
>
> I was preparing to upgrade from 1.1.0 to 3.5.1 kraft mode (new cluster
> deployment), and when I recently compared and tested, I found that when using
> the following stress test command, the throughput gap is obvious
>
> {code:java}
> ./kafka-producer-perf-test.sh --topic test321 --num-records 3000
> --record-size 1024 --throughput -1 --producer-props
> bootstrap.servers=xxx: acks=1
> 419813 records sent, 83962.6 records/sec (81.99 MB/sec), 241.1 ms avg
> latency, 588.0 ms max latency.
> 555300 records sent, 111015.6 records/sec (108.41 MB/sec), 275.1 ms avg
> latency, 460.0 ms max latency.
> 552795 records sent, 110536.9 records/sec (107.95 MB/sec), 265.9 ms avg
> latency, 1120.0 ms max latency.
> 552600 records sent, 110520.0 records/sec (107.93 MB/sec), 284.5 ms avg
> latency, 1097.0 ms max latency.
> 538500 records sent, 107656.9 records/sec (105.13 MB/sec), 277.5 ms avg
> latency, 610.0 ms max latency.
> 511545 records sent, 102309.0 records/sec (99.91 MB/sec), 304.1 ms avg
> latency, 1892.0 ms max latency.
> 511890 records sent, 102337.1 records/sec (99.94 MB/sec), 288.4 ms avg
> latency, 3000.0 ms max latency.
> 519165 records sent, 103812.2 records/sec (101.38 MB/sec), 262.1 ms avg
> latency, 1781.0 ms max latency.
> 513555 records sent, 102669.9 records/sec (100.26 MB/sec), 338.2 ms avg
> latency, 2590.0 ms max latency.
> 463329 records sent, 92665.8 records/sec (90.49 MB/sec), 276.8 ms avg
> latency, 1463.0 ms max latency.
> 494248 records sent, 98849.6 records/sec (96.53 MB/sec), 327.2 ms avg
> latency, 2362.0 ms max latency.
> 506272 records sent, 101254.4 records/sec (98.88 MB/sec), 322.1 ms avg
> latency, 2986.0 ms max latency.
> 393758 records sent, 78735.9 records/sec (76.89 MB/sec), 387.0 ms avg
> latency, 2958.0 ms max latency.
> 426435 records sent, 85252.9 records/sec (83.25 MB/sec), 363.3 ms avg
> latency, 1959.0 ms max latency.
> 412560 records sent, 82298.0 records/sec (80.37 MB/sec), 374.1 ms avg
> latency, 1995.0 ms max latency.
> 370137 records sent, 73997.8 records/sec (72.26 MB/sec), 396.8 ms avg
> latency, 1496.0 ms max latency.
> 391781 records sent, 78340.5 records/sec (76.50 MB/sec), 410.7 ms avg
> latency, 2446.0 ms max latency.
> 355901 records sent, 71166.0 records/sec (69.50 MB/sec), 397.5 ms avg
> latency, 2715.0 ms max latency.
> 385410 records sent, 77082.0 records/sec (75.28 MB/sec), 417.5 ms avg
> latency, 2702.0 ms max latency.
> 381160 records sent, 76232.0 records/sec (74.45 MB/sec), 407.7 ms avg
> latency, 1846.0 ms max latency.
> 67 records sent, 0.1 records/sec (65.10 MB/sec), 456.2 ms avg
> latency, 1414.0 ms max latency.
> 376251 records sent, 75175.0 records/sec (73.41 MB/sec), 401.9 ms avg
> latency, 1897.0 ms max latency.
> 354434 records sent, 70886.8 records/sec (69.23 MB/sec), 425.8 ms avg
> latency, 1601.0 ms max latency.
> 353795 records sent, 70744.9 records/sec (69.09 MB/sec), 411.7 ms avg
> latency, 1563.0 ms max latency.
> 321993 records sent, 64360.0 records/sec (62.85 MB/sec), 447.3 ms avg
> latency, 1975.0 ms max latency.
> 404075 records sent, 80750.4 records/sec (78.86 MB/sec), 408.4 ms avg
> latency, 1753.0 ms max latency.
> 384526 records sent, 76905.2 records/sec (75.10 MB/sec), 406.0 ms avg
> latency, 1833.0 ms max latency.
> 387652 records sent, 77483.9 records/sec (75.67 MB/sec), 397.3 ms avg
> latency, 1927.0 ms max latency.
> 343286 records sent, 68629.7 records/sec (67.02 MB/sec), 455.6 ms avg
> latency, 1685.0 ms max latency.
> 00 records sent, 66646.7 records/sec (65.08 MB/sec), 456.6 ms avg
> latency, 2146.0 ms max latency.
> 361191 records sent, 72238.2 records/sec (70.55 MB/sec), 409.4 ms avg
> latency, 2125.0 ms max latency.
> 357525 records sent, 71490.7 records/sec (69.82 MB/sec), 436.0 ms avg
> latency, 1502.0 ms max latency.
> 340238 records sent, 68047.6 records/sec (66.45 MB/sec), 427.9 ms avg
> latency, 1932.0 ms max latency.
> 390016 records sent, 77956.4 records/sec (76.13 MB/sec), 418.5 ms avg
> latency, 1807.0 ms max latency.
> 352830 records sent, 70523.7 records/sec (68.87
[jira] [Updated] (KAFKA-15264) Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
[
https://issues.apache.org/jira/browse/KAFKA-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jianbin.chen updated KAFKA-15264:
-
Attachment: image-2023-08-01-10-26-07-146.png
> Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
> ---
>
> Key: KAFKA-15264
> URL: https://issues.apache.org/jira/browse/KAFKA-15264
> Project: Kafka
> Issue Type: Bug
>Reporter: jianbin.chen
>Priority: Major
> Attachments: image-2023-07-28-09-51-01-662.png,
> image-2023-07-28-09-52-38-941.png, image-2023-07-31-18-04-54-112.png,
> image-2023-07-31-18-05-21-772.png, image-2023-08-01-10-24-10-227.png,
> image-2023-08-01-10-26-07-146.png
>
>
> I was preparing to upgrade from 1.1.0 to 3.5.1 kraft mode (new cluster
> deployment), and when I recently compared and tested, I found that when using
> the following stress test command, the throughput gap is obvious
>
> {code:java}
> ./kafka-producer-perf-test.sh --topic test321 --num-records 3000
> --record-size 1024 --throughput -1 --producer-props
> bootstrap.servers=xxx: acks=1
> 419813 records sent, 83962.6 records/sec (81.99 MB/sec), 241.1 ms avg
> latency, 588.0 ms max latency.
> 555300 records sent, 111015.6 records/sec (108.41 MB/sec), 275.1 ms avg
> latency, 460.0 ms max latency.
> 552795 records sent, 110536.9 records/sec (107.95 MB/sec), 265.9 ms avg
> latency, 1120.0 ms max latency.
> 552600 records sent, 110520.0 records/sec (107.93 MB/sec), 284.5 ms avg
> latency, 1097.0 ms max latency.
> 538500 records sent, 107656.9 records/sec (105.13 MB/sec), 277.5 ms avg
> latency, 610.0 ms max latency.
> 511545 records sent, 102309.0 records/sec (99.91 MB/sec), 304.1 ms avg
> latency, 1892.0 ms max latency.
> 511890 records sent, 102337.1 records/sec (99.94 MB/sec), 288.4 ms avg
> latency, 3000.0 ms max latency.
> 519165 records sent, 103812.2 records/sec (101.38 MB/sec), 262.1 ms avg
> latency, 1781.0 ms max latency.
> 513555 records sent, 102669.9 records/sec (100.26 MB/sec), 338.2 ms avg
> latency, 2590.0 ms max latency.
> 463329 records sent, 92665.8 records/sec (90.49 MB/sec), 276.8 ms avg
> latency, 1463.0 ms max latency.
> 494248 records sent, 98849.6 records/sec (96.53 MB/sec), 327.2 ms avg
> latency, 2362.0 ms max latency.
> 506272 records sent, 101254.4 records/sec (98.88 MB/sec), 322.1 ms avg
> latency, 2986.0 ms max latency.
> 393758 records sent, 78735.9 records/sec (76.89 MB/sec), 387.0 ms avg
> latency, 2958.0 ms max latency.
> 426435 records sent, 85252.9 records/sec (83.25 MB/sec), 363.3 ms avg
> latency, 1959.0 ms max latency.
> 412560 records sent, 82298.0 records/sec (80.37 MB/sec), 374.1 ms avg
> latency, 1995.0 ms max latency.
> 370137 records sent, 73997.8 records/sec (72.26 MB/sec), 396.8 ms avg
> latency, 1496.0 ms max latency.
> 391781 records sent, 78340.5 records/sec (76.50 MB/sec), 410.7 ms avg
> latency, 2446.0 ms max latency.
> 355901 records sent, 71166.0 records/sec (69.50 MB/sec), 397.5 ms avg
> latency, 2715.0 ms max latency.
> 385410 records sent, 77082.0 records/sec (75.28 MB/sec), 417.5 ms avg
> latency, 2702.0 ms max latency.
> 381160 records sent, 76232.0 records/sec (74.45 MB/sec), 407.7 ms avg
> latency, 1846.0 ms max latency.
> 67 records sent, 0.1 records/sec (65.10 MB/sec), 456.2 ms avg
> latency, 1414.0 ms max latency.
> 376251 records sent, 75175.0 records/sec (73.41 MB/sec), 401.9 ms avg
> latency, 1897.0 ms max latency.
> 354434 records sent, 70886.8 records/sec (69.23 MB/sec), 425.8 ms avg
> latency, 1601.0 ms max latency.
> 353795 records sent, 70744.9 records/sec (69.09 MB/sec), 411.7 ms avg
> latency, 1563.0 ms max latency.
> 321993 records sent, 64360.0 records/sec (62.85 MB/sec), 447.3 ms avg
> latency, 1975.0 ms max latency.
> 404075 records sent, 80750.4 records/sec (78.86 MB/sec), 408.4 ms avg
> latency, 1753.0 ms max latency.
> 384526 records sent, 76905.2 records/sec (75.10 MB/sec), 406.0 ms avg
> latency, 1833.0 ms max latency.
> 387652 records sent, 77483.9 records/sec (75.67 MB/sec), 397.3 ms avg
> latency, 1927.0 ms max latency.
> 343286 records sent, 68629.7 records/sec (67.02 MB/sec), 455.6 ms avg
> latency, 1685.0 ms max latency.
> 00 records sent, 66646.7 records/sec (65.08 MB/sec), 456.6 ms avg
> latency, 2146.0 ms max latency.
> 361191 records sent, 72238.2 records/sec (70.55 MB/sec), 409.4 ms avg
> latency, 2125.0 ms max latency.
> 357525 records sent, 71490.7 records/sec (69.82 MB/sec), 436.0 ms avg
> latency, 1502.0 ms max latency.
> 340238 records sent, 68047.6 records/sec (66.45 MB/sec), 427.9 ms avg
> latency, 1932.0 ms max latency.
> 390016 records sent, 77956.4 records/sec (76.13 MB/sec), 418.5 ms avg
> latency, 1807.0 ms max latency.
> 352830 records sent, 70523.7 records/sec (68.87 MB/sec), 439.4 ms avg
[jira] [Updated] (KAFKA-15264) Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
[
https://issues.apache.org/jira/browse/KAFKA-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jianbin.chen updated KAFKA-15264:
-
Attachment: image-2023-08-01-10-24-10-227.png
> Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
> ---
>
> Key: KAFKA-15264
> URL: https://issues.apache.org/jira/browse/KAFKA-15264
> Project: Kafka
> Issue Type: Bug
>Reporter: jianbin.chen
>Priority: Major
> Attachments: image-2023-07-28-09-51-01-662.png,
> image-2023-07-28-09-52-38-941.png, image-2023-07-31-18-04-54-112.png,
> image-2023-07-31-18-05-21-772.png, image-2023-08-01-10-24-10-227.png
>
>
> I was preparing to upgrade from 1.1.0 to 3.5.1 kraft mode (new cluster
> deployment), and when I recently compared and tested, I found that when using
> the following stress test command, the throughput gap is obvious
>
> {code:java}
> ./kafka-producer-perf-test.sh --topic test321 --num-records 3000
> --record-size 1024 --throughput -1 --producer-props
> bootstrap.servers=xxx: acks=1
> 419813 records sent, 83962.6 records/sec (81.99 MB/sec), 241.1 ms avg
> latency, 588.0 ms max latency.
> 555300 records sent, 111015.6 records/sec (108.41 MB/sec), 275.1 ms avg
> latency, 460.0 ms max latency.
> 552795 records sent, 110536.9 records/sec (107.95 MB/sec), 265.9 ms avg
> latency, 1120.0 ms max latency.
> 552600 records sent, 110520.0 records/sec (107.93 MB/sec), 284.5 ms avg
> latency, 1097.0 ms max latency.
> 538500 records sent, 107656.9 records/sec (105.13 MB/sec), 277.5 ms avg
> latency, 610.0 ms max latency.
> 511545 records sent, 102309.0 records/sec (99.91 MB/sec), 304.1 ms avg
> latency, 1892.0 ms max latency.
> 511890 records sent, 102337.1 records/sec (99.94 MB/sec), 288.4 ms avg
> latency, 3000.0 ms max latency.
> 519165 records sent, 103812.2 records/sec (101.38 MB/sec), 262.1 ms avg
> latency, 1781.0 ms max latency.
> 513555 records sent, 102669.9 records/sec (100.26 MB/sec), 338.2 ms avg
> latency, 2590.0 ms max latency.
> 463329 records sent, 92665.8 records/sec (90.49 MB/sec), 276.8 ms avg
> latency, 1463.0 ms max latency.
> 494248 records sent, 98849.6 records/sec (96.53 MB/sec), 327.2 ms avg
> latency, 2362.0 ms max latency.
> 506272 records sent, 101254.4 records/sec (98.88 MB/sec), 322.1 ms avg
> latency, 2986.0 ms max latency.
> 393758 records sent, 78735.9 records/sec (76.89 MB/sec), 387.0 ms avg
> latency, 2958.0 ms max latency.
> 426435 records sent, 85252.9 records/sec (83.25 MB/sec), 363.3 ms avg
> latency, 1959.0 ms max latency.
> 412560 records sent, 82298.0 records/sec (80.37 MB/sec), 374.1 ms avg
> latency, 1995.0 ms max latency.
> 370137 records sent, 73997.8 records/sec (72.26 MB/sec), 396.8 ms avg
> latency, 1496.0 ms max latency.
> 391781 records sent, 78340.5 records/sec (76.50 MB/sec), 410.7 ms avg
> latency, 2446.0 ms max latency.
> 355901 records sent, 71166.0 records/sec (69.50 MB/sec), 397.5 ms avg
> latency, 2715.0 ms max latency.
> 385410 records sent, 77082.0 records/sec (75.28 MB/sec), 417.5 ms avg
> latency, 2702.0 ms max latency.
> 381160 records sent, 76232.0 records/sec (74.45 MB/sec), 407.7 ms avg
> latency, 1846.0 ms max latency.
> 67 records sent, 0.1 records/sec (65.10 MB/sec), 456.2 ms avg
> latency, 1414.0 ms max latency.
> 376251 records sent, 75175.0 records/sec (73.41 MB/sec), 401.9 ms avg
> latency, 1897.0 ms max latency.
> 354434 records sent, 70886.8 records/sec (69.23 MB/sec), 425.8 ms avg
> latency, 1601.0 ms max latency.
> 353795 records sent, 70744.9 records/sec (69.09 MB/sec), 411.7 ms avg
> latency, 1563.0 ms max latency.
> 321993 records sent, 64360.0 records/sec (62.85 MB/sec), 447.3 ms avg
> latency, 1975.0 ms max latency.
> 404075 records sent, 80750.4 records/sec (78.86 MB/sec), 408.4 ms avg
> latency, 1753.0 ms max latency.
> 384526 records sent, 76905.2 records/sec (75.10 MB/sec), 406.0 ms avg
> latency, 1833.0 ms max latency.
> 387652 records sent, 77483.9 records/sec (75.67 MB/sec), 397.3 ms avg
> latency, 1927.0 ms max latency.
> 343286 records sent, 68629.7 records/sec (67.02 MB/sec), 455.6 ms avg
> latency, 1685.0 ms max latency.
> 00 records sent, 66646.7 records/sec (65.08 MB/sec), 456.6 ms avg
> latency, 2146.0 ms max latency.
> 361191 records sent, 72238.2 records/sec (70.55 MB/sec), 409.4 ms avg
> latency, 2125.0 ms max latency.
> 357525 records sent, 71490.7 records/sec (69.82 MB/sec), 436.0 ms avg
> latency, 1502.0 ms max latency.
> 340238 records sent, 68047.6 records/sec (66.45 MB/sec), 427.9 ms avg
> latency, 1932.0 ms max latency.
> 390016 records sent, 77956.4 records/sec (76.13 MB/sec), 418.5 ms avg
> latency, 1807.0 ms max latency.
> 352830 records sent, 70523.7 records/sec (68.87 MB/sec), 439.4 ms avg
> latency, 1892.0 ms max latency.
>
[jira] [Updated] (KAFKA-15264) Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
[
https://issues.apache.org/jira/browse/KAFKA-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jianbin.chen updated KAFKA-15264:
-
Attachment: image-2023-07-31-18-05-21-772.png
> Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
> ---
>
> Key: KAFKA-15264
> URL: https://issues.apache.org/jira/browse/KAFKA-15264
> Project: Kafka
> Issue Type: Bug
>Reporter: jianbin.chen
>Priority: Major
> Attachments: image-2023-07-28-09-51-01-662.png,
> image-2023-07-28-09-52-38-941.png, image-2023-07-31-18-04-54-112.png,
> image-2023-07-31-18-05-21-772.png
>
>
> I was preparing to upgrade from 1.1.0 to 3.5.1 kraft mode (new cluster
> deployment), and when I recently compared and tested, I found that when using
> the following stress test command, the throughput gap is obvious
>
> {code:java}
> ./kafka-producer-perf-test.sh --topic test321 --num-records 3000
> --record-size 1024 --throughput -1 --producer-props
> bootstrap.servers=xxx: acks=1
> 419813 records sent, 83962.6 records/sec (81.99 MB/sec), 241.1 ms avg
> latency, 588.0 ms max latency.
> 555300 records sent, 111015.6 records/sec (108.41 MB/sec), 275.1 ms avg
> latency, 460.0 ms max latency.
> 552795 records sent, 110536.9 records/sec (107.95 MB/sec), 265.9 ms avg
> latency, 1120.0 ms max latency.
> 552600 records sent, 110520.0 records/sec (107.93 MB/sec), 284.5 ms avg
> latency, 1097.0 ms max latency.
> 538500 records sent, 107656.9 records/sec (105.13 MB/sec), 277.5 ms avg
> latency, 610.0 ms max latency.
> 511545 records sent, 102309.0 records/sec (99.91 MB/sec), 304.1 ms avg
> latency, 1892.0 ms max latency.
> 511890 records sent, 102337.1 records/sec (99.94 MB/sec), 288.4 ms avg
> latency, 3000.0 ms max latency.
> 519165 records sent, 103812.2 records/sec (101.38 MB/sec), 262.1 ms avg
> latency, 1781.0 ms max latency.
> 513555 records sent, 102669.9 records/sec (100.26 MB/sec), 338.2 ms avg
> latency, 2590.0 ms max latency.
> 463329 records sent, 92665.8 records/sec (90.49 MB/sec), 276.8 ms avg
> latency, 1463.0 ms max latency.
> 494248 records sent, 98849.6 records/sec (96.53 MB/sec), 327.2 ms avg
> latency, 2362.0 ms max latency.
> 506272 records sent, 101254.4 records/sec (98.88 MB/sec), 322.1 ms avg
> latency, 2986.0 ms max latency.
> 393758 records sent, 78735.9 records/sec (76.89 MB/sec), 387.0 ms avg
> latency, 2958.0 ms max latency.
> 426435 records sent, 85252.9 records/sec (83.25 MB/sec), 363.3 ms avg
> latency, 1959.0 ms max latency.
> 412560 records sent, 82298.0 records/sec (80.37 MB/sec), 374.1 ms avg
> latency, 1995.0 ms max latency.
> 370137 records sent, 73997.8 records/sec (72.26 MB/sec), 396.8 ms avg
> latency, 1496.0 ms max latency.
> 391781 records sent, 78340.5 records/sec (76.50 MB/sec), 410.7 ms avg
> latency, 2446.0 ms max latency.
> 355901 records sent, 71166.0 records/sec (69.50 MB/sec), 397.5 ms avg
> latency, 2715.0 ms max latency.
> 385410 records sent, 77082.0 records/sec (75.28 MB/sec), 417.5 ms avg
> latency, 2702.0 ms max latency.
> 381160 records sent, 76232.0 records/sec (74.45 MB/sec), 407.7 ms avg
> latency, 1846.0 ms max latency.
> 67 records sent, 0.1 records/sec (65.10 MB/sec), 456.2 ms avg
> latency, 1414.0 ms max latency.
> 376251 records sent, 75175.0 records/sec (73.41 MB/sec), 401.9 ms avg
> latency, 1897.0 ms max latency.
> 354434 records sent, 70886.8 records/sec (69.23 MB/sec), 425.8 ms avg
> latency, 1601.0 ms max latency.
> 353795 records sent, 70744.9 records/sec (69.09 MB/sec), 411.7 ms avg
> latency, 1563.0 ms max latency.
> 321993 records sent, 64360.0 records/sec (62.85 MB/sec), 447.3 ms avg
> latency, 1975.0 ms max latency.
> 404075 records sent, 80750.4 records/sec (78.86 MB/sec), 408.4 ms avg
> latency, 1753.0 ms max latency.
> 384526 records sent, 76905.2 records/sec (75.10 MB/sec), 406.0 ms avg
> latency, 1833.0 ms max latency.
> 387652 records sent, 77483.9 records/sec (75.67 MB/sec), 397.3 ms avg
> latency, 1927.0 ms max latency.
> 343286 records sent, 68629.7 records/sec (67.02 MB/sec), 455.6 ms avg
> latency, 1685.0 ms max latency.
> 00 records sent, 66646.7 records/sec (65.08 MB/sec), 456.6 ms avg
> latency, 2146.0 ms max latency.
> 361191 records sent, 72238.2 records/sec (70.55 MB/sec), 409.4 ms avg
> latency, 2125.0 ms max latency.
> 357525 records sent, 71490.7 records/sec (69.82 MB/sec), 436.0 ms avg
> latency, 1502.0 ms max latency.
> 340238 records sent, 68047.6 records/sec (66.45 MB/sec), 427.9 ms avg
> latency, 1932.0 ms max latency.
> 390016 records sent, 77956.4 records/sec (76.13 MB/sec), 418.5 ms avg
> latency, 1807.0 ms max latency.
> 352830 records sent, 70523.7 records/sec (68.87 MB/sec), 439.4 ms avg
> latency, 1892.0 ms max latency.
> 354526 records sent, 70905.2
[jira] [Updated] (KAFKA-15264) Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
[
https://issues.apache.org/jira/browse/KAFKA-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jianbin.chen updated KAFKA-15264:
-
Attachment: image-2023-07-31-18-04-54-112.png
> Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
> ---
>
> Key: KAFKA-15264
> URL: https://issues.apache.org/jira/browse/KAFKA-15264
> Project: Kafka
> Issue Type: Bug
>Reporter: jianbin.chen
>Priority: Major
> Attachments: image-2023-07-28-09-51-01-662.png,
> image-2023-07-28-09-52-38-941.png, image-2023-07-31-18-04-54-112.png,
> image-2023-07-31-18-05-21-772.png
>
>
> I was preparing to upgrade from 1.1.0 to 3.5.1 kraft mode (new cluster
> deployment), and when I recently compared and tested, I found that when using
> the following stress test command, the throughput gap is obvious
>
> {code:java}
> ./kafka-producer-perf-test.sh --topic test321 --num-records 3000
> --record-size 1024 --throughput -1 --producer-props
> bootstrap.servers=xxx: acks=1
> 419813 records sent, 83962.6 records/sec (81.99 MB/sec), 241.1 ms avg
> latency, 588.0 ms max latency.
> 555300 records sent, 111015.6 records/sec (108.41 MB/sec), 275.1 ms avg
> latency, 460.0 ms max latency.
> 552795 records sent, 110536.9 records/sec (107.95 MB/sec), 265.9 ms avg
> latency, 1120.0 ms max latency.
> 552600 records sent, 110520.0 records/sec (107.93 MB/sec), 284.5 ms avg
> latency, 1097.0 ms max latency.
> 538500 records sent, 107656.9 records/sec (105.13 MB/sec), 277.5 ms avg
> latency, 610.0 ms max latency.
> 511545 records sent, 102309.0 records/sec (99.91 MB/sec), 304.1 ms avg
> latency, 1892.0 ms max latency.
> 511890 records sent, 102337.1 records/sec (99.94 MB/sec), 288.4 ms avg
> latency, 3000.0 ms max latency.
> 519165 records sent, 103812.2 records/sec (101.38 MB/sec), 262.1 ms avg
> latency, 1781.0 ms max latency.
> 513555 records sent, 102669.9 records/sec (100.26 MB/sec), 338.2 ms avg
> latency, 2590.0 ms max latency.
> 463329 records sent, 92665.8 records/sec (90.49 MB/sec), 276.8 ms avg
> latency, 1463.0 ms max latency.
> 494248 records sent, 98849.6 records/sec (96.53 MB/sec), 327.2 ms avg
> latency, 2362.0 ms max latency.
> 506272 records sent, 101254.4 records/sec (98.88 MB/sec), 322.1 ms avg
> latency, 2986.0 ms max latency.
> 393758 records sent, 78735.9 records/sec (76.89 MB/sec), 387.0 ms avg
> latency, 2958.0 ms max latency.
> 426435 records sent, 85252.9 records/sec (83.25 MB/sec), 363.3 ms avg
> latency, 1959.0 ms max latency.
> 412560 records sent, 82298.0 records/sec (80.37 MB/sec), 374.1 ms avg
> latency, 1995.0 ms max latency.
> 370137 records sent, 73997.8 records/sec (72.26 MB/sec), 396.8 ms avg
> latency, 1496.0 ms max latency.
> 391781 records sent, 78340.5 records/sec (76.50 MB/sec), 410.7 ms avg
> latency, 2446.0 ms max latency.
> 355901 records sent, 71166.0 records/sec (69.50 MB/sec), 397.5 ms avg
> latency, 2715.0 ms max latency.
> 385410 records sent, 77082.0 records/sec (75.28 MB/sec), 417.5 ms avg
> latency, 2702.0 ms max latency.
> 381160 records sent, 76232.0 records/sec (74.45 MB/sec), 407.7 ms avg
> latency, 1846.0 ms max latency.
> 67 records sent, 0.1 records/sec (65.10 MB/sec), 456.2 ms avg
> latency, 1414.0 ms max latency.
> 376251 records sent, 75175.0 records/sec (73.41 MB/sec), 401.9 ms avg
> latency, 1897.0 ms max latency.
> 354434 records sent, 70886.8 records/sec (69.23 MB/sec), 425.8 ms avg
> latency, 1601.0 ms max latency.
> 353795 records sent, 70744.9 records/sec (69.09 MB/sec), 411.7 ms avg
> latency, 1563.0 ms max latency.
> 321993 records sent, 64360.0 records/sec (62.85 MB/sec), 447.3 ms avg
> latency, 1975.0 ms max latency.
> 404075 records sent, 80750.4 records/sec (78.86 MB/sec), 408.4 ms avg
> latency, 1753.0 ms max latency.
> 384526 records sent, 76905.2 records/sec (75.10 MB/sec), 406.0 ms avg
> latency, 1833.0 ms max latency.
> 387652 records sent, 77483.9 records/sec (75.67 MB/sec), 397.3 ms avg
> latency, 1927.0 ms max latency.
> 343286 records sent, 68629.7 records/sec (67.02 MB/sec), 455.6 ms avg
> latency, 1685.0 ms max latency.
> 00 records sent, 66646.7 records/sec (65.08 MB/sec), 456.6 ms avg
> latency, 2146.0 ms max latency.
> 361191 records sent, 72238.2 records/sec (70.55 MB/sec), 409.4 ms avg
> latency, 2125.0 ms max latency.
> 357525 records sent, 71490.7 records/sec (69.82 MB/sec), 436.0 ms avg
> latency, 1502.0 ms max latency.
> 340238 records sent, 68047.6 records/sec (66.45 MB/sec), 427.9 ms avg
> latency, 1932.0 ms max latency.
> 390016 records sent, 77956.4 records/sec (76.13 MB/sec), 418.5 ms avg
> latency, 1807.0 ms max latency.
> 352830 records sent, 70523.7 records/sec (68.87 MB/sec), 439.4 ms avg
> latency, 1892.0 ms max latency.
> 354526 records sent, 70905.2
[jira] [Updated] (KAFKA-15264) Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
[
https://issues.apache.org/jira/browse/KAFKA-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jianbin.chen updated KAFKA-15264:
-
Description:
I was preparing to upgrade from 1.1.0 to 3.5.1 kraft mode (new cluster
deployment), and when I recently compared and tested, I found that when using
the following stress test command, the throughput gap is obvious
{code:java}
./kafka-producer-perf-test.sh --topic test321 --num-records 3000
--record-size 1024 --throughput -1 --producer-props bootstrap.servers=xxx:
acks=1
419813 records sent, 83962.6 records/sec (81.99 MB/sec), 241.1 ms avg latency,
588.0 ms max latency.
555300 records sent, 111015.6 records/sec (108.41 MB/sec), 275.1 ms avg
latency, 460.0 ms max latency.
552795 records sent, 110536.9 records/sec (107.95 MB/sec), 265.9 ms avg
latency, 1120.0 ms max latency.
552600 records sent, 110520.0 records/sec (107.93 MB/sec), 284.5 ms avg
latency, 1097.0 ms max latency.
538500 records sent, 107656.9 records/sec (105.13 MB/sec), 277.5 ms avg
latency, 610.0 ms max latency.
511545 records sent, 102309.0 records/sec (99.91 MB/sec), 304.1 ms avg latency,
1892.0 ms max latency.
511890 records sent, 102337.1 records/sec (99.94 MB/sec), 288.4 ms avg latency,
3000.0 ms max latency.
519165 records sent, 103812.2 records/sec (101.38 MB/sec), 262.1 ms avg
latency, 1781.0 ms max latency.
513555 records sent, 102669.9 records/sec (100.26 MB/sec), 338.2 ms avg
latency, 2590.0 ms max latency.
463329 records sent, 92665.8 records/sec (90.49 MB/sec), 276.8 ms avg latency,
1463.0 ms max latency.
494248 records sent, 98849.6 records/sec (96.53 MB/sec), 327.2 ms avg latency,
2362.0 ms max latency.
506272 records sent, 101254.4 records/sec (98.88 MB/sec), 322.1 ms avg latency,
2986.0 ms max latency.
393758 records sent, 78735.9 records/sec (76.89 MB/sec), 387.0 ms avg latency,
2958.0 ms max latency.
426435 records sent, 85252.9 records/sec (83.25 MB/sec), 363.3 ms avg latency,
1959.0 ms max latency.
412560 records sent, 82298.0 records/sec (80.37 MB/sec), 374.1 ms avg latency,
1995.0 ms max latency.
370137 records sent, 73997.8 records/sec (72.26 MB/sec), 396.8 ms avg latency,
1496.0 ms max latency.
391781 records sent, 78340.5 records/sec (76.50 MB/sec), 410.7 ms avg latency,
2446.0 ms max latency.
355901 records sent, 71166.0 records/sec (69.50 MB/sec), 397.5 ms avg latency,
2715.0 ms max latency.
385410 records sent, 77082.0 records/sec (75.28 MB/sec), 417.5 ms avg latency,
2702.0 ms max latency.
381160 records sent, 76232.0 records/sec (74.45 MB/sec), 407.7 ms avg latency,
1846.0 ms max latency.
67 records sent, 0.1 records/sec (65.10 MB/sec), 456.2 ms avg latency,
1414.0 ms max latency.
376251 records sent, 75175.0 records/sec (73.41 MB/sec), 401.9 ms avg latency,
1897.0 ms max latency.
354434 records sent, 70886.8 records/sec (69.23 MB/sec), 425.8 ms avg latency,
1601.0 ms max latency.
353795 records sent, 70744.9 records/sec (69.09 MB/sec), 411.7 ms avg latency,
1563.0 ms max latency.
321993 records sent, 64360.0 records/sec (62.85 MB/sec), 447.3 ms avg latency,
1975.0 ms max latency.
404075 records sent, 80750.4 records/sec (78.86 MB/sec), 408.4 ms avg latency,
1753.0 ms max latency.
384526 records sent, 76905.2 records/sec (75.10 MB/sec), 406.0 ms avg latency,
1833.0 ms max latency.
387652 records sent, 77483.9 records/sec (75.67 MB/sec), 397.3 ms avg latency,
1927.0 ms max latency.
343286 records sent, 68629.7 records/sec (67.02 MB/sec), 455.6 ms avg latency,
1685.0 ms max latency.
00 records sent, 66646.7 records/sec (65.08 MB/sec), 456.6 ms avg latency,
2146.0 ms max latency.
361191 records sent, 72238.2 records/sec (70.55 MB/sec), 409.4 ms avg latency,
2125.0 ms max latency.
357525 records sent, 71490.7 records/sec (69.82 MB/sec), 436.0 ms avg latency,
1502.0 ms max latency.
340238 records sent, 68047.6 records/sec (66.45 MB/sec), 427.9 ms avg latency,
1932.0 ms max latency.
390016 records sent, 77956.4 records/sec (76.13 MB/sec), 418.5 ms avg latency,
1807.0 ms max latency.
352830 records sent, 70523.7 records/sec (68.87 MB/sec), 439.4 ms avg latency,
1892.0 ms max latency.
354526 records sent, 70905.2 records/sec (69.24 MB/sec), 429.6 ms avg latency,
2128.0 ms max latency.
356670 records sent, 71305.5 records/sec (69.63 MB/sec), 408.9 ms avg latency,
1329.0 ms max latency.
309204 records sent, 60687.7 records/sec (59.27 MB/sec), 438.6 ms avg latency,
2566.0 ms max latency.
366715 records sent, 72316.1 records/sec (70.62 MB/sec), 474.5 ms avg latency,
2169.0 ms max latency.
375174 records sent, 75034.8 records/sec (73.28 MB/sec), 429.9 ms avg latency,
1722.0 ms max latency.
359400 records sent, 70346.4 records/sec (68.70 MB/sec), 432.1 ms avg latency,
1961.0 ms max latency.
312276 records sent, 62430.2 records/sec (60.97 MB/sec), 477.4 ms avg latency,
2006.0 ms max latency.
361875 records sent, 72360.5 records/sec (70.66 MB/sec),
[jira] [Updated] (KAFKA-15264) Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
[
https://issues.apache.org/jira/browse/KAFKA-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jianbin.chen updated KAFKA-15264:
-
Attachment: image-2023-07-28-09-52-38-941.png
Description:
I was preparing to upgrade from 1.1.0 to 3.5.1's kraft mode (new cluster
deployment), and when I recently compared and tested, I found that when using
the following stress test command, the throughput gap is obvious
{code:java}
./kafka-producer-perf-test.sh --topic test321 --num-records 3000
--record-size 1024 --throughput -1 --producer-props bootstrap.servers=xxx:
acks=1
419813 records sent, 83962.6 records/sec (81.99 MB/sec), 241.1 ms avg latency,
588.0 ms max latency.
555300 records sent, 111015.6 records/sec (108.41 MB/sec), 275.1 ms avg
latency, 460.0 ms max latency.
552795 records sent, 110536.9 records/sec (107.95 MB/sec), 265.9 ms avg
latency, 1120.0 ms max latency.
552600 records sent, 110520.0 records/sec (107.93 MB/sec), 284.5 ms avg
latency, 1097.0 ms max latency.
538500 records sent, 107656.9 records/sec (105.13 MB/sec), 277.5 ms avg
latency, 610.0 ms max latency.
511545 records sent, 102309.0 records/sec (99.91 MB/sec), 304.1 ms avg latency,
1892.0 ms max latency.
511890 records sent, 102337.1 records/sec (99.94 MB/sec), 288.4 ms avg latency,
3000.0 ms max latency.
519165 records sent, 103812.2 records/sec (101.38 MB/sec), 262.1 ms avg
latency, 1781.0 ms max latency.
513555 records sent, 102669.9 records/sec (100.26 MB/sec), 338.2 ms avg
latency, 2590.0 ms max latency.
463329 records sent, 92665.8 records/sec (90.49 MB/sec), 276.8 ms avg latency,
1463.0 ms max latency.
494248 records sent, 98849.6 records/sec (96.53 MB/sec), 327.2 ms avg latency,
2362.0 ms max latency.
506272 records sent, 101254.4 records/sec (98.88 MB/sec), 322.1 ms avg latency,
2986.0 ms max latency.
393758 records sent, 78735.9 records/sec (76.89 MB/sec), 387.0 ms avg latency,
2958.0 ms max latency.
426435 records sent, 85252.9 records/sec (83.25 MB/sec), 363.3 ms avg latency,
1959.0 ms max latency.
412560 records sent, 82298.0 records/sec (80.37 MB/sec), 374.1 ms avg latency,
1995.0 ms max latency.
370137 records sent, 73997.8 records/sec (72.26 MB/sec), 396.8 ms avg latency,
1496.0 ms max latency.
391781 records sent, 78340.5 records/sec (76.50 MB/sec), 410.7 ms avg latency,
2446.0 ms max latency.
355901 records sent, 71166.0 records/sec (69.50 MB/sec), 397.5 ms avg latency,
2715.0 ms max latency.
385410 records sent, 77082.0 records/sec (75.28 MB/sec), 417.5 ms avg latency,
2702.0 ms max latency.
381160 records sent, 76232.0 records/sec (74.45 MB/sec), 407.7 ms avg latency,
1846.0 ms max latency.
67 records sent, 0.1 records/sec (65.10 MB/sec), 456.2 ms avg latency,
1414.0 ms max latency.
376251 records sent, 75175.0 records/sec (73.41 MB/sec), 401.9 ms avg latency,
1897.0 ms max latency.
354434 records sent, 70886.8 records/sec (69.23 MB/sec), 425.8 ms avg latency,
1601.0 ms max latency.
353795 records sent, 70744.9 records/sec (69.09 MB/sec), 411.7 ms avg latency,
1563.0 ms max latency.
321993 records sent, 64360.0 records/sec (62.85 MB/sec), 447.3 ms avg latency,
1975.0 ms max latency.
404075 records sent, 80750.4 records/sec (78.86 MB/sec), 408.4 ms avg latency,
1753.0 ms max latency.
384526 records sent, 76905.2 records/sec (75.10 MB/sec), 406.0 ms avg latency,
1833.0 ms max latency.
387652 records sent, 77483.9 records/sec (75.67 MB/sec), 397.3 ms avg latency,
1927.0 ms max latency.
343286 records sent, 68629.7 records/sec (67.02 MB/sec), 455.6 ms avg latency,
1685.0 ms max latency.
00 records sent, 66646.7 records/sec (65.08 MB/sec), 456.6 ms avg latency,
2146.0 ms max latency.
361191 records sent, 72238.2 records/sec (70.55 MB/sec), 409.4 ms avg latency,
2125.0 ms max latency.
357525 records sent, 71490.7 records/sec (69.82 MB/sec), 436.0 ms avg latency,
1502.0 ms max latency.
340238 records sent, 68047.6 records/sec (66.45 MB/sec), 427.9 ms avg latency,
1932.0 ms max latency.
390016 records sent, 77956.4 records/sec (76.13 MB/sec), 418.5 ms avg latency,
1807.0 ms max latency.
352830 records sent, 70523.7 records/sec (68.87 MB/sec), 439.4 ms avg latency,
1892.0 ms max latency.
354526 records sent, 70905.2 records/sec (69.24 MB/sec), 429.6 ms avg latency,
2128.0 ms max latency.
356670 records sent, 71305.5 records/sec (69.63 MB/sec), 408.9 ms avg latency,
1329.0 ms max latency.
309204 records sent, 60687.7 records/sec (59.27 MB/sec), 438.6 ms avg latency,
2566.0 ms max latency.
366715 records sent, 72316.1 records/sec (70.62 MB/sec), 474.5 ms avg latency,
2169.0 ms max latency.
375174 records sent, 75034.8 records/sec (73.28 MB/sec), 429.9 ms avg latency,
1722.0 ms max latency.
359400 records sent, 70346.4 records/sec (68.70 MB/sec), 432.1 ms avg latency,
1961.0 ms max latency.
312276 records sent, 62430.2 records/sec (60.97 MB/sec), 477.4 ms avg latency,
2006.0 ms max latency.
[jira] [Updated] (KAFKA-15264) Compared with 1.1.0zk, the peak throughput of 3.5.1kraft is very jitter
[
https://issues.apache.org/jira/browse/KAFKA-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jianbin.chen updated KAFKA-15264:
-
Attachment: image-2023-07-28-09-51-01-662.png
Description:
I was preparing to upgrade from 1.1.0 to 3.5.1's kraft mode (new cluster
deployment), and when I recently compared and tested, I found that when using
the following stress test command, the throughput gap is obvious
{code:java}
./kafka-producer-perf-test.sh --topic test321 --num-records 3000
--record-size 1024 --throughput -1 --producer-props bootstrap.servers=xxx:
acks=1
419813 records sent, 83962.6 records/sec (81.99 MB/sec), 241.1 ms avg latency,
588.0 ms max latency.
555300 records sent, 111015.6 records/sec (108.41 MB/sec), 275.1 ms avg
latency, 460.0 ms max latency.
552795 records sent, 110536.9 records/sec (107.95 MB/sec), 265.9 ms avg
latency, 1120.0 ms max latency.
552600 records sent, 110520.0 records/sec (107.93 MB/sec), 284.5 ms avg
latency, 1097.0 ms max latency.
538500 records sent, 107656.9 records/sec (105.13 MB/sec), 277.5 ms avg
latency, 610.0 ms max latency.
511545 records sent, 102309.0 records/sec (99.91 MB/sec), 304.1 ms avg latency,
1892.0 ms max latency.
511890 records sent, 102337.1 records/sec (99.94 MB/sec), 288.4 ms avg latency,
3000.0 ms max latency.
519165 records sent, 103812.2 records/sec (101.38 MB/sec), 262.1 ms avg
latency, 1781.0 ms max latency.
513555 records sent, 102669.9 records/sec (100.26 MB/sec), 338.2 ms avg
latency, 2590.0 ms max latency.
463329 records sent, 92665.8 records/sec (90.49 MB/sec), 276.8 ms avg latency,
1463.0 ms max latency.
494248 records sent, 98849.6 records/sec (96.53 MB/sec), 327.2 ms avg latency,
2362.0 ms max latency.
506272 records sent, 101254.4 records/sec (98.88 MB/sec), 322.1 ms avg latency,
2986.0 ms max latency.
393758 records sent, 78735.9 records/sec (76.89 MB/sec), 387.0 ms avg latency,
2958.0 ms max latency.
426435 records sent, 85252.9 records/sec (83.25 MB/sec), 363.3 ms avg latency,
1959.0 ms max latency.
412560 records sent, 82298.0 records/sec (80.37 MB/sec), 374.1 ms avg latency,
1995.0 ms max latency.
370137 records sent, 73997.8 records/sec (72.26 MB/sec), 396.8 ms avg latency,
1496.0 ms max latency.
391781 records sent, 78340.5 records/sec (76.50 MB/sec), 410.7 ms avg latency,
2446.0 ms max latency.
355901 records sent, 71166.0 records/sec (69.50 MB/sec), 397.5 ms avg latency,
2715.0 ms max latency.
385410 records sent, 77082.0 records/sec (75.28 MB/sec), 417.5 ms avg latency,
2702.0 ms max latency.
381160 records sent, 76232.0 records/sec (74.45 MB/sec), 407.7 ms avg latency,
1846.0 ms max latency.
67 records sent, 0.1 records/sec (65.10 MB/sec), 456.2 ms avg latency,
1414.0 ms max latency.
376251 records sent, 75175.0 records/sec (73.41 MB/sec), 401.9 ms avg latency,
1897.0 ms max latency.
354434 records sent, 70886.8 records/sec (69.23 MB/sec), 425.8 ms avg latency,
1601.0 ms max latency.
353795 records sent, 70744.9 records/sec (69.09 MB/sec), 411.7 ms avg latency,
1563.0 ms max latency.
321993 records sent, 64360.0 records/sec (62.85 MB/sec), 447.3 ms avg latency,
1975.0 ms max latency.
404075 records sent, 80750.4 records/sec (78.86 MB/sec), 408.4 ms avg latency,
1753.0 ms max latency.
384526 records sent, 76905.2 records/sec (75.10 MB/sec), 406.0 ms avg latency,
1833.0 ms max latency.
387652 records sent, 77483.9 records/sec (75.67 MB/sec), 397.3 ms avg latency,
1927.0 ms max latency.
343286 records sent, 68629.7 records/sec (67.02 MB/sec), 455.6 ms avg latency,
1685.0 ms max latency.
00 records sent, 66646.7 records/sec (65.08 MB/sec), 456.6 ms avg latency,
2146.0 ms max latency.
361191 records sent, 72238.2 records/sec (70.55 MB/sec), 409.4 ms avg latency,
2125.0 ms max latency.
357525 records sent, 71490.7 records/sec (69.82 MB/sec), 436.0 ms avg latency,
1502.0 ms max latency.
340238 records sent, 68047.6 records/sec (66.45 MB/sec), 427.9 ms avg latency,
1932.0 ms max latency.
390016 records sent, 77956.4 records/sec (76.13 MB/sec), 418.5 ms avg latency,
1807.0 ms max latency.
352830 records sent, 70523.7 records/sec (68.87 MB/sec), 439.4 ms avg latency,
1892.0 ms max latency.
354526 records sent, 70905.2 records/sec (69.24 MB/sec), 429.6 ms avg latency,
2128.0 ms max latency.
356670 records sent, 71305.5 records/sec (69.63 MB/sec), 408.9 ms avg latency,
1329.0 ms max latency.
309204 records sent, 60687.7 records/sec (59.27 MB/sec), 438.6 ms avg latency,
2566.0 ms max latency.
366715 records sent, 72316.1 records/sec (70.62 MB/sec), 474.5 ms avg latency,
2169.0 ms max latency.
375174 records sent, 75034.8 records/sec (73.28 MB/sec), 429.9 ms avg latency,
1722.0 ms max latency.
359400 records sent, 70346.4 records/sec (68.70 MB/sec), 432.1 ms avg latency,
1961.0 ms max latency.
312276 records sent, 62430.2 records/sec (60.97 MB/sec), 477.4 ms avg latency,
2006.0 ms max latency.
