[
https://issues.apache.org/jira/browse/KAFKA-17971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pratheesh M updated KAFKA-17971:
--------------------------------
Affects Version/s: 3.8.1
> Kafka consumer performance has been reduced drastically when added RTT of
> 50ms to Kafka cluster
> -----------------------------------------------------------------------------------------------
>
> Key: KAFKA-17971
> URL: https://issues.apache.org/jira/browse/KAFKA-17971
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 3.7.1, 3.8.1
> Reporter: Pratheesh M
> Priority: Major
> Attachments: kafka-cluster.png
>
>
> First of all my test setup looks like this.(Kafka cluster in machine 1 and
> kafka consumer is running from machine 2. The RTT between both machines is
> 50ms)
> !kafka-cluster.png|width=314,height=241!
> I tested the performance of both producer and consumer using the Kafka files
> kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh.
> *The steps I followed are*
> _1. Create the topic and populate the topic with 100k messages of size 5kb_
> {code:java}
> I have no name!@kafka-0:/opt/kafka/bin$ ./kafka-producer-perf-test.sh
> --topic test-perf --num-records 100000 --record-size 5120 --print-metrics
> --producer-props bootstrap.servers=kafka:9092 --throughput 2000
> 10001 records sent, 2000.2 records/sec (9.77 MB/sec), 14.7 ms avg latency,
> 409.0 ms max latency.
> 10009 records sent, 2001.8 records/sec (9.77 MB/sec), 1.7 ms avg latency,
> 58.0 ms max latency.
> 10000 records sent, 2000.0 records/sec (9.77 MB/sec), 1.9 ms avg latency,
> 49.0 ms max latency.
> 10001 records sent, 2000.2 records/sec (9.77 MB/sec), 1.8 ms avg latency,
> 50.0 ms max latency.
> 10003 records sent, 2000.6 records/sec (9.77 MB/sec), 2.0 ms avg latency,
> 49.0 ms max latency.
> 10004 records sent, 2000.4 records/sec (9.77 MB/sec), 1.6 ms avg latency,
> 46.0 ms max latency.
> 10002 records sent, 2000.4 records/sec (9.77 MB/sec), 1.8 ms avg latency,
> 47.0 ms max latency.
> 10004 records sent, 2000.4 records/sec (9.77 MB/sec), 1.6 ms avg latency,
> 46.0 ms max latency.
> 10004 records sent, 2000.4 records/sec (9.77 MB/sec), 1.8 ms avg latency,
> 46.0 ms max latency.
> 100000 records sent, 1999.800020 records/sec (9.76 MB/sec), 3.04 ms avg
> latency, 409.00 ms max latency, 1 ms 50th, 5 ms 95th, 50 ms 99th, 165 ms
> 99.9th.
> Metric Name
> Value
> app-info:commit-id:{client-id=perf-producer-client}
> : e2494e6ffb89f828
> app-info:start-time-ms:{client-id=perf-producer-client}
> : 1731051100416
> app-info:version:{client-id=perf-producer-client}
> : 3.7.1
> kafka-metrics-count:count:{client-id=perf-producer-client}
> : 124.000
> producer-metrics:batch-size-avg:{client-id=perf-producer-client}
> : 9949.754
> producer-metrics:batch-size-max:{client-id=perf-producer-client}
> : 15448.000
> producer-metrics:batch-split-rate:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:batch-split-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:buffer-available-bytes:{client-id=perf-producer-client}
> : 33554432.000
> producer-metrics:buffer-exhausted-rate:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:buffer-exhausted-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:buffer-total-bytes:{client-id=perf-producer-client}
> : 33554432.000
> producer-metrics:bufferpool-wait-ratio:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:bufferpool-wait-time-ns-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:bufferpool-wait-time-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:compression-rate-avg:{client-id=perf-producer-client}
> : 1.000
> producer-metrics:connection-close-rate:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:connection-close-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:connection-count:{client-id=perf-producer-client}
> : 3.000
> producer-metrics:connection-creation-rate:{client-id=perf-producer-client}
> : 0.060
> producer-metrics:connection-creation-total:{client-id=perf-producer-client}
> : 3.000
> producer-metrics:failed-authentication-rate:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:failed-authentication-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:failed-reauthentication-rate:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:failed-reauthentication-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:flush-time-ns-total:{client-id=perf-producer-client}
> : 4807328.000
> producer-metrics:incoming-byte-rate:{client-id=perf-producer-client}
> : 62595.167
> producer-metrics:incoming-byte-total:{client-id=perf-producer-client}
> : 3113734.000
> producer-metrics:io-ratio:{client-id=perf-producer-client}
> : 0.109
> producer-metrics:io-time-ns-avg:{client-id=perf-producer-client}
> : 42741.093
> producer-metrics:io-time-ns-total:{client-id=perf-producer-client}
> : 5413202225.000
> producer-metrics:io-wait-ratio:{client-id=perf-producer-client}
> : 0.787
> producer-metrics:io-wait-time-ns-avg:{client-id=perf-producer-client}
> : 309425.910
> producer-metrics:io-wait-time-ns-total:{client-id=perf-producer-client}
> : 39189100908.000
> producer-metrics:io-waittime-total:{client-id=perf-producer-client}
> : 39189100908.000
> producer-metrics:iotime-total:{client-id=perf-producer-client}
> : 5413202225.000
> producer-metrics:metadata-age:{client-id=perf-producer-client}
> : 49.672
> producer-metrics:metadata-wait-time-ns-total:{client-id=perf-producer-client}
> : 354633968.000
> producer-metrics:network-io-rate:{client-id=perf-producer-client}
> : 2085.558
> producer-metrics:network-io-total:{client-id=perf-producer-client}
> : 103744.000
> producer-metrics:outgoing-byte-rate:{client-id=perf-producer-client}
> : 10440089.358
> producer-metrics:outgoing-byte-total:{client-id=perf-producer-client}
> : 519331805.000
> producer-metrics:produce-throttle-time-avg:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:produce-throttle-time-max:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:reauthentication-latency-avg:{client-id=perf-producer-client}
> : NaN
> producer-metrics:reauthentication-latency-max:{client-id=perf-producer-client}
> : NaN
> producer-metrics:record-error-rate:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:record-error-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:record-queue-time-avg:{client-id=perf-producer-client}
> : 1.374
> producer-metrics:record-queue-time-max:{client-id=perf-producer-client}
> : 170.000
> producer-metrics:record-retry-rate:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:record-retry-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:record-send-rate:{client-id=perf-producer-client}
> : 2015.032
> producer-metrics:record-send-total:{client-id=perf-producer-client}
> : 100000.000
> producer-metrics:record-size-avg:{client-id=perf-producer-client}
> : 5206.000
> producer-metrics:record-size-max:{client-id=perf-producer-client}
> : 5206.000
> producer-metrics:records-per-request-avg:{client-id=perf-producer-client}
> : 1.928
> producer-metrics:request-latency-avg:{client-id=perf-producer-client}
> : 0.910
> producer-metrics:request-latency-max:{client-id=perf-producer-client}
> : 59.000
> producer-metrics:request-rate:{client-id=perf-producer-client}
> : 1042.800
> producer-metrics:request-size-avg:{client-id=perf-producer-client}
> : 10011.795
> producer-metrics:request-size-max:{client-id=perf-producer-client}
> : 15511.000
> producer-metrics:request-total:{client-id=perf-producer-client}
> : 51872.000
> producer-metrics:requests-in-flight:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:response-rate:{client-id=perf-producer-client}
> : 1042.821
> producer-metrics:response-total:{client-id=perf-producer-client}
> : 51872.000
> producer-metrics:select-rate:{client-id=perf-producer-client}
> : 2544.266
> producer-metrics:select-total:{client-id=perf-producer-client}
> : 126651.000
> producer-metrics:successful-authentication-no-reauth-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:successful-authentication-rate:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:successful-authentication-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:successful-reauthentication-rate:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:successful-reauthentication-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:txn-abort-time-ns-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:txn-begin-time-ns-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:txn-commit-time-ns-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:txn-init-time-ns-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:txn-send-offsets-time-ns-total:{client-id=perf-producer-client}
> : 0.000
> producer-metrics:waiting-threads:{client-id=perf-producer-client}
> : 0.000
> producer-node-metrics:incoming-byte-rate:{client-id=perf-producer-client,
> node-id=node--1} : 15.359
> producer-node-metrics:incoming-byte-rate:{client-id=perf-producer-client,
> node-id=node-0} : 9.565
> producer-node-metrics:incoming-byte-rate:{client-id=perf-producer-client,
> node-id=node-2} : 62707.666
> producer-node-metrics:incoming-byte-total:{client-id=perf-producer-client,
> node-id=node--1} : 764.000
> producer-node-metrics:incoming-byte-total:{client-id=perf-producer-client,
> node-id=node-0} : 475.000
> producer-node-metrics:incoming-byte-total:{client-id=perf-producer-client,
> node-id=node-2} : 3112495.000
> producer-node-metrics:outgoing-byte-rate:{client-id=perf-producer-client,
> node-id=node--1} : 3.558
> producer-node-metrics:outgoing-byte-rate:{client-id=perf-producer-client,
> node-id=node-0} : 1.208
> producer-node-metrics:outgoing-byte-rate:{client-id=perf-producer-client,
> node-id=node-2} : 10463011.343
> producer-node-metrics:outgoing-byte-total:{client-id=perf-producer-client,
> node-id=node--1} : 177.000
> producer-node-metrics:outgoing-byte-total:{client-id=perf-producer-client,
> node-id=node-0} : 60.000
> producer-node-metrics:outgoing-byte-total:{client-id=perf-producer-client,
> node-id=node-2} : 519331568.000
> producer-node-metrics:request-latency-avg:{client-id=perf-producer-client,
> node-id=node--1} : NaN
> producer-node-metrics:request-latency-avg:{client-id=perf-producer-client,
> node-id=node-0} : NaN
> producer-node-metrics:request-latency-avg:{client-id=perf-producer-client,
> node-id=node-2} : 0.910
> producer-node-metrics:request-latency-max:{client-id=perf-producer-client,
> node-id=node--1} : NaN
> producer-node-metrics:request-latency-max:{client-id=perf-producer-client,
> node-id=node-0} : NaN
> producer-node-metrics:request-latency-max:{client-id=perf-producer-client,
> node-id=node-2} : 59.000
> producer-node-metrics:request-rate:{client-id=perf-producer-client,
> node-id=node--1} : 0.060
> producer-node-metrics:request-rate:{client-id=perf-producer-client,
> node-id=node-0} : 0.020
> producer-node-metrics:request-rate:{client-id=perf-producer-client,
> node-id=node-2} : 1045.009
> producer-node-metrics:request-size-avg:{client-id=perf-producer-client,
> node-id=node--1} : 59.000
> producer-node-metrics:request-size-avg:{client-id=perf-producer-client,
> node-id=node-0} : 60.000
> producer-node-metrics:request-size-avg:{client-id=perf-producer-client,
> node-id=node-2} : 10012.562
> producer-node-metrics:request-size-max:{client-id=perf-producer-client,
> node-id=node--1} : 66.000
> producer-node-metrics:request-size-max:{client-id=perf-producer-client,
> node-id=node-0} : 60.000
> producer-node-metrics:request-size-max:{client-id=perf-producer-client,
> node-id=node-2} : 15511.000
> producer-node-metrics:request-total:{client-id=perf-producer-client,
> node-id=node--1} : 3.000
> producer-node-metrics:request-total:{client-id=perf-producer-client,
> node-id=node-0} : 1.000
> producer-node-metrics:request-total:{client-id=perf-producer-client,
> node-id=node-2} : 51868.000
> producer-node-metrics:response-rate:{client-id=perf-producer-client,
> node-id=node--1} : 0.060
> producer-node-metrics:response-rate:{client-id=perf-producer-client,
> node-id=node-0} : 0.020
> producer-node-metrics:response-rate:{client-id=perf-producer-client,
> node-id=node-2} : 1045.031
> producer-node-metrics:response-total:{client-id=perf-producer-client,
> node-id=node--1} : 3.000
> producer-node-metrics:response-total:{client-id=perf-producer-client,
> node-id=node-0} : 1.000
> producer-node-metrics:response-total:{client-id=perf-producer-client,
> node-id=node-2} : 51868.000
> producer-topic-metrics:byte-rate:{client-id=perf-producer-client,
> topic=test-perf} : 10399481.843
> producer-topic-metrics:byte-total:{client-id=perf-producer-client,
> topic=test-perf} : 516063887.000
> producer-topic-metrics:compression-rate:{client-id=perf-producer-client,
> topic=test-perf} : 1.000
> producer-topic-metrics:record-error-rate:{client-id=perf-producer-client,
> topic=test-perf} : 0.000
> producer-topic-metrics:record-error-total:{client-id=perf-producer-client,
> topic=test-perf} : 0.000
> producer-topic-metrics:record-retry-rate:{client-id=perf-producer-client,
> topic=test-perf} : 0.000
> producer-topic-metrics:record-retry-total:{client-id=perf-producer-client,
> topic=test-perf} : 0.000
> producer-topic-metrics:record-send-rate:{client-id=perf-producer-client,
> topic=test-perf} : 2015.154
> producer-topic-metrics:record-send-total:{client-id=perf-producer-client,
> topic=test-perf} : 100000.000
> I have no name!@kafka-0:/opt/kafka/bin$ {code}
> _2. Do consumer performance test without latency to the kafka cluster_
> {code:java}
> I have no name!@kafka-2:/opt/kafka/bin$ ./kafka-consumer-perf-test.sh
> --bootstrap-server 10.193.27.216:10092 --topic test-perf --socket-buffer-size
> 8388608 --print-metrics --messages 100000 --fetch-size 8388608
> start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg,
> nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
> 2024-11-08 07:43:24:389, 2024-11-08 07:43:27:466, 488.2813, 158.6874, 100000,
> 32499.1875, 375, 2702, 180.7110, 37009.6225
> Metric Name
> Value
> consumer-coordinator-metrics:assigned-partitions:{client-id=perf-consumer-client}
> : 0.000
> consumer-coordinator-metrics:commit-latency-avg:{client-id=perf-consumer-client}
> : 4.000
> consumer-coordinator-metrics:commit-latency-max:{client-id=perf-consumer-client}
> : 4.000
> consumer-coordinator-metrics:commit-rate:{client-id=perf-consumer-client}
> : 0.033
> consumer-coordinator-metrics:commit-total:{client-id=perf-consumer-client}
> : 1.000
> consumer-coordinator-metrics:failed-rebalance-rate-per-hour:{client-id=perf-consumer-client}
> : 108.176
> consumer-coordinator-metrics:failed-rebalance-total:{client-id=perf-consumer-client}
> : 1.000
> consumer-coordinator-metrics:heartbeat-rate:{client-id=perf-consumer-client}
> : 0.000
> consumer-coordinator-metrics:heartbeat-response-time-max:{client-id=perf-consumer-client}
> : NaN
> consumer-coordinator-metrics:heartbeat-total:{client-id=perf-consumer-client}
> : 0.000
> consumer-coordinator-metrics:join-rate:{client-id=perf-consumer-client}
> : 0.030
> consumer-coordinator-metrics:join-time-avg:{client-id=perf-consumer-client}
> : 4.000
> consumer-coordinator-metrics:join-time-max:{client-id=perf-consumer-client}
> : 4.000
> consumer-coordinator-metrics:join-total:{client-id=perf-consumer-client}
> : 1.000
> consumer-coordinator-metrics:last-heartbeat-seconds-ago:{client-id=perf-consumer-client}
> : -1.000
> consumer-coordinator-metrics:last-rebalance-seconds-ago:{client-id=perf-consumer-client}
> : 3.000
> consumer-coordinator-metrics:partition-assigned-latency-avg:{client-id=perf-consumer-client}
> : 0.000
> consumer-coordinator-metrics:partition-assigned-latency-max:{client-id=perf-consumer-client}
> : 0.000
> consumer-coordinator-metrics:partition-lost-latency-avg:{client-id=perf-consumer-client}
> : NaN
> consumer-coordinator-metrics:partition-lost-latency-max:{client-id=perf-consumer-client}
> : NaN
> consumer-coordinator-metrics:partition-revoked-latency-avg:{client-id=perf-consumer-client}
> : 0.000
> consumer-coordinator-metrics:partition-revoked-latency-max:{client-id=perf-consumer-client}
> : 0.000
> consumer-coordinator-metrics:rebalance-latency-avg:{client-id=perf-consumer-client}
> : 35.000
> consumer-coordinator-metrics:rebalance-latency-max:{client-id=perf-consumer-client}
> : 35.000
> consumer-coordinator-metrics:rebalance-latency-total:{client-id=perf-consumer-client}
> : 35.000
> consumer-coordinator-metrics:rebalance-rate-per-hour:{client-id=perf-consumer-client}
> : 108.258
> consumer-coordinator-metrics:rebalance-total:{client-id=perf-consumer-client}
> : 1.000
> consumer-coordinator-metrics:sync-rate:{client-id=perf-consumer-client}
> : 0.030
> consumer-coordinator-metrics:sync-time-avg:{client-id=perf-consumer-client}
> : 11.000
> consumer-coordinator-metrics:sync-time-max:{client-id=perf-consumer-client}
> : 11.000
> consumer-coordinator-metrics:sync-total:{client-id=perf-consumer-client}
> : 1.000
> consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=perf-consumer-client,
> topic=test-perf} : 15500619.541
> consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=perf-consumer-client}
> : 15500619.541
> consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=perf-consumer-client,
> topic=test-perf} : 512900000.000
> consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=perf-consumer-client}
> : 512900000.000
> consumer-fetch-manager-metrics:fetch-latency-avg:{client-id=perf-consumer-client}
> : 39.333
> consumer-fetch-manager-metrics:fetch-latency-max:{client-id=perf-consumer-client}
> : 500.000
> consumer-fetch-manager-metrics:fetch-rate:{client-id=perf-consumer-client}
> : 1.899
> consumer-fetch-manager-metrics:fetch-size-avg:{client-id=perf-consumer-client,
> topic=test-perf} : 8272580.645
> consumer-fetch-manager-metrics:fetch-size-avg:{client-id=perf-consumer-client}
> : 8272580.645
> consumer-fetch-manager-metrics:fetch-size-max:{client-id=perf-consumer-client,
> topic=test-perf} : 8344883.000
> consumer-fetch-manager-metrics:fetch-size-max:{client-id=perf-consumer-client}
> : 8344883.000
> consumer-fetch-manager-metrics:fetch-throttle-time-avg:{client-id=perf-consumer-client}
> : 0.000
> consumer-fetch-manager-metrics:fetch-throttle-time-max:{client-id=perf-consumer-client}
> : 0.000
> consumer-fetch-manager-metrics:fetch-total:{client-id=perf-consumer-client}
> : 63.000
> consumer-fetch-manager-metrics:preferred-read-replica:{client-id=perf-consumer-client,
> topic=test-perf, partition=0} : -1
> consumer-fetch-manager-metrics:records-consumed-rate:{client-id=perf-consumer-client,
> topic=test-perf} : 3022.244
> consumer-fetch-manager-metrics:records-consumed-rate:{client-id=perf-consumer-client}
> : 3021.970
> consumer-fetch-manager-metrics:records-consumed-total:{client-id=perf-consumer-client,
> topic=test-perf} : 100000.000
> consumer-fetch-manager-metrics:records-consumed-total:{client-id=perf-consumer-client}
> : 100000.000
> consumer-fetch-manager-metrics:records-lag-avg:{client-id=perf-consumer-client,
> topic=test-perf, partition=0} : 49702.008
> consumer-fetch-manager-metrics:records-lag-max:{client-id=perf-consumer-client,
> topic=test-perf, partition=0} : 99500.000
> consumer-fetch-manager-metrics:records-lag-max:{client-id=perf-consumer-client}
> : 99500.000
> consumer-fetch-manager-metrics:records-lag:{client-id=perf-consumer-client,
> topic=test-perf, partition=0} : 0.000
> consumer-fetch-manager-metrics:records-lead-avg:{client-id=perf-consumer-client,
> topic=test-perf, partition=0} : 50297.992
> consumer-fetch-manager-metrics:records-lead-min:{client-id=perf-consumer-client,
> topic=test-perf, partition=0} : 500.000
> consumer-fetch-manager-metrics:records-lead-min:{client-id=perf-consumer-client}
> : 500.000
> consumer-fetch-manager-metrics:records-lead:{client-id=perf-consumer-client,
> topic=test-perf, partition=0} : 100000.000
> consumer-fetch-manager-metrics:records-per-request-avg:{client-id=perf-consumer-client,
> topic=test-perf} : 1612.903
> consumer-fetch-manager-metrics:records-per-request-avg:{client-id=perf-consumer-client}
> : 1612.903
> kafka-metrics-count:count:{client-id=perf-consumer-client}
> : 61.000
> I have no name!@kafka-2:/opt/kafka/bin$ {code}
> _3. Do consumer performance test with latency(RTT 50ms)to the kafka cluster_
> {code:java}
> I have no name!@kafka-2:/opt/kafka/bin$ ./kafka-consumer-perf-test.sh
> --bootstrap-server 10.193.27.216:10092 --topic test-perf --socket-buffer-size
> 8388608 --print-metrics --messages 100000 --fetch-size 8388608
> start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg,
> nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
> 2024-11-08 07:38:27:986, 2024-11-08 07:40:40:368, 488.2813, 3.6884, 100000,
> 755.3897, 903, 131479, 3.7138, 760.5777
> Metric Name
> Value
> consumer-coordinator-metrics:assigned-partitions:{client-id=perf-consumer-client}
> : 0.000
> consumer-coordinator-metrics:commit-latency-avg:{client-id=perf-consumer-client}
> : 64.571
> consumer-coordinator-metrics:commit-latency-max:{client-id=perf-consumer-client}
> : 96.000
> consumer-coordinator-metrics:commit-rate:{client-id=perf-consumer-client}
> : 0.123
> consumer-coordinator-metrics:commit-total:{client-id=perf-consumer-client}
> : 27.000
> consumer-coordinator-metrics:failed-rebalance-rate-per-hour:{client-id=perf-consumer-client}
> : 0.000
> consumer-coordinator-metrics:failed-rebalance-total:{client-id=perf-consumer-client}
> : 1.000
> consumer-coordinator-metrics:heartbeat-rate:{client-id=perf-consumer-client}
> : 0.332
> consumer-coordinator-metrics:heartbeat-response-time-max:{client-id=perf-consumer-client}
> : 64.000
> consumer-coordinator-metrics:heartbeat-total:{client-id=perf-consumer-client}
> : 43.000
> consumer-coordinator-metrics:join-rate:{client-id=perf-consumer-client}
> : 0.000
> consumer-coordinator-metrics:join-time-avg:{client-id=perf-consumer-client}
> : NaN
> consumer-coordinator-metrics:join-time-max:{client-id=perf-consumer-client}
> : NaN
> consumer-coordinator-metrics:join-total:{client-id=perf-consumer-client}
> : 1.000
> consumer-coordinator-metrics:last-heartbeat-seconds-ago:{client-id=perf-consumer-client}
> : 3.000
> consumer-coordinator-metrics:last-rebalance-seconds-ago:{client-id=perf-consumer-client}
> : 132.000
> consumer-coordinator-metrics:partition-assigned-latency-avg:{client-id=perf-consumer-client}
> : NaN
> consumer-coordinator-metrics:partition-assigned-latency-max:{client-id=perf-consumer-client}
> : NaN
> consumer-coordinator-metrics:partition-lost-latency-avg:{client-id=perf-consumer-client}
> : NaN
> consumer-coordinator-metrics:partition-lost-latency-max:{client-id=perf-consumer-client}
> : NaN
> consumer-coordinator-metrics:partition-revoked-latency-avg:{client-id=perf-consumer-client}
> : 0.000
> consumer-coordinator-metrics:partition-revoked-latency-max:{client-id=perf-consumer-client}
> : 0.000
> consumer-coordinator-metrics:rebalance-latency-avg:{client-id=perf-consumer-client}
> : NaN
> consumer-coordinator-metrics:rebalance-latency-max:{client-id=perf-consumer-client}
> : NaN
> consumer-coordinator-metrics:rebalance-latency-total:{client-id=perf-consumer-client}
> : 363.000
> consumer-coordinator-metrics:rebalance-rate-per-hour:{client-id=perf-consumer-client}
> : 0.000
> consumer-coordinator-metrics:rebalance-total:{client-id=perf-consumer-client}
> : 1.000
> consumer-coordinator-metrics:sync-rate:{client-id=perf-consumer-client}
> : 0.000
> consumer-coordinator-metrics:sync-time-avg:{client-id=perf-consumer-client}
> : NaN
> consumer-coordinator-metrics:sync-time-max:{client-id=perf-consumer-client}
> : NaN
> consumer-coordinator-metrics:sync-total:{client-id=perf-consumer-client}
> : 1.000
> consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=perf-consumer-client,
> topic=test-perf} : 4091208.379
> consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=perf-consumer-client}
> : 4091322.799
> consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=perf-consumer-client,
> topic=test-perf} : 512900000.000
> consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=perf-consumer-client}
> : 512900000.000
> consumer-fetch-manager-metrics:fetch-latency-avg:{client-id=perf-consumer-client}
> : 1984.789
> consumer-fetch-manager-metrics:fetch-latency-max:{client-id=perf-consumer-client}
> : 2383.000
> consumer-fetch-manager-metrics:fetch-rate:{client-id=perf-consumer-client}
> : 0.531
> consumer-fetch-manager-metrics:fetch-size-avg:{client-id=perf-consumer-client,
> topic=test-perf} : 8127185.444
> consumer-fetch-manager-metrics:fetch-size-avg:{client-id=perf-consumer-client}
> : 8127185.444
> consumer-fetch-manager-metrics:fetch-size-max:{client-id=perf-consumer-client,
> topic=test-perf} : 8334625.000
> consumer-fetch-manager-metrics:fetch-size-max:{client-id=perf-consumer-client}
> : 8334625.000
> consumer-fetch-manager-metrics:fetch-throttle-time-avg:{client-id=perf-consumer-client}
> : 0.000
> consumer-fetch-manager-metrics:fetch-throttle-time-max:{client-id=perf-consumer-client}
> : 0.000
> consumer-fetch-manager-metrics:fetch-total:{client-id=perf-consumer-client}
> : 63.000
> consumer-fetch-manager-metrics:preferred-read-replica:{client-id=perf-consumer-client,
> topic=test-perf, partition=0} : -1
> consumer-fetch-manager-metrics:records-consumed-rate:{client-id=perf-consumer-client,
> topic=test-perf} : 797.662
> consumer-fetch-manager-metrics:records-consumed-rate:{client-id=perf-consumer-client}
> : 797.662
> consumer-fetch-manager-metrics:records-consumed-total:{client-id=perf-consumer-client,
> topic=test-perf} : 100000.000
> consumer-fetch-manager-metrics:records-consumed-total:{client-id=perf-consumer-client}
> : 100000.000
> consumer-fetch-manager-metrics:records-lag-avg:{client-id=perf-consumer-client,
> topic=test-perf, partition=0} : 13969.314
> consumer-fetch-manager-metrics:records-lag-max:{client-id=perf-consumer-client,
> topic=test-perf, partition=0} : 28022.000
> consumer-fetch-manager-metrics:records-lag-max:{client-id=perf-consumer-client}
> : 28022.000
> consumer-fetch-manager-metrics:records-lag:{client-id=perf-consumer-client,
> topic=test-perf, partition=0} : 0.000
> consumer-fetch-manager-metrics:records-lead-avg:{client-id=perf-consumer-client,
> topic=test-perf, partition=0} : 86030.686
> consumer-fetch-manager-metrics:records-lead-min:{client-id=perf-consumer-client,
> topic=test-perf, partition=0} : 71978.000
> consumer-fetch-manager-metrics:records-lead-min:{client-id=perf-consumer-client}
> : 71978.000
> consumer-fetch-manager-metrics:records-lead:{client-id=perf-consumer-client,
> topic=test-perf, partition=0} : 100000.000
> consumer-fetch-manager-metrics:records-per-request-avg:{client-id=perf-consumer-client,
> topic=test-perf} : 1584.556
> consumer-fetch-manager-metrics:records-per-request-avg:{client-id=perf-consumer-client}
> : 1584.556
> kafka-metrics-count:count:{client-id=perf-consumer-client}
> : 61.000
> I have no name!@kafka-2:/opt/kafka/bin$ {code}
> *Observations:*
> * The consumer rate is decreased from 158.6874 MB.sec to 3.6884 MB.sec.
> * Some metrics to compare
> {code:java}
> with latency
> ------------
> consumer-fetch-manager-metrics:fetch-latency-avg:{client-id=perf-consumer-client}
> : 1984.789
> consumer-fetch-manager-metrics:fetch-latency-max:{client-id=perf-consumer-client}
> : 2383.000
> consumer-fetch-manager-metrics:fetch-total:{client-id=perf-consumer-client}
> : 63.000
> without latency
> ---------------
> consumer-fetch-manager-metrics:fetch-latency-avg:{client-id=perf-consumer-client}
> : 39.333
> consumer-fetch-manager-metrics:fetch-latency-max:{client-id=perf-consumer-client}
> : 500.000
> consumer-fetch-manager-metrics:fetch-total:{client-id=perf-consumer-client}
> : 63.000 {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
