Pratheesh M created KAFKA-17971:
-----------------------------------
Summary: Kafka consumer performance has been reduced drastically
when added RTT of 50ms to Kafka cluster
Key: KAFKA-17971
URL: https://issues.apache.org/jira/browse/KAFKA-17971
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 3.7.1
Reporter: Pratheesh M
Attachments: kafka-cluster.png
First of all my test setup looks like this.(Kafka cluster in machine 1 and
kafka consumer is running from machine 2. The RTT between both machines is 50ms)
!kafka-cluster.png|width=314,height=241!
I tested the performance of both producer and consumer using the Kafka files
kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh.
*The steps I followed are*
# _Create the topic and populate the topic with 100k messages of size 5kb_
{code:java}
I have no name!@kafka-0:/opt/kafka/bin$ ./kafka-producer-perf-test.sh --topic
test-perf --num-records 100000 --record-size 5120 --print-metrics
--producer-props bootstrap.servers=kafka:9092 --throughput 2000
10001 records sent, 2000.2 records/sec (9.77 MB/sec), 14.7 ms avg latency,
409.0 ms max latency.
10009 records sent, 2001.8 records/sec (9.77 MB/sec), 1.7 ms avg latency, 58.0
ms max latency.
10000 records sent, 2000.0 records/sec (9.77 MB/sec), 1.9 ms avg latency, 49.0
ms max latency.
10001 records sent, 2000.2 records/sec (9.77 MB/sec), 1.8 ms avg latency, 50.0
ms max latency.
10003 records sent, 2000.6 records/sec (9.77 MB/sec), 2.0 ms avg latency, 49.0
ms max latency.
10004 records sent, 2000.4 records/sec (9.77 MB/sec), 1.6 ms avg latency, 46.0
ms max latency.
10002 records sent, 2000.4 records/sec (9.77 MB/sec), 1.8 ms avg latency, 47.0
ms max latency.
10004 records sent, 2000.4 records/sec (9.77 MB/sec), 1.6 ms avg latency, 46.0
ms max latency.
10004 records sent, 2000.4 records/sec (9.77 MB/sec), 1.8 ms avg latency, 46.0
ms max latency.
100000 records sent, 1999.800020 records/sec (9.76 MB/sec), 3.04 ms avg
latency, 409.00 ms max latency, 1 ms 50th, 5 ms 95th, 50 ms 99th, 165 ms 99.9th.
Metric Name
Value
app-info:commit-id:{client-id=perf-producer-client}
: e2494e6ffb89f828
app-info:start-time-ms:{client-id=perf-producer-client}
: 1731051100416
app-info:version:{client-id=perf-producer-client}
: 3.7.1
kafka-metrics-count:count:{client-id=perf-producer-client}
: 124.000
producer-metrics:batch-size-avg:{client-id=perf-producer-client}
: 9949.754
producer-metrics:batch-size-max:{client-id=perf-producer-client}
: 15448.000
producer-metrics:batch-split-rate:{client-id=perf-producer-client}
: 0.000
producer-metrics:batch-split-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:buffer-available-bytes:{client-id=perf-producer-client}
: 33554432.000
producer-metrics:buffer-exhausted-rate:{client-id=perf-producer-client}
: 0.000
producer-metrics:buffer-exhausted-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:buffer-total-bytes:{client-id=perf-producer-client}
: 33554432.000
producer-metrics:bufferpool-wait-ratio:{client-id=perf-producer-client}
: 0.000
producer-metrics:bufferpool-wait-time-ns-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:bufferpool-wait-time-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:compression-rate-avg:{client-id=perf-producer-client}
: 1.000
producer-metrics:connection-close-rate:{client-id=perf-producer-client}
: 0.000
producer-metrics:connection-close-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:connection-count:{client-id=perf-producer-client}
: 3.000
producer-metrics:connection-creation-rate:{client-id=perf-producer-client}
: 0.060
producer-metrics:connection-creation-total:{client-id=perf-producer-client}
: 3.000
producer-metrics:failed-authentication-rate:{client-id=perf-producer-client}
: 0.000
producer-metrics:failed-authentication-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:failed-reauthentication-rate:{client-id=perf-producer-client}
: 0.000
producer-metrics:failed-reauthentication-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:flush-time-ns-total:{client-id=perf-producer-client}
: 4807328.000
producer-metrics:incoming-byte-rate:{client-id=perf-producer-client}
: 62595.167
producer-metrics:incoming-byte-total:{client-id=perf-producer-client}
: 3113734.000
producer-metrics:io-ratio:{client-id=perf-producer-client}
: 0.109
producer-metrics:io-time-ns-avg:{client-id=perf-producer-client}
: 42741.093
producer-metrics:io-time-ns-total:{client-id=perf-producer-client}
: 5413202225.000
producer-metrics:io-wait-ratio:{client-id=perf-producer-client}
: 0.787
producer-metrics:io-wait-time-ns-avg:{client-id=perf-producer-client}
: 309425.910
producer-metrics:io-wait-time-ns-total:{client-id=perf-producer-client}
: 39189100908.000
producer-metrics:io-waittime-total:{client-id=perf-producer-client}
: 39189100908.000
producer-metrics:iotime-total:{client-id=perf-producer-client}
: 5413202225.000
producer-metrics:metadata-age:{client-id=perf-producer-client}
: 49.672
producer-metrics:metadata-wait-time-ns-total:{client-id=perf-producer-client}
: 354633968.000
producer-metrics:network-io-rate:{client-id=perf-producer-client}
: 2085.558
producer-metrics:network-io-total:{client-id=perf-producer-client}
: 103744.000
producer-metrics:outgoing-byte-rate:{client-id=perf-producer-client}
: 10440089.358
producer-metrics:outgoing-byte-total:{client-id=perf-producer-client}
: 519331805.000
producer-metrics:produce-throttle-time-avg:{client-id=perf-producer-client}
: 0.000
producer-metrics:produce-throttle-time-max:{client-id=perf-producer-client}
: 0.000
producer-metrics:reauthentication-latency-avg:{client-id=perf-producer-client}
: NaN
producer-metrics:reauthentication-latency-max:{client-id=perf-producer-client}
: NaN
producer-metrics:record-error-rate:{client-id=perf-producer-client}
: 0.000
producer-metrics:record-error-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:record-queue-time-avg:{client-id=perf-producer-client}
: 1.374
producer-metrics:record-queue-time-max:{client-id=perf-producer-client}
: 170.000
producer-metrics:record-retry-rate:{client-id=perf-producer-client}
: 0.000
producer-metrics:record-retry-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:record-send-rate:{client-id=perf-producer-client}
: 2015.032
producer-metrics:record-send-total:{client-id=perf-producer-client}
: 100000.000
producer-metrics:record-size-avg:{client-id=perf-producer-client}
: 5206.000
producer-metrics:record-size-max:{client-id=perf-producer-client}
: 5206.000
producer-metrics:records-per-request-avg:{client-id=perf-producer-client}
: 1.928
producer-metrics:request-latency-avg:{client-id=perf-producer-client}
: 0.910
producer-metrics:request-latency-max:{client-id=perf-producer-client}
: 59.000
producer-metrics:request-rate:{client-id=perf-producer-client}
: 1042.800
producer-metrics:request-size-avg:{client-id=perf-producer-client}
: 10011.795
producer-metrics:request-size-max:{client-id=perf-producer-client}
: 15511.000
producer-metrics:request-total:{client-id=perf-producer-client}
: 51872.000
producer-metrics:requests-in-flight:{client-id=perf-producer-client}
: 0.000
producer-metrics:response-rate:{client-id=perf-producer-client}
: 1042.821
producer-metrics:response-total:{client-id=perf-producer-client}
: 51872.000
producer-metrics:select-rate:{client-id=perf-producer-client}
: 2544.266
producer-metrics:select-total:{client-id=perf-producer-client}
: 126651.000
producer-metrics:successful-authentication-no-reauth-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:successful-authentication-rate:{client-id=perf-producer-client}
: 0.000
producer-metrics:successful-authentication-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:successful-reauthentication-rate:{client-id=perf-producer-client}
: 0.000
producer-metrics:successful-reauthentication-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:txn-abort-time-ns-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:txn-begin-time-ns-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:txn-commit-time-ns-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:txn-init-time-ns-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:txn-send-offsets-time-ns-total:{client-id=perf-producer-client}
: 0.000
producer-metrics:waiting-threads:{client-id=perf-producer-client}
: 0.000
producer-node-metrics:incoming-byte-rate:{client-id=perf-producer-client,
node-id=node--1} : 15.359
producer-node-metrics:incoming-byte-rate:{client-id=perf-producer-client,
node-id=node-0} : 9.565
producer-node-metrics:incoming-byte-rate:{client-id=perf-producer-client,
node-id=node-2} : 62707.666
producer-node-metrics:incoming-byte-total:{client-id=perf-producer-client,
node-id=node--1} : 764.000
producer-node-metrics:incoming-byte-total:{client-id=perf-producer-client,
node-id=node-0} : 475.000
producer-node-metrics:incoming-byte-total:{client-id=perf-producer-client,
node-id=node-2} : 3112495.000
producer-node-metrics:outgoing-byte-rate:{client-id=perf-producer-client,
node-id=node--1} : 3.558
producer-node-metrics:outgoing-byte-rate:{client-id=perf-producer-client,
node-id=node-0} : 1.208
producer-node-metrics:outgoing-byte-rate:{client-id=perf-producer-client,
node-id=node-2} : 10463011.343
producer-node-metrics:outgoing-byte-total:{client-id=perf-producer-client,
node-id=node--1} : 177.000
producer-node-metrics:outgoing-byte-total:{client-id=perf-producer-client,
node-id=node-0} : 60.000
producer-node-metrics:outgoing-byte-total:{client-id=perf-producer-client,
node-id=node-2} : 519331568.000
producer-node-metrics:request-latency-avg:{client-id=perf-producer-client,
node-id=node--1} : NaN
producer-node-metrics:request-latency-avg:{client-id=perf-producer-client,
node-id=node-0} : NaN
producer-node-metrics:request-latency-avg:{client-id=perf-producer-client,
node-id=node-2} : 0.910
producer-node-metrics:request-latency-max:{client-id=perf-producer-client,
node-id=node--1} : NaN
producer-node-metrics:request-latency-max:{client-id=perf-producer-client,
node-id=node-0} : NaN
producer-node-metrics:request-latency-max:{client-id=perf-producer-client,
node-id=node-2} : 59.000
producer-node-metrics:request-rate:{client-id=perf-producer-client,
node-id=node--1} : 0.060
producer-node-metrics:request-rate:{client-id=perf-producer-client,
node-id=node-0} : 0.020
producer-node-metrics:request-rate:{client-id=perf-producer-client,
node-id=node-2} : 1045.009
producer-node-metrics:request-size-avg:{client-id=perf-producer-client,
node-id=node--1} : 59.000
producer-node-metrics:request-size-avg:{client-id=perf-producer-client,
node-id=node-0} : 60.000
producer-node-metrics:request-size-avg:{client-id=perf-producer-client,
node-id=node-2} : 10012.562
producer-node-metrics:request-size-max:{client-id=perf-producer-client,
node-id=node--1} : 66.000
producer-node-metrics:request-size-max:{client-id=perf-producer-client,
node-id=node-0} : 60.000
producer-node-metrics:request-size-max:{client-id=perf-producer-client,
node-id=node-2} : 15511.000
producer-node-metrics:request-total:{client-id=perf-producer-client,
node-id=node--1} : 3.000
producer-node-metrics:request-total:{client-id=perf-producer-client,
node-id=node-0} : 1.000
producer-node-metrics:request-total:{client-id=perf-producer-client,
node-id=node-2} : 51868.000
producer-node-metrics:response-rate:{client-id=perf-producer-client,
node-id=node--1} : 0.060
producer-node-metrics:response-rate:{client-id=perf-producer-client,
node-id=node-0} : 0.020
producer-node-metrics:response-rate:{client-id=perf-producer-client,
node-id=node-2} : 1045.031
producer-node-metrics:response-total:{client-id=perf-producer-client,
node-id=node--1} : 3.000
producer-node-metrics:response-total:{client-id=perf-producer-client,
node-id=node-0} : 1.000
producer-node-metrics:response-total:{client-id=perf-producer-client,
node-id=node-2} : 51868.000
producer-topic-metrics:byte-rate:{client-id=perf-producer-client,
topic=test-perf} : 10399481.843
producer-topic-metrics:byte-total:{client-id=perf-producer-client,
topic=test-perf} : 516063887.000
producer-topic-metrics:compression-rate:{client-id=perf-producer-client,
topic=test-perf} : 1.000
producer-topic-metrics:record-error-rate:{client-id=perf-producer-client,
topic=test-perf} : 0.000
producer-topic-metrics:record-error-total:{client-id=perf-producer-client,
topic=test-perf} : 0.000
producer-topic-metrics:record-retry-rate:{client-id=perf-producer-client,
topic=test-perf} : 0.000
producer-topic-metrics:record-retry-total:{client-id=perf-producer-client,
topic=test-perf} : 0.000
producer-topic-metrics:record-send-rate:{client-id=perf-producer-client,
topic=test-perf} : 2015.154
producer-topic-metrics:record-send-total:{client-id=perf-producer-client,
topic=test-perf} : 100000.000
I have no name!@kafka-0:/opt/kafka/bin$ {code}
# _Do consumer performance test without latency to the kafka cluster_
{code:java}
I have no name!@kafka-2:/opt/kafka/bin$ ./kafka-consumer-perf-test.sh
--bootstrap-server 10.193.27.216:10092 --topic test-perf --socket-buffer-size
8388608 --print-metrics --messages 100000 --fetch-size 8388608
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg,
nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2024-11-08 07:43:24:389, 2024-11-08 07:43:27:466, 488.2813, 158.6874, 100000,
32499.1875, 375, 2702, 180.7110, 37009.6225
Metric Name
Value
consumer-coordinator-metrics:assigned-partitions:{client-id=perf-consumer-client}
: 0.000
consumer-coordinator-metrics:commit-latency-avg:{client-id=perf-consumer-client}
: 4.000
consumer-coordinator-metrics:commit-latency-max:{client-id=perf-consumer-client}
: 4.000
consumer-coordinator-metrics:commit-rate:{client-id=perf-consumer-client}
: 0.033
consumer-coordinator-metrics:commit-total:{client-id=perf-consumer-client}
: 1.000
consumer-coordinator-metrics:failed-rebalance-rate-per-hour:{client-id=perf-consumer-client}
: 108.176
consumer-coordinator-metrics:failed-rebalance-total:{client-id=perf-consumer-client}
: 1.000
consumer-coordinator-metrics:heartbeat-rate:{client-id=perf-consumer-client}
: 0.000
consumer-coordinator-metrics:heartbeat-response-time-max:{client-id=perf-consumer-client}
: NaN
consumer-coordinator-metrics:heartbeat-total:{client-id=perf-consumer-client}
: 0.000
consumer-coordinator-metrics:join-rate:{client-id=perf-consumer-client}
: 0.030
consumer-coordinator-metrics:join-time-avg:{client-id=perf-consumer-client}
: 4.000
consumer-coordinator-metrics:join-time-max:{client-id=perf-consumer-client}
: 4.000
consumer-coordinator-metrics:join-total:{client-id=perf-consumer-client}
: 1.000
consumer-coordinator-metrics:last-heartbeat-seconds-ago:{client-id=perf-consumer-client}
: -1.000
consumer-coordinator-metrics:last-rebalance-seconds-ago:{client-id=perf-consumer-client}
: 3.000
consumer-coordinator-metrics:partition-assigned-latency-avg:{client-id=perf-consumer-client}
: 0.000
consumer-coordinator-metrics:partition-assigned-latency-max:{client-id=perf-consumer-client}
: 0.000
consumer-coordinator-metrics:partition-lost-latency-avg:{client-id=perf-consumer-client}
: NaN
consumer-coordinator-metrics:partition-lost-latency-max:{client-id=perf-consumer-client}
: NaN
consumer-coordinator-metrics:partition-revoked-latency-avg:{client-id=perf-consumer-client}
: 0.000
consumer-coordinator-metrics:partition-revoked-latency-max:{client-id=perf-consumer-client}
: 0.000
consumer-coordinator-metrics:rebalance-latency-avg:{client-id=perf-consumer-client}
: 35.000
consumer-coordinator-metrics:rebalance-latency-max:{client-id=perf-consumer-client}
: 35.000
consumer-coordinator-metrics:rebalance-latency-total:{client-id=perf-consumer-client}
: 35.000
consumer-coordinator-metrics:rebalance-rate-per-hour:{client-id=perf-consumer-client}
: 108.258
consumer-coordinator-metrics:rebalance-total:{client-id=perf-consumer-client}
: 1.000
consumer-coordinator-metrics:sync-rate:{client-id=perf-consumer-client}
: 0.030
consumer-coordinator-metrics:sync-time-avg:{client-id=perf-consumer-client}
: 11.000
consumer-coordinator-metrics:sync-time-max:{client-id=perf-consumer-client}
: 11.000
consumer-coordinator-metrics:sync-total:{client-id=perf-consumer-client}
: 1.000
consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=perf-consumer-client,
topic=test-perf} : 15500619.541
consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=perf-consumer-client}
: 15500619.541
consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=perf-consumer-client,
topic=test-perf} : 512900000.000
consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=perf-consumer-client}
: 512900000.000
consumer-fetch-manager-metrics:fetch-latency-avg:{client-id=perf-consumer-client}
: 39.333
consumer-fetch-manager-metrics:fetch-latency-max:{client-id=perf-consumer-client}
: 500.000
consumer-fetch-manager-metrics:fetch-rate:{client-id=perf-consumer-client}
: 1.899
consumer-fetch-manager-metrics:fetch-size-avg:{client-id=perf-consumer-client,
topic=test-perf} : 8272580.645
consumer-fetch-manager-metrics:fetch-size-avg:{client-id=perf-consumer-client}
: 8272580.645
consumer-fetch-manager-metrics:fetch-size-max:{client-id=perf-consumer-client,
topic=test-perf} : 8344883.000
consumer-fetch-manager-metrics:fetch-size-max:{client-id=perf-consumer-client}
: 8344883.000
consumer-fetch-manager-metrics:fetch-throttle-time-avg:{client-id=perf-consumer-client}
: 0.000
consumer-fetch-manager-metrics:fetch-throttle-time-max:{client-id=perf-consumer-client}
: 0.000
consumer-fetch-manager-metrics:fetch-total:{client-id=perf-consumer-client}
: 63.000
consumer-fetch-manager-metrics:preferred-read-replica:{client-id=perf-consumer-client,
topic=test-perf, partition=0} : -1
consumer-fetch-manager-metrics:records-consumed-rate:{client-id=perf-consumer-client,
topic=test-perf} : 3022.244
consumer-fetch-manager-metrics:records-consumed-rate:{client-id=perf-consumer-client}
: 3021.970
consumer-fetch-manager-metrics:records-consumed-total:{client-id=perf-consumer-client,
topic=test-perf} : 100000.000
consumer-fetch-manager-metrics:records-consumed-total:{client-id=perf-consumer-client}
: 100000.000
consumer-fetch-manager-metrics:records-lag-avg:{client-id=perf-consumer-client,
topic=test-perf, partition=0} : 49702.008
consumer-fetch-manager-metrics:records-lag-max:{client-id=perf-consumer-client,
topic=test-perf, partition=0} : 99500.000
consumer-fetch-manager-metrics:records-lag-max:{client-id=perf-consumer-client}
: 99500.000
consumer-fetch-manager-metrics:records-lag:{client-id=perf-consumer-client,
topic=test-perf, partition=0} : 0.000
consumer-fetch-manager-metrics:records-lead-avg:{client-id=perf-consumer-client,
topic=test-perf, partition=0} : 50297.992
consumer-fetch-manager-metrics:records-lead-min:{client-id=perf-consumer-client,
topic=test-perf, partition=0} : 500.000
consumer-fetch-manager-metrics:records-lead-min:{client-id=perf-consumer-client}
: 500.000
consumer-fetch-manager-metrics:records-lead:{client-id=perf-consumer-client,
topic=test-perf, partition=0} : 100000.000
consumer-fetch-manager-metrics:records-per-request-avg:{client-id=perf-consumer-client,
topic=test-perf} : 1612.903
consumer-fetch-manager-metrics:records-per-request-avg:{client-id=perf-consumer-client}
: 1612.903
kafka-metrics-count:count:{client-id=perf-consumer-client}
: 61.000
I have no name!@kafka-2:/opt/kafka/bin$ {code}
# _Do consumer performance test with latency(RTT 50ms)to the kafka cluster_
{code:java}
I have no name!@kafka-2:/opt/kafka/bin$ ./kafka-consumer-perf-test.sh
--bootstrap-server 10.193.27.216:10092 --topic test-perf --socket-buffer-size
8388608 --print-metrics --messages 100000 --fetch-size 8388608
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg,
nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2024-11-08 07:38:27:986, 2024-11-08 07:40:40:368, 488.2813, 3.6884, 100000,
755.3897, 903, 131479, 3.7138, 760.5777
Metric Name
Value
consumer-coordinator-metrics:assigned-partitions:{client-id=perf-consumer-client}
: 0.000
consumer-coordinator-metrics:commit-latency-avg:{client-id=perf-consumer-client}
: 64.571
consumer-coordinator-metrics:commit-latency-max:{client-id=perf-consumer-client}
: 96.000
consumer-coordinator-metrics:commit-rate:{client-id=perf-consumer-client}
: 0.123
consumer-coordinator-metrics:commit-total:{client-id=perf-consumer-client}
: 27.000
consumer-coordinator-metrics:failed-rebalance-rate-per-hour:{client-id=perf-consumer-client}
: 0.000
consumer-coordinator-metrics:failed-rebalance-total:{client-id=perf-consumer-client}
: 1.000
consumer-coordinator-metrics:heartbeat-rate:{client-id=perf-consumer-client}
: 0.332
consumer-coordinator-metrics:heartbeat-response-time-max:{client-id=perf-consumer-client}
: 64.000
consumer-coordinator-metrics:heartbeat-total:{client-id=perf-consumer-client}
: 43.000
consumer-coordinator-metrics:join-rate:{client-id=perf-consumer-client}
: 0.000
consumer-coordinator-metrics:join-time-avg:{client-id=perf-consumer-client}
: NaN
consumer-coordinator-metrics:join-time-max:{client-id=perf-consumer-client}
: NaN
consumer-coordinator-metrics:join-total:{client-id=perf-consumer-client}
: 1.000
consumer-coordinator-metrics:last-heartbeat-seconds-ago:{client-id=perf-consumer-client}
: 3.000
consumer-coordinator-metrics:last-rebalance-seconds-ago:{client-id=perf-consumer-client}
: 132.000
consumer-coordinator-metrics:partition-assigned-latency-avg:{client-id=perf-consumer-client}
: NaN
consumer-coordinator-metrics:partition-assigned-latency-max:{client-id=perf-consumer-client}
: NaN
consumer-coordinator-metrics:partition-lost-latency-avg:{client-id=perf-consumer-client}
: NaN
consumer-coordinator-metrics:partition-lost-latency-max:{client-id=perf-consumer-client}
: NaN
consumer-coordinator-metrics:partition-revoked-latency-avg:{client-id=perf-consumer-client}
: 0.000
consumer-coordinator-metrics:partition-revoked-latency-max:{client-id=perf-consumer-client}
: 0.000
consumer-coordinator-metrics:rebalance-latency-avg:{client-id=perf-consumer-client}
: NaN
consumer-coordinator-metrics:rebalance-latency-max:{client-id=perf-consumer-client}
: NaN
consumer-coordinator-metrics:rebalance-latency-total:{client-id=perf-consumer-client}
: 363.000
consumer-coordinator-metrics:rebalance-rate-per-hour:{client-id=perf-consumer-client}
: 0.000
consumer-coordinator-metrics:rebalance-total:{client-id=perf-consumer-client}
: 1.000
consumer-coordinator-metrics:sync-rate:{client-id=perf-consumer-client}
: 0.000
consumer-coordinator-metrics:sync-time-avg:{client-id=perf-consumer-client}
: NaN
consumer-coordinator-metrics:sync-time-max:{client-id=perf-consumer-client}
: NaN
consumer-coordinator-metrics:sync-total:{client-id=perf-consumer-client}
: 1.000
consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=perf-consumer-client,
topic=test-perf} : 4091208.379
consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=perf-consumer-client}
: 4091322.799
consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=perf-consumer-client,
topic=test-perf} : 512900000.000
consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=perf-consumer-client}
: 512900000.000
consumer-fetch-manager-metrics:fetch-latency-avg:{client-id=perf-consumer-client}
: 1984.789
consumer-fetch-manager-metrics:fetch-latency-max:{client-id=perf-consumer-client}
: 2383.000
consumer-fetch-manager-metrics:fetch-rate:{client-id=perf-consumer-client}
: 0.531
consumer-fetch-manager-metrics:fetch-size-avg:{client-id=perf-consumer-client,
topic=test-perf} : 8127185.444
consumer-fetch-manager-metrics:fetch-size-avg:{client-id=perf-consumer-client}
: 8127185.444
consumer-fetch-manager-metrics:fetch-size-max:{client-id=perf-consumer-client,
topic=test-perf} : 8334625.000
consumer-fetch-manager-metrics:fetch-size-max:{client-id=perf-consumer-client}
: 8334625.000
consumer-fetch-manager-metrics:fetch-throttle-time-avg:{client-id=perf-consumer-client}
: 0.000
consumer-fetch-manager-metrics:fetch-throttle-time-max:{client-id=perf-consumer-client}
: 0.000
consumer-fetch-manager-metrics:fetch-total:{client-id=perf-consumer-client}
: 63.000
consumer-fetch-manager-metrics:preferred-read-replica:{client-id=perf-consumer-client,
topic=test-perf, partition=0} : -1
consumer-fetch-manager-metrics:records-consumed-rate:{client-id=perf-consumer-client,
topic=test-perf} : 797.662
consumer-fetch-manager-metrics:records-consumed-rate:{client-id=perf-consumer-client}
: 797.662
consumer-fetch-manager-metrics:records-consumed-total:{client-id=perf-consumer-client,
topic=test-perf} : 100000.000
consumer-fetch-manager-metrics:records-consumed-total:{client-id=perf-consumer-client}
: 100000.000
consumer-fetch-manager-metrics:records-lag-avg:{client-id=perf-consumer-client,
topic=test-perf, partition=0} : 13969.314
consumer-fetch-manager-metrics:records-lag-max:{client-id=perf-consumer-client,
topic=test-perf, partition=0} : 28022.000
consumer-fetch-manager-metrics:records-lag-max:{client-id=perf-consumer-client}
: 28022.000
consumer-fetch-manager-metrics:records-lag:{client-id=perf-consumer-client,
topic=test-perf, partition=0} : 0.000
consumer-fetch-manager-metrics:records-lead-avg:{client-id=perf-consumer-client,
topic=test-perf, partition=0} : 86030.686
consumer-fetch-manager-metrics:records-lead-min:{client-id=perf-consumer-client,
topic=test-perf, partition=0} : 71978.000
consumer-fetch-manager-metrics:records-lead-min:{client-id=perf-consumer-client}
: 71978.000
consumer-fetch-manager-metrics:records-lead:{client-id=perf-consumer-client,
topic=test-perf, partition=0} : 100000.000
consumer-fetch-manager-metrics:records-per-request-avg:{client-id=perf-consumer-client,
topic=test-perf} : 1584.556
consumer-fetch-manager-metrics:records-per-request-avg:{client-id=perf-consumer-client}
: 1584.556
kafka-metrics-count:count:{client-id=perf-consumer-client}
: 61.000
I have no name!@kafka-2:/opt/kafka/bin$ {code}
*Observations:*
* The consumer rate is decreased from 158.6874 MB.sec to 3.6884 MB.sec.
* Some metrics to compare
{code:java}
with latency
------------
consumer-fetch-manager-metrics:fetch-latency-avg:{client-id=perf-consumer-client}
: 1984.789
consumer-fetch-manager-metrics:fetch-latency-max:{client-id=perf-consumer-client}
: 2383.000
consumer-fetch-manager-metrics:fetch-total:{client-id=perf-consumer-client}
: 63.000
without latency
---------------
consumer-fetch-manager-metrics:fetch-latency-avg:{client-id=perf-consumer-client}
: 39.333
consumer-fetch-manager-metrics:fetch-latency-max:{client-id=perf-consumer-client}
: 500.000
consumer-fetch-manager-metrics:fetch-total:{client-id=perf-consumer-client}
: 63.000 {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
