[ https://issues.apache.org/jira/browse/KAFKA-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744347#comment-17744347 ]
Lixin Yao commented on KAFKA-15214: ----------------------------------- Here is one of the example scenarios motivates me to create this request. When I am testing the tiered storage feature, I noticed unbalanced byte out traffic rate across brokers. From the available metrics, it's very confusing because all the topic partitions should be balanced across cluster without any skew. I then check the existing error metrics RemoteReadErrorsPerSec, there is only 0 value reported. This confuse me with the impression that there is no problem on remote downloading. so I have to deep dive into logs for more information. At the end, what I find is one broker is not able to fetch remote segments because of this OffsetOutOfRangeException exception consistently happening repeatedly. Example error looks like this: {code:java} 2023-07-15 00:12:42,471 kafkaLogLevel="INFO" [RemoteLogReader-2]: OffsetOutOfRangeException occurred while reading the remote data for mytopic-247: org.apache.kafka.common.errors.OffsetOutOfRangeException: Received request for offset 0 for leader epoch 0 and partition mytopic-247 which does not exist in remote tier. Try again later. kafkaLoggerClass="kafka.log.remote.RemoteLogReader" kafkaLoggerThread="RemoteLogReader-2" {code} Like I said why this partition is requesting for offset 0 repeatedly could be due to other reason, e.g. corrupted metadata or other issues, but if this error is included as part of RemoteReadErrorsPerSec metrics, it could help me a lot on identifying the root cause and setup alerting. Hope this makes sense to you. I am ok to include this as a tag on existing metrics. As long as I have a way to quickly identify and alert on the abnormal behavior, I am ok with it. > Add metrics for OffsetOutOfRangeException when tiered storage is enabled > ------------------------------------------------------------------------ > > Key: KAFKA-15214 > URL: https://issues.apache.org/jira/browse/KAFKA-15214 > Project: Kafka > Issue Type: Improvement > Components: metrics > Affects Versions: 3.6.0 > Reporter: Lixin Yao > Priority: Minor > Labels: KIP-405 > Fix For: 3.6.0 > > > In the current metrics RemoteReadErrorsPerSec, the exception type > OffsetOutOfRangeException is not included. > In our testing with tiered storage feature (at Apple), we noticed several > cases where remote download is affected and stuck due to repeatedly > OffsetOutOfRangeException in some particular broker or topic partitions. The > root cause could be various but currently without a metrics it's very hard to > catch this issue and debug in a timely fashion. It's understandable that the > exception itself could not be the root cause but this exception metric could > be a good metrics for us to alert and investigate. > Related discussion > [https://github.com/apache/kafka/pull/13944#discussion_r1266243006] > I am happy to contribute to this if the request is agreed. -- This message was sent by Atlassian Jira (v8.20.10#820010)