zlzhang0122 created FLINK-24542:
-----------------------------------
Summary: Expose the freshness metrics for kafka connector
Key: FLINK-24542
URL: https://issues.apache.org/jira/browse/FLINK-24542
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Affects Versions: 1.13.1, 1.14.0, 1.12.2
Reporter: zlzhang0122
Fix For: 1.15.0
When we start a flink job to consume apache kafka, we usually use offsetLag,
which can be calulated by current-offsets minus committed-offsets, but
sometimes the offsetLag is hard to understand, we can hardly to judge wether
the value is normal or not. Kafka have proposed a new metric: freshness(see [a
guide to kafka consumer
freshness|[https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat&from=timeline&isappinstalled=0])|https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat&from=timeline&isappinstalled=0]),].
So we can also expose the freshness metric for kafka connector to improve the
user experience.From this freshness metric, user can easily know wether the
kafka message is backlog and need to deal with it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
