[jira] [Updated] (FLINK-24542) Expose the freshness metrics for kafka connector
[ https://issues.apache.org/jira/browse/FLINK-24542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-24542: --- Fix Version/s: (was: 1.19.0) > Expose the freshness metrics for kafka connector > > > Key: FLINK-24542 > URL: https://issues.apache.org/jira/browse/FLINK-24542 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.12.2, 1.14.0, 1.13.1 >Reporter: zlzhang0122 >Priority: Major > > When we start a flink job to consume apache kafka, we usually use offsetLag, > which can be calulated by current-offsets minus committed-offsets, but > sometimes the offsetLag is hard to understand, we can hardly to judge wether > the value is normal or not. Kafka have proposed a new metric: freshness(see > [a-guide-to-kafka-consumer-freshness|https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0]). > So we can also expose the freshness metric for kafka connector to improve the > user experience.From this freshness metric, user can easily know wether the > kafka message is backlog and need to deal with it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-24542) Expose the freshness metrics for kafka connector
[ https://issues.apache.org/jira/browse/FLINK-24542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Ge updated FLINK-24542: Fix Version/s: 1.19.0 (was: 1.18.0) > Expose the freshness metrics for kafka connector > > > Key: FLINK-24542 > URL: https://issues.apache.org/jira/browse/FLINK-24542 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.12.2, 1.14.0, 1.13.1 >Reporter: zlzhang0122 >Priority: Major > Fix For: 1.19.0 > > > When we start a flink job to consume apache kafka, we usually use offsetLag, > which can be calulated by current-offsets minus committed-offsets, but > sometimes the offsetLag is hard to understand, we can hardly to judge wether > the value is normal or not. Kafka have proposed a new metric: freshness(see > [a-guide-to-kafka-consumer-freshness|https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0]). > So we can also expose the freshness metric for kafka connector to improve the > user experience.From this freshness metric, user can easily know wether the > kafka message is backlog and need to deal with it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-24542) Expose the freshness metrics for kafka connector
[ https://issues.apache.org/jira/browse/FLINK-24542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-24542: - Fix Version/s: 1.18.0 (was: 1.17.0) > Expose the freshness metrics for kafka connector > > > Key: FLINK-24542 > URL: https://issues.apache.org/jira/browse/FLINK-24542 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.12.2, 1.14.0, 1.13.1 >Reporter: zlzhang0122 >Priority: Major > Fix For: 1.18.0 > > > When we start a flink job to consume apache kafka, we usually use offsetLag, > which can be calulated by current-offsets minus committed-offsets, but > sometimes the offsetLag is hard to understand, we can hardly to judge wether > the value is normal or not. Kafka have proposed a new metric: freshness(see > [a-guide-to-kafka-consumer-freshness|https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0]). > So we can also expose the freshness metric for kafka connector to improve the > user experience.From this freshness metric, user can easily know wether the > kafka message is backlog and need to deal with it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-24542) Expose the freshness metrics for kafka connector
[ https://issues.apache.org/jira/browse/FLINK-24542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingbo Huang updated FLINK-24542: - Fix Version/s: 1.17.0 (was: 1.16.0) > Expose the freshness metrics for kafka connector > > > Key: FLINK-24542 > URL: https://issues.apache.org/jira/browse/FLINK-24542 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.12.2, 1.14.0, 1.13.1 >Reporter: zlzhang0122 >Priority: Major > Fix For: 1.17.0 > > > When we start a flink job to consume apache kafka, we usually use offsetLag, > which can be calulated by current-offsets minus committed-offsets, but > sometimes the offsetLag is hard to understand, we can hardly to judge wether > the value is normal or not. Kafka have proposed a new metric: freshness(see > [a-guide-to-kafka-consumer-freshness|https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0]). > So we can also expose the freshness metric for kafka connector to improve the > user experience.From this freshness metric, user can easily know wether the > kafka message is backlog and need to deal with it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-24542) Expose the freshness metrics for kafka connector
[ https://issues.apache.org/jira/browse/FLINK-24542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-24542: Fix Version/s: 1.16.0 > Expose the freshness metrics for kafka connector > > > Key: FLINK-24542 > URL: https://issues.apache.org/jira/browse/FLINK-24542 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.12.2, 1.14.0, 1.13.1 >Reporter: zlzhang0122 >Priority: Major > Fix For: 1.15.0, 1.16.0 > > > When we start a flink job to consume apache kafka, we usually use offsetLag, > which can be calulated by current-offsets minus committed-offsets, but > sometimes the offsetLag is hard to understand, we can hardly to judge wether > the value is normal or not. Kafka have proposed a new metric: freshness(see > [a-guide-to-kafka-consumer-freshness|https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0]). > So we can also expose the freshness metric for kafka connector to improve the > user experience.From this freshness metric, user can easily know wether the > kafka message is backlog and need to deal with it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24542) Expose the freshness metrics for kafka connector
[ https://issues.apache.org/jira/browse/FLINK-24542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zlzhang0122 updated FLINK-24542: Description: When we start a flink job to consume apache kafka, we usually use offsetLag, which can be calulated by current-offsets minus committed-offsets, but sometimes the offsetLag is hard to understand, we can hardly to judge wether the value is normal or not. Kafka have proposed a new metric: freshness(see [a-guide-to-kafka-consumer-freshness|https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0]). So we can also expose the freshness metric for kafka connector to improve the user experience.From this freshness metric, user can easily know wether the kafka message is backlog and need to deal with it. was: When we start a flink job to consume apache kafka, we usually use offsetLag, which can be calulated by current-offsets minus committed-offsets, but sometimes the offsetLag is hard to understand, we can hardly to judge wether the value is normal or not. Kafka have proposed a new metric: freshness(see [a-guide-to-kafka-consumer-freshness|[https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0])|https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0]),]. So we can also expose the freshness metric for kafka connector to improve the user experience.From this freshness metric, user can easily know wether the kafka message is backlog and need to deal with it. > Expose the freshness metrics for kafka connector > > > Key: FLINK-24542 > URL: https://issues.apache.org/jira/browse/FLINK-24542 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.12.2, 1.14.0, 1.13.1 >Reporter: zlzhang0122 >Priority: Major > Fix For: 1.15.0 > > > When we start a flink job to consume apache kafka, we usually use offsetLag, > which can be calulated by current-offsets minus committed-offsets, but > sometimes the offsetLag is hard to understand, we can hardly to judge wether > the value is normal or not. Kafka have proposed a new metric: freshness(see > [a-guide-to-kafka-consumer-freshness|https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0]). > So we can also expose the freshness metric for kafka connector to improve the > user experience.From this freshness metric, user can easily know wether the > kafka message is backlog and need to deal with it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-24542) Expose the freshness metrics for kafka connector
[ https://issues.apache.org/jira/browse/FLINK-24542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zlzhang0122 updated FLINK-24542: Description: When we start a flink job to consume apache kafka, we usually use offsetLag, which can be calulated by current-offsets minus committed-offsets, but sometimes the offsetLag is hard to understand, we can hardly to judge wether the value is normal or not. Kafka have proposed a new metric: freshness(see [a-guide-to-kafka-consumer-freshness|[https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0])|https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0]),]. So we can also expose the freshness metric for kafka connector to improve the user experience.From this freshness metric, user can easily know wether the kafka message is backlog and need to deal with it. was: When we start a flink job to consume apache kafka, we usually use offsetLag, which can be calulated by current-offsets minus committed-offsets, but sometimes the offsetLag is hard to understand, we can hardly to judge wether the value is normal or not. Kafka have proposed a new metric: freshness(see [a guide to kafka consumer freshness|[https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0])|https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0]),]. So we can also expose the freshness metric for kafka connector to improve the user experience.From this freshness metric, user can easily know wether the kafka message is backlog and need to deal with it. > Expose the freshness metrics for kafka connector > > > Key: FLINK-24542 > URL: https://issues.apache.org/jira/browse/FLINK-24542 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.12.2, 1.14.0, 1.13.1 >Reporter: zlzhang0122 >Priority: Major > Fix For: 1.15.0 > > > When we start a flink job to consume apache kafka, we usually use offsetLag, > which can be calulated by current-offsets minus committed-offsets, but > sometimes the offsetLag is hard to understand, we can hardly to judge wether > the value is normal or not. Kafka have proposed a new metric: freshness(see > [a-guide-to-kafka-consumer-freshness|[https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0])|https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat=timeline=0]),]. > So we can also expose the freshness metric for kafka connector to improve the > user experience.From this freshness metric, user can easily know wether the > kafka message is backlog and need to deal with it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
