[ https://issues.apache.org/jira/browse/SPARK-24863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shixiong Zhu reopened SPARK-24863: ---------------------------------- Assignee: (was: Arun Mahadevan) > Report offset lag as a custom metrics for Kafka structured streaming source > --------------------------------------------------------------------------- > > Key: SPARK-24863 > URL: https://issues.apache.org/jira/browse/SPARK-24863 > Project: Spark > Issue Type: Sub-task > Components: Structured Streaming > Affects Versions: 2.4.0 > Reporter: Arun Mahadevan > Priority: Major > > We can build on top of SPARK-24748 to report offset lag as a custom metrics > for Kafka structured streaming source. > This is the difference between the latest offsets in Kafka the time the > metrics is reported (just after a micro-batch completes) and the latest > offset Spark has processed. It can be 0 (or close to 0) if spark keeps up > with the rate at which messages are ingested into Kafka topics in steady > state. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org