Bryan Qiang created SPARK-44932:
-----------------------------------
Summary: Continuous Structured Streaming
Key: SPARK-44932
URL: https://issues.apache.org/jira/browse/SPARK-44932
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 3.4.1
Reporter: Bryan Qiang
Hello, we've been running spark continuous structured streaming on standalone
cluster and happy with the performance however we noticed streaming metrics
like input rate and process rate are not updated by the `ProgressReporter` in
`ContinuousExecution` because the
[`finishTrigger`|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala#L293]
function is never invoked in `ContinuousExecution`. I'm wondering why and how
may get metrics like in micro-batch structured streaming.
!https://preview.redd.it/mzh4oc0cbojb1.png?width=1901&format=png&auto=webp&s=8d649ae515e6adb7d6ce853802e0a4134c9fa277!!https://preview.redd.it/8ou3uyuofojb1.png?width=1523&format=png&auto=webp&s=ac6bf7fa05cb90b09cb10b9ac815f64b5e97175e!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
