Burak Yavuz created SPARK-19378: ----------------------------------- Summary: StateOperator metrics should still return the total number of rows in state even if there was no data for a trigger Key: SPARK-19378 URL: https://issues.apache.org/jira/browse/SPARK-19378 Project: Spark Issue Type: Bug Components: Structured Streaming Affects Versions: 2.1.0 Reporter: Burak Yavuz Assignee: Burak Yavuz
If you have a StreamingDataFrame with an aggregation, we report a metric called stateOperators which consists of a list of data points per aggregation for our query (With Spark 2.1, only one aggregation is supported). These data points report: - numUpdatedStateRows - numTotalStateRows If a trigger had no data - therefore was not fired - we return 0 data points, however we should actually return a data point with - numTotalStateRows: numTotalStateRows in lastExecution - numUpdatedStateRows: 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org