[ https://issues.apache.org/jira/browse/SPARK-19378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tathagata Das resolved SPARK-19378. ----------------------------------- Resolution: Fixed Fix Version/s: 3.0.0 2.1.1 Issue resolved by pull request 16716 [https://github.com/apache/spark/pull/16716] > StateOperator metrics should still return the total number of rows in state > even if there was no data for a trigger > ------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-19378 > URL: https://issues.apache.org/jira/browse/SPARK-19378 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 2.1.0 > Reporter: Burak Yavuz > Assignee: Burak Yavuz > Fix For: 2.1.1, 3.0.0 > > > If you have a StreamingDataFrame with an aggregation, we report a metric > called stateOperators which consists of a list of data points per aggregation > for our query (With Spark 2.1, only one aggregation is supported). > These data points report: > - numUpdatedStateRows > - numTotalStateRows > If a trigger had no data - therefore was not fired - we return 0 data points, > however we should actually return a data point with > - numTotalStateRows: numTotalStateRows in lastExecution > - numUpdatedStateRows: 0 > This also affects eventTime statistics. We should still provide the min, max, > avg even through the data didn't change. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org