bjkonglu created SPARK-26135: -------------------------------- Summary: Structured Streaming reporting metrics programmatically using asynchronous APIs can't get all queries metrics Key: SPARK-26135 URL: https://issues.apache.org/jira/browse/SPARK-26135 Project: Spark Issue Type: Improvement Components: Structured Streaming Affects Versions: 2.3.1 Environment: h3.
Reporter: bjkonglu h3. Background When I use Structured Streaming handle real-time data, I also want to know the streaming application metrics, for example prcessedRowsPerSecond、inputRowsPerSeconds etc. So I report metrics programmatically using asynchronous APIs. {code:java} val spark: SparkSession = ... spark.streams.addListener(new StreamingQueryListener() { override def onQueryStarted(queryStarted: QueryStartedEvent): Unit = { println("Query started: " + queryStarted.id) } override def onQueryTerminated(queryTerminated: QueryTerminatedEvent): Unit = { println("Query terminated: " + queryTerminated.id) } override def onQueryProgress(queryProgress: QueryProgressEvent): Unit = { println("Query made progress: " + queryProgress.progress) } }) {code} h3. Questions When the streaming application has a single query, asynchronous APIs work well. But when the streaming application has many queries, asynchronous APIs can't report metrics exactly, some queries can report well, some queries report delay and metrics number lower. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org