Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21469 @arunmahadevan Added custom metrics in state store to streaming query status as well. You can see `providerLoadedMapSize` is added to `stateOperators.customMetrics` in below output. I have to exclude `providerLoadedMapCountOfVersions` from the list, since average metric is implemented a bit tricky and doesn't look like easy to aggregate for streaming query status. We may want to reimplement SQLMetric and subclasses to make sure everything works correctly without any tricky approach, but that doesn't look like trivial to address and I think this is out of scope on this PR. ``` 18/06/06 22:51:23 INFO MicroBatchExecution: Streaming query made progress: { "id" : "7564a0b7-e3b2-4d53-b246-b774ab04e586", "runId" : "8dd34784-080c-4f86-afaf-ac089902252d", "name" : null, "timestamp" : "2018-06-06T13:51:15.467Z", "batchId" : 4, "numInputRows" : 547, "inputRowsPerSecond" : 67.15776550030694, "processedRowsPerSecond" : 65.94333936106088, "durationMs" : { "addBatch" : 7944, "getBatch" : 1, "getEndOffset" : 0, "queryPlanning" : 61, "setOffsetRange" : 5, "triggerExecution" : 8295, "walCommit" : 158 }, "eventTime" : { "avg" : "2018-06-06T13:51:10.313Z", "max" : "2018-06-06T13:51:14.250Z", "min" : "2018-06-06T13:51:07.098Z", "watermark" : "2018-06-06T13:50:36.676Z" }, "stateOperators" : [ { "numRowsTotal" : 20, "numRowsUpdated" : 16, "memoryUsedBytes" : 26679, "customMetrics" : { "providerLoadedMapSize" : 181911 } } ], "sources" : [ { "description" : "KafkaV2[Subscribe[apachelogs-v2]]", "startOffset" : { "apachelogs-v2" : { "2" : 489056, "4" : 489053, "1" : 489055, "3" : 489051, "0" : 489053 } }, "endOffset" : { "apachelogs-v2" : { "2" : 489056, "4" : 489053, "1" : 489055, "3" : 489051, "0" : 489053 } }, "numInputRows" : 547, "inputRowsPerSecond" : 67.15776550030694, "processedRowsPerSecond" : 65.94333936106088 } ], "sink" : { "description" : "org.apache.spark.sql.execution.streaming.ConsoleSinkProvider@60999714" } } ```
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org