Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21721 My 2 cents, the root reason is the lifecycle of reporting query progress is tied to `finishTrigger` and we read updated metrics from executed plan which continuous mode doesn't have both `finishTrigger` as well as finished plan to be executed. I'm not aware of how/when updated information of nodes of physical plan are transmitted from executor to the driver, but we should avoid using executed plan as a source to read information, and find alternative to be compatible between micro-batch and continuous mode. It doesn't apply only metrics but also watermarks. I'm not sure it is viable, but It could be via RPC or whatever once we can aggregate the information from driver. Then each operators can send information on driver directly and driver can aggregate them and utilize once a batch or an epoch is finished.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org