Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21721 Since I'm continuously working on data source v2 API, this gets my attention. Do we have a story for the metrics in data source v2 streaming API? It's weird to me that we add public APIs that only work for micro-batch. For streaming API, the abstraction is that, we have a logical scan for a streaming source in a query(to keep query specific states like offsets), and a physical scan to do the actual job, for each micro-batch or for the entire continuous query(if `needsReconfigure` is false). Where does the metrics fit in with the abstraction? It's ok that it only works for micro-batch now, but we must have a clear plan that we can and will make it work for continuous. cc @tdas @zsxwing @rdblue
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org