GitHub user arunmahadevan opened a pull request:
https://github.com/apache/spark/pull/21721
[SPARK-24748][SS] Support for reporting custom metrics via StreamingQuery
Progress
## What changes were proposed in this pull request?
Currently the Structured Streaming sources and sinks does not have a way to
report custom metrics. Providing an option to report custom metrics and making
it available via Streaming Query progress can enable sources and sinks to
report custom progress information (E.g. the lag metrics for Kafka source).
Similar metrics can be reported for Sinks as well, but would like to get
initial feedback before proceeding further.
## How was this patch tested?
New and existing unit tests.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/arunmahadevan/spark SPARK-24748
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21721.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21721
----
commit b7b2c3b1c9242fe205869f108548248f71ff8203
Author: Arun Mahadevan <arunm@...>
Date: 2018-07-06T01:51:50Z
[SPARK-24748][SS] Support for reporting custom metrics via Streaming Query
Progress
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
