Hi Devs, Currently DS V2 does not update any input metrics. SPARK-30362 aims at solving this problem.
We can have the below approach. Have marker interface let's say "ReportMetrics" If the DataSource Implements this interface, then it will be easy to collect the metrics. For e.g. FilePartitionReaderFactory can support metrics. So it will be easy to collect the metrics if FilePartitionReaderFactory implements ReportMetrics Please let me know the views, or even if we want to have new solution or design.
