Hi all, I am currently in the middle of developing a new data source (for an internal tool) using data source V2. I noticed that SPARK-24882 <https://issues.apache.org/jira/browse/SPARK-24882> is planned for 2.4 and includes interface changes.
I was wondering if those are planned in addition to the current interfaces or are aimed to replace them (specifically the most basic reading as this is what I am using). As a side note, I was wondering if there is any means to expose metrics from the data source, e.g. I would like to expose a metric of the number of rows read to the application (currently I am adding a per partition index column and doing a custom idempotent accumulator which collects the maximum index for each partition). Thanks, Assaf. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org