Hi all,
I am currently in the middle of developing a new data source (for an
internal tool) using data source V2.
I noticed that SPARK-24882
<https://issues.apache.org/jira/browse/SPARK-24882> is planned for 2.4 and
includes interface changes.
I was wondering if those are planned in addition to the current interfaces
or are aimed to replace them (specifically the most basic reading as this is
what I am using).
As a side note, I was wondering if there is any means to expose metrics from
the data source, e.g. I would like to expose a metric of the number of rows
read to the application (currently I am adding a per partition index column
and doing a custom idempotent accumulator which collects the maximum index
for each partition).
Thanks,
Assaf.
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
