Hi assaf,

Thanks for trying data source v2! Data source v2 is still evolving(we
marked all the data source v2 interface as @Evolving), and we've already
made a lot of API changes in this release(some renaming, switching to
InternalRow, etc.). So I'd not encourage people to use data source v2 in
long-term productions until we mark data source v2 as stable(or
experimental at least). SPARK-24882 is also an API change, and I'd say
people should implement data source after it gets merged or rejected.

About metrics, it should be easy to add a mixin interface to report metrics.

Thanks,
Wenchen

On Tue, Jul 31, 2018 at 2:07 PM assaf.mendelson <assaf.mendel...@rsa.com>
wrote:

> Hi all,
> I am currently in the middle of developing a new data source (for an
> internal tool) using data source V2.
> I noticed that  SPARK-24882
> <https://issues.apache.org/jira/browse/SPARK-24882>   is planned for 2.4
> and
> includes interface changes.
>
> I was wondering if those are planned in addition to the current interfaces
> or are aimed to replace them (specifically the most basic reading as this
> is
> what I am using).
>
> As a side note, I was wondering if there is any means to expose metrics
> from
> the data source, e.g. I would like to expose a metric of the number of rows
> read to the application (currently I am adding a per partition index column
> and doing a custom idempotent accumulator which collects the maximum index
> for each partition).
>
> Thanks,
>     Assaf.
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Reply via email to