Re: Datasource V2 support in Spark 3.x

2020-03-05 Thread Wenchen Fan
Data Source V2 has evolved to Connector API which supports both data (the data source API) and metadata (the catalog API). The new APIs are under package org.apache.spark.sql.connector You can keep using Data Source V1 as there is no plan to deprecate it in the near future. But if you'd like to

Datasource V2 support in Spark 3.x

2020-02-28 Thread Mihir Sahu
Hi Team, Wanted to know ahead of developing new datasource for Spark 3.x. Shall it be done using Datasource V2 or Datasource V1(via Relation) or there is any other plan. When I tried to build datasource using V2 for Spark 3.0, I could not find the associated classes and they seems to be