GitHub user rdblue opened a pull request:
https://github.com/apache/spark/pull/21877
[SPARK-24923][SQL][WIP] Add unpartitioned CTAS and RTAS support for
DataSourceV2
## What changes were proposed in this pull request?
* Remove extends from `ReadSupport` and `WriteSupport` classes for use with
`Table`
* Add CTAS and RTAS logical plans
* Refactor physical write plans so AppendData, CTAS, and RTAS use the same
base class
* Add support for `TableCatalog` to `DataFrameReader` and `DataFrameWriter`
* Add `TableV2Relation` for tables that are loaded by `TableCatalog` and
have no `DataSource` instance
* Move implicit helpers into `DataSourceV2Implicits` to avoid future churn
Note that this doesn't handle `partitionBy` in `DataFrameWriter`. Adding
support for partitioned tables will require validation rules.
This is based on unmerged work and includes the commits from #21306 and
#21305.
## How was this patch tested?
Adding unit tests for CTAS and RTAS.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rdblue/spark add-ctas-rtas-v2-plans
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21877.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21877
----
commit 8372f5bb47a0d6269bb16b3dc16f6f3278d2f5fd
Author: Ryan Blue <blue@...>
Date: 2018-05-05T01:13:01Z
SPARK-24252: Add v2 data source mix-in for catalog support.
commit 1238af73872b0105d0c5dfbbd8da5c8f18afe408
Author: Ryan Blue <blue@...>
Date: 2018-05-07T15:54:37Z
SPARK-24251: Add AppendData logical plan.
This adds a new logical plan, AppendData, that was proposed in
SPARK-23521. This also adds an analyzer rule to validate data written
with AppendData against the target table. DataFrameWriter is also
updated so that v2 writes use the new AppendData logical plan.
commit d308d3c75f78242c822eab6d11fb651d94f10aa6
Author: Ryan Blue <blue@...>
Date: 2018-07-25T18:11:45Z
Add CTAS and RTAS support.
This uses the catalog API introduced in SPARK-24252 to implement CTAS
and RTAS plans.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
