GitHub user rdblue opened a pull request:
https://github.com/apache/spark/pull/20488
[SPARK-23321][SQL]: Validate datasource v2 writes
## What changes were proposed in this pull request?
DataSourceV2 does not currently apply any validation rules when writing.
Other write paths attempt to validate that a data frame can be written to a
target table or path and these changes add the same logic to v2.
This updates the logical plan to use InsertIntoTable and applies the insert
preprocess rules to writes. It also adds a conversion rule from InserIntoTable
to DataSourceV2Write because InsertIntoTable cannot be used in logical plans
after analysis.
InsertIntoTable is not necessarily the right logical plan. It assumes that
the table exists and can report its schema.
## How was this patch tested?
Added a test that fails analysis in the preprocess rule.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rdblue/spark
SPARK-23321-validate-datasource-v2-writes
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20488.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20488
commit 62c569672083c0fa633da1d6edaba40d0bb05819
Author: Ryan Blue
Date: 2018-01-17T21:58:12Z
SPARK-22386: DataSourceV2: Use immutable logical plans.
commit f0bd45d3c931941b8092cdac738cb29954e0acdd
Author: Ryan Blue
Date: 2018-01-24T19:34:42Z
SPARK-23203: Fix scala style check.
commit 2fdeb4556cd22a092630b341a22a16a59e377183
Author: Ryan Blue
Date: 2018-01-24T19:54:10Z
SPARK-23203: Fix Kafka tests, use StreamingDataSourceV2Relation.
This also removes unused imports.
commit ab945a19efe666c41deae9c044002f3455220c1d
Author: Ryan Blue
Date: 2018-02-02T20:30:33Z
SPARK-23204: DataFrameReader: Remove v2 table identifier parsing.
commit 3580daf15497a1d49112a0eddd556f74b9b3e280
Author: Ryan Blue
Date: 2018-02-02T19:04:23Z
SPARK-23321: Apply preprocess insert rules to DataSourceV2.
This updates the DataSourceV2 write path to use DataSourceV2Relation and
InsertIntoTable to apply the insert preprocess rules.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
