jzhuge commented on a change in pull request #25330: [SPARK-28565][SQL]
DataFrameWriter saveAsTable support for V2 catalogs
URL: https://github.com/apache/spark/pull/25330#discussion_r311324593
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -648,8 +648,11 @@ class Analyzer(
if catalog.isTemporaryTable(ident) =>
u // temporary views take precedence over catalog table names
- case u @ UnresolvedRelation(CatalogObjectIdentifier(Some(catalogPlugin),
ident)) =>
- loadTable(catalogPlugin,
ident).map(DataSourceV2Relation.create).getOrElse(u)
+ case u @ UnresolvedRelation(CatalogObjectIdentifier(maybeCatalog,
ident)) =>
+ maybeCatalog.orElse(sessionCatalog)
Review comment:
This changes the SELECT path, so more unit tests for table relation are
needed, e.g.:
```
test("Relation: basic") {
val t1 = "tbl"
withTable(t1) {
sql(s"CREATE TABLE $t1 USING $v2Format AS SELECT id, data FROM source")
checkAnswer(spark.table(t1), spark.table("source"))
checkAnswer(sql(s"TABLE $t1"), spark.table("source"))
checkAnswer(sql(s"SELECT * FROM $t1"), spark.table("source"))
}
}
```
IMHO this change v2 session catalog and related tests in
`DataSourceV2SessionDataFrameSuite` can be split to a separate PR with
additional unit tests for SQL SELECT and spark.table().
Another PR to update `ResolveInsertInto` to support v2 session catalog in
SQL INSERT INTO and DFW.insertInto.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
