jzhuge commented on a change in pull request #25330: [SPARK-28565][SQL]
DataFrameWriter saveAsTable support for V2 catalogs
URL: https://github.com/apache/spark/pull/25330#discussion_r311324593
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -648,8 +648,11 @@ class Analyzer(
if catalog.isTemporaryTable(ident) =>
u // temporary views take precedence over catalog table names
- case u @ UnresolvedRelation(CatalogObjectIdentifier(Some(catalogPlugin),
ident)) =>
- loadTable(catalogPlugin,
ident).map(DataSourceV2Relation.create).getOrElse(u)
+ case u @ UnresolvedRelation(CatalogObjectIdentifier(maybeCatalog,
ident)) =>
+ maybeCatalog.orElse(sessionCatalog)
Review comment:
This changes the SELECT path, so more unit tests for table relation are
needed, e.g.:
```
test("Relation: basic") {
val t1 = "tbl"
withTable(t1) {
sql(s"CREATE TABLE $t1 USING $v2Format AS SELECT id, data FROM source")
checkAnswer(spark.table(t1), spark.table("source"))
checkAnswer(sql(s"TABLE $t1"), spark.table("source"))
checkAnswer(sql(s"SELECT * FROM $t1"), spark.table("source"))
}
}
```
IMHO this v2 session catalog support and related tests can be split to a
separate PR with additional unit tests for SQL SELECT and spark.table().
Also another PR to update `ResolveInsertInto` to support v2 session catalog
in SQL INSERT INTO and DFW.insertInto.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
