This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new f83053c [SPARK-34599][SQL] Fix the issue that INSERT INTO OVERWRITE
doesn't support partition columns containing dot for DSv2
f83053c is described below
commit f83053c2e2facd27f89744f73af6d80d05cc0cf9
Author: Shixiong Zhu <zsxw...@gmail.com>
AuthorDate: Thu Mar 4 15:12:53 2021 +0800
[SPARK-34599][SQL] Fix the issue that INSERT INTO OVERWRITE doesn't support
partition columns containing dot for DSv2
### What changes were proposed in this pull request?
`ResolveInsertInto.staticDeleteExpression` should use
`UnresolvedAttribute.quoted` to create the delete expression so that we will
treat the entire `attr.name` as a column name.
### Why are the changes needed?
When users use `dot` in a partition column name, queries like ```INSERT
OVERWRITE $t1 PARTITION (`a.b` = 'a') (`c.d`) VALUES('b')``` is not working.
### Does this PR introduce _any_ user-facing change?
Without this test, the above query will throw
```
[info] org.apache.spark.sql.AnalysisException: cannot resolve '`a.b`'
given input columns: [a.b, c.d];
[info] 'OverwriteByExpression RelationV2[a.b#17, c.d#18] default.tbl, ('a.b
<=> cast(a as string)), false
[info] +- Project [a.b#19, ansi_cast(col1#16 as string) AS c.d#20]
[info] +- Project [cast(a as string) AS a.b#19, col1#16]
[info] +- LocalRelation [col1#16]
```
With the fix, the query will run correctly.
### How was this patch tested?
The new added test.
Closes #31713 from zsxwing/SPARK-34599.
Authored-by: Shixiong Zhu <zsxw...@gmail.com>
Signed-off-by: Wenchen Fan <wenc...@databricks.com>
(cherry picked from commit 53e4dba7c489ac5c0ad61f0121c4e247de5b485c)
Signed-off-by: Wenchen Fan <wenc...@databricks.com>
---
.../org/apache/spark/sql/catalyst/analysis/Analyzer.scala | 4 +++-
.../scala/org/apache/spark/sql/connector/InsertIntoTests.scala | 10 ++++++++++
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index f71139f..771b817 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -1336,7 +1336,9 @@ class Analyzer(override val catalogManager:
CatalogManager)
// ResolveOutputRelation runs, using the query's column names
that will match the
// table names at that point. because resolution happens after a
future rule, create
// an UnresolvedAttribute.
- EqualNullSafe(UnresolvedAttribute(attr.name),
Cast(Literal(value), attr.dataType))
+ EqualNullSafe(
+ UnresolvedAttribute.quoted(attr.name),
+ Cast(Literal(value), attr.dataType))
case None =>
throw QueryCompilationErrors.unknownStaticPartitionColError(name)
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/InsertIntoTests.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/InsertIntoTests.scala
index 2cc7a1f..ad73037 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/InsertIntoTests.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/InsertIntoTests.scala
@@ -477,5 +477,15 @@ trait InsertIntoSQLOnlyTests
verifyTable(t1, spark.table(view))
}
}
+
+ test("SPARK-34599: InsertInto: overwrite - dot in the partition column
name - static mode") {
+ import testImplicits._
+ val t1 = "tbl"
+ withTable(t1) {
+ sql(s"CREATE TABLE $t1 (`a.b` string, `c.d` string) USING $v2Format
PARTITIONED BY (`a.b`)")
+ sql(s"INSERT OVERWRITE $t1 PARTITION (`a.b` = 'a') (`c.d`)
VALUES('b')")
+ verifyTable(t1, Seq("a" -> "b").toDF("id", "data"))
+ }
+ }
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
