dongjoon-hyun commented on a change in pull request #26929: [SPARK-30289][SQL]
DSv2's partitioning should not accept nested columns
URL: https://github.com/apache/spark/pull/26929#discussion_r359113057
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/FileTableSuite.scala
##########
@@ -85,4 +86,35 @@ class FileTableSuite extends QueryTest with
SharedSparkSession {
assert(table.dataSchema == expectedDataSchema)
}
}
+
+ test("SPARK-30289: DSv2 `FileTable` partitioning should not accept nested
columns") {
+ val nestedSchema = StructType(Array(
+ StructField("nested", StructType(Array(
+ StructField("id", IntegerType, true),
+ StructField("data", StringType, true)
+ )), true)
+ ))
+
+ val e = intercept[AnalysisException] {
+ new FileTable(spark, CaseInsensitiveStringMap.empty(), Seq(),
Some(nestedSchema)) {
+ override def partitioning: Array[Transform] =
+ Array(IdentityTransform(FieldReference(Seq("nested", "id"))))
Review comment:
Shall we update like the following? Then, we can remove the rest (line 102 ~
115).
```scala
- new FileTable(spark, CaseInsensitiveStringMap.empty(), Seq(),
Some(nestedSchema)) {
+ new DummyFileTable(spark, CaseInsensitiveStringMap.empty(), Seq(),
nestedSchema, None) {
override def partitioning: Array[Transform] =
Array(IdentityTransform(FieldReference(Seq("nested", "id"))))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
