dongjoon-hyun commented on code in PR #59757:
URL: https://github.com/apache/spark/pull/59757#discussion_r4368296458
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala:
##########
@@ -306,6 +306,16 @@ private[spark] class HiveExternalCatalog(conf: SparkConf,
hadoopConf: Configurat
}
}
+ /**
+ * Runs the checks that Hive runs on a new Hive serde table before creating
it in Hive
+ * metastore, e.g., the SerDe initialization with the table columns, without
creating it.
+ * A command that replaces an existing table calls this before dropping the
existing table.
+ */
+ private[hive] def validateHiveSerdeTable(tableDefinition: CatalogTable):
Unit = withClient {
+ // Like `createTable`, remove the collations, which Hive doesn't support.
+ client.validateTable(tableDefinition.copy(schema =
removeCollation(tableDefinition.schema)))
Review Comment:
Thank you. You're right. In 415dc8789ad, I added a test case which
overwrites an existing Hive serde table through `saveAsTable` with a collated
string column, and checks the new rows and the collated type. Without the
`removeCollation` call, `validateTable` rejects the new table of this test with
`Error: type expected at the position 7 of 'bigint:string collate UNICODE_CI'
but 'string collate UNICODE_CI' is found.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
