dongjoon-hyun opened a new pull request, #59757:
URL: https://github.com/apache/spark/pull/59757
### What changes were proposed in this pull request?
This PR makes `CreateHiveTableAsSelectCommand` run the checks of Hive on the
new table before it drops the existing table for `saveAsTable` with
`SaveMode.Overwrite` and `format("hive")`.
- `HiveClient.validateTable` runs the checks that `Hive.createTable` runs
before calling Hive metastore, i.e., the SerDe initialization with the table
columns and `Table.checkValidity`, without creating the table. Like
`Hive.createTable`, it wraps the errors in `HiveException`.
- `HiveExternalCatalog.validateHiveSerdeTable` calls it after removing the
collations, like `HiveExternalCatalog.createTable`.
- `CreateHiveTableAsSelectCommand` calls it before dropping the existing
table.
### Why are the changes needed?
SPARK-60387 made `saveAsTable` with `SaveMode.Overwrite` drop the existing
table as late as it can. `CreateHiveTableAsSelectCommand` drops it after the
analysis, because it creates the new table before writing. When Hive rejects
the new table there, the existing table, and the data of a managed table, are
already gone.
```scala
sql("CREATE TABLE t (i INT) USING HIVE")
sql("INSERT INTO t VALUES (1)")
sql("SELECT 1 AS i, CAST(NULL AS VARIANT) AS
c").write.format("hive").mode("overwrite").saveAsTable("t")
// AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IllegalArgumentException: Error: type expected at the position 4 of
'int:variant' but 'variant' is found.
spark.table("t")
// AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `t` cannot
be found. Verify the spelling and correctness of the schema and catalog. ...
```
Hive rejects, for example:
- data column types that the Hive SerDe can't parse: `TIME`,
`TIMESTAMP_NTZ`, `TIMESTAMP_NTZ(9)`, `VARIANT`, `GEOMETRY`, `GEOGRAPHY`, and
nested ANSI intervals
- the void type in Parquet and ORC tables
- nested field names like `` `a-b` ``, and `` `a b` `` in Avro tables
- column names that differ only in case when `spark.sql.caseSensitive` is
`true`
This has been the behavior since Apache Spark 2.2.0 (SPARK-19151).
### Does this PR introduce _any_ user-facing change?
Yes. When Hive rejects the new table, such an overwrite keeps the existing
table and its data. The error is the same as before. Valid overwrites are
unchanged. A failure after creating the new table, e.g., in the write, still
loses the existing table.
### How was this patch tested?
Added tests to `MetastoreDataSourcesSuite` and `HiveClientSuite`. The test
in `MetastoreDataSourcesSuite` fails without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
