dongjoon-hyun opened a new pull request, #59784:
URL: https://github.com/apache/spark/pull/59784
### What changes were proposed in this pull request?
This PR makes `CreateHiveTableAsSelectCommand` run the dynamic partition
checks of the Hive write on the new table before it drops the existing table
for `saveAsTable` with `SaveMode.Overwrite` and `format("hive")`.
`CreateHiveTableAsSelectCommand` now mixes in `V1WritesHiveUtils` and calls
`getDynamicPartitionColumns` with the new table definition and the partition
spec of the CTAS, right after `validateHiveSerdeTable` (SPARK-60515) and before
the drop. `InsertIntoHiveTable` calls the same method for the table that the
command creates, so the checks and their errors are the same.
### Why are the changes needed?
Since SPARK-60515, `CreateHiveTableAsSelectCommand` runs the checks of Hive
on the new table before it drops the existing table. However, the dynamic
partition checks in `V1WritesHiveUtils.getDynamicPartitionColumns` still run
later, when the command builds the `InsertIntoHiveTable` for the table that it
has just created. Under the default `hive.exec.dynamic.partition.mode=strict`,
or with `hive.exec.dynamic.partition=false`, a partitioned overwrite fails
there, and the existing table, and the data of a managed table, are already
gone.
```scala
spark.range(3).write.format("hive").saveAsTable("t")
spark.range(5).selectExpr("id", "id % 2 AS
p").write.format("hive").partitionBy("p").mode("overwrite").saveAsTable("t")
// SparkException: Dynamic partition strict mode requires at least one
static partition column. To turn this off set
hive.exec.dynamic.partition.mode=nonstrict
spark.catalog.tableExists("t")
// false
```
It happens with `option("fileFormat", "parquet")` too, because
`RelationConversions` converts only a non-partitioned Hive CTAS to a data
source write. The checks only need the partition columns of the new table and
the Hive conf, so they can run before the drop.
@peter-toth reported this in the review of #59757
(https://github.com/apache/spark/pull/59757#discussion_r4370977080).
### Does this PR introduce _any_ user-facing change?
Yes. When the dynamic partition checks fail, such an overwrite keeps the
existing table and its data. The error is the same as before. Valid overwrites
are unchanged.
### How was this patch tested?
Added a test to `MetastoreDataSourcesSuite` for
`hive.exec.dynamic.partition.mode=strict` and
`hive.exec.dynamic.partition=false` with the `textfile` and `parquet` file
formats. It fails without this fix. The existing test `create partitioned hive
serde table as select` in `HiveDDLSuite` covers a valid partitioned overwrite
of an existing table.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
