dongjoon-hyun opened a new pull request, #59610:
URL: https://github.com/apache/spark/pull/59610
### What changes were proposed in this pull request?
This PR makes a static-mode overwrite delete the existing data only after
the write passes its driver side checks.
`InsertIntoHadoopFsRelationCommand` used to delete the matching output
before calling `FileFormatWriter.write`, which then checks the data schema
(`verifySchema`, `checkFieldNames`) and the write options
(`FileFormat.prepareWrite`). Now `FileFormatWriter.write` takes a
`beforeSetupJob` callback (a no-op by default) that runs right before
`committer.setupJob`, and `InsertIntoHadoopFsRelationCommand` deletes the
matching output there.
### Why are the changes needed?
When these checks fail, the existing data has already been deleted.
```scala
spark.range(3).write.parquet(path)
spark.range(5).write.mode("overwrite").option("compression",
"bogus").parquet(path)
// SparkIllegalArgumentException:
[CODEC_NOT_AVAILABLE.WITH_AVAILABLE_CODECS_SUGGESTION] ...
spark.read.parquet(path)
// AnalysisException: [PATH_NOT_FOUND] Path does not exist: ...
```
The same happens with an unsupported data type (e.g. a map column in CSV),
with `binaryFile`, which does not support writes, and with `INSERT OVERWRITE`
into tables (including converted Hive tables) and directories. This has been
the behavior since Spark 1.4.
### Does this PR introduce _any_ user-facing change?
Yes. A static-mode overwrite that fails these checks keeps the existing data
instead of deleting it. Valid writes are unchanged.
### How was this patch tested?
Added tests to `InsertSuite` of `sql/core` and `sql/hive`. They fail without
this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
