viirya opened a new issue, #6820:
URL: https://github.com/apache/datafusion-comet/issues/6820
### Describe the bug
On Spark 3.x, `CometDataWritingCommand` replaces the whole
`DataWritingCommandExec(InsertIntoHadoopFsRelationCommand)` with
`CometNativeWriteExec`. That exec runs the job and task commit lifecycle, but
none of the work `InsertIntoHadoopFsRelationCommand.run` does around
`FileFormatWriter.write`:
- after the commit: `fileIndex.foreach(_.refresh())`,
`cacheManager.recacheByPath(...)`, and `CommandUtils.updateTableStats(...)` for
catalog tables
- before writing: `SchemaUtils.checkColumnNameDuplication(outputColumnNames,
...)`
With `spark.comet.parquet.write.enabled=true` and the operator's
allowIncompatible config set:
1. `val df = spark.read.parquet(p).cache(); df.count()`, then a native
`write.mode("overwrite").parquet(p)`: `df` still returns the old rows.
2. `CACHE TABLE t`, then a native `INSERT INTO t SELECT ...`:
`spark.table("t")` returns the old rows.
3. A native `INSERT INTO` / `INSERT OVERWRITE` on a non-partitioned parquet
table leaves the table stats unchanged. With
`spark.sql.statistics.size.autoUpdate.enabled=true` they keep the old size;
with it false, stale stats are not cleared.
4. A write whose columns differ only by case (`a`, `A`) succeeds natively,
while Spark fails with `COLUMN_ALREADY_EXISTS`.
#3521 reported the INSERT ... SELECT read-back symptom and was closed by
#5763, which fixed Spark 4.0+ only: there Comet replaces only `WriteFilesExec`
and Spark's command still runs. The Spark 3.x path is still affected.
### Expected behavior
The native write matches Spark: caches and the relation's file listing are
refreshed, table stats are updated or cleared, and duplicate column names are
rejected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
