andygrove opened a new issue, #5303:
URL: https://github.com/apache/datafusion-comet/issues/5303
Found while reviewing #5293 against the Iceberg split-writer work in #4658.
### Describe the bug
Spark's `WriteFilesExec.doExecuteWrite` guards against a child RDD with zero
partitions (SPARK-23271):
```scala
val rddWithNonEmptyPartitions = if (rdd.partitions.length == 0) {
session.sparkContext.parallelize(Array.empty[InternalRow], 1)
} else {
rdd
}
```
Without it no write task runs at all, so the output directory ends up with
no data file — not even the schema-only file Spark produces. `IcebergWriteExec`
added in #4658 carries the same guard.
`CometWriteFilesExec.doExecuteWrite` has no equivalent: it calls
`child.executeColumnar()` and goes straight to `mapPartitionsInternal`. With
zero partitions, `FileFormatWriter.writeAndCommit` collects zero
`WriteTaskResult`s and commits an empty job. The result is a `_SUCCESS` marker
over an empty directory, where Spark writes one file carrying the schema.
Note this is not a regression from #5293 — the `CometNativeWriteExec` it
replaced had the same hole.
### Steps to reproduce
Any write whose input plan yields a zero-partition RDD (for example a scan
over an empty relation), with `spark.comet.parquet.write.enabled=true`.
### Expected behavior
Match Spark: run one write task so the output carries the schema.
Harder here than in the Iceberg case because the dummy RDD must be columnar
— `parallelize(Array.empty[InternalRow], 1)` is row-based, and
`CometWriteFilesExec` feeds batches into an Arrow stream. Options are a
single-partition empty `ColumnarBatch` RDD, or bypassing the native writer for
this case and writing the schema-only file on the JVM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
