malinjawi opened a new pull request, #13390: URL: https://github.com/apache/gluten/pull/13390
## What changes are proposed in this pull request? Spark has the first task of an empty write job write a file that carries the schema. The native Delta writer did the same, but the Velox Parquet writer writes no footer when it gets no rows. So that task left a zero-byte file that no Parquet reader can open. It happens on any native Delta write whose first task gets no rows, such as an insert from a query that returns nothing. When Delta has the file's row count, it drops the file from the commit, and the file stays in the table directory as an orphan. Spark's reader still fails on it when it lists the directory, which is why Delta Lake's `RowTrackingReadWriteSuite#write and read table with all-null materialized columns` fails. With `spark.databricks.delta.stats.collect=false`, Delta has no row count and keeps the file, so the table gets a data file that cannot be read (from code). Delta keeps the schema in its log, so `GlutenDeltaFileFormatWriter` now writes no file for a task without rows. Spark already does this for every task but the first. Removes `RowTrackingReadWriteSuite#write and read table with all-null materialized columns` from the Delta Lake test baseline. ## How was this patch tested? `VeloxDeltaEmptyWriteSuite` writes an input without partitions and an input whose partitions are all empty, with statistics on and off. It then checks that: - no Parquet file in the table directory or the Delta log is empty; - Delta and Spark's Parquet reader both read the table. The suite is in the Delta 3.3 tree and linked into the Delta 4 tree. It loads the native library, so it runs in CI. Compiled main and test code on Spark 3.5 and 4.1. cc @rui-mo @felipepessoto ## Was this patch authored or co-authored using generative AI tooling? Generated-by: IBM Bob -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
