xudong963 opened a new issue, #24647: URL: https://github.com/apache/datafusion/issues/24647
### Describe the bug A Parquet file with exactly zero rows can prevent DataFusion from using the ordering of the non-empty files in the same listing table. An empty file contributes no rows, so it cannot violate any ordering. However, an empty Parquet file normally has no `sorting_columns` ordering and no column min/max values. This currently has two user-visible effects: 1. `derive_common_ordering_from_files` treats the empty file's `None` ordering as incompatible with the ordering inferred from non-empty files. The scan loses its output ordering and can retain an unnecessary `SortExec`. 2. With `datafusion.execution.split_file_groups_by_statistics = true`, `MinMaxStatistics::new_from_files` attempts to read min/max values from the empty file and planning fails with `statistics not found`. ### To reproduce Create two or more non-overlapping Parquet files using `COPY (... ORDER BY key)`, then add a schema-only file to the same directory: ```sql SET datafusion.execution.collect_statistics = true; SET datafusion.execution.split_file_groups_by_statistics = true; CREATE TABLE source(key INT) AS VALUES (1), (2), (3), (4); COPY (SELECT * FROM source WHERE key <= 2 ORDER BY key) TO 'sorted/data1.parquet' STORED AS PARQUET; COPY (SELECT * FROM source WHERE key > 2 ORDER BY key) TO 'sorted/data2.parquet' STORED AS PARQUET; COPY (SELECT * FROM source WHERE FALSE) TO 'sorted/empty.parquet' STORED AS PARQUET; CREATE EXTERNAL TABLE sorted(key INT) STORED AS PARQUET LOCATION 'sorted/'; EXPLAIN SELECT * FROM sorted ORDER BY key; ``` The non-empty file has Parquet ordering metadata and min/max statistics. The empty file has `num_rows = Precision::Exact(0)`, but no ordering or column min/max values. ### Expected behavior Files with `num_rows == Precision::Exact(0)` should be ignored when deriving and validating file ordering and when constructing min/max statistics for file grouping. The scan should retain the ordering proven by the non-empty files, and statistics-based grouping should not fail. Files whose row count is absent or inexact should remain conservative; only exact zero-row statistics are sufficient to ignore a file. The row-count statistics are already collected from the Parquet footer during listing, so this requires no additional I/O. ### Additional context This affects both explicitly declared file ordering and ordering inferred from Parquet `sorting_columns` metadata. A regression test can cover a directory containing ordered non-empty Parquet files plus a zero-row file without ordering metadata, with statistics-based file-group splitting enabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
