xudong963 opened a new pull request, #24648: URL: https://github.com/apache/datafusion/pull/24648
## Which issue does this PR close? - Closes #24647. ## Rationale for this change A file with exactly zero rows cannot violate a sort order, but schema-only Parquet files normally have neither `sorting_columns` ordering nor column min/max values. Treating those missing values as evidence against the non-empty files can add an unnecessary `SortExec`; with statistics-based file-group splitting enabled, it can also fail planning with `statistics not found`. The Parquet footer row count is already available during listing, so exact empty files can be ignored without additional I/O. Files with absent or inexact row counts remain conservative. ## What changes are included in this PR? - Ignore files with `num_rows == Precision::Exact(0)` when deriving a common file ordering. - Exclude exact empty files from `MinMaxStatistics` and sort-column NULL checks. - Preserve the mapping from min/max statistics rows to the original input file indices, so filtering an empty file cannot shift or misidentify later files. - Support the all-files-empty case with correctly typed zero-length arrays. ## Are these changes tested? Yes. - Added unit coverage for common ordering with an empty file, missing min/max values, original file-index mapping, the all-empty case, and NULL checks. - Extended `parquet_sorted_statistics.slt` with ordered non-empty Parquet files plus a zero-row file without ordering metadata. The test enables statistics-based grouping, verifies inferred ordering remains available, and verifies query results. - `cargo test --profile ci -p datafusion-datasource --lib` - `cargo test --profile ci -p datafusion-catalog-listing --lib` - `cargo test --profile=ci --test sqllogictests -- parquet_sorted_statistics` - `cargo clippy --profile ci -p datafusion-datasource -p datafusion-catalog-listing --all-targets --all-features -- -D warnings` - `cargo fmt --all` ## Are there any user-facing changes? Yes. Scans over ordered Parquet datasets can retain their output ordering when exact empty files are present, avoiding an unnecessary sort, and statistics-based file-group splitting no longer fails on those files. There are no public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
