mattsu2020 opened a new pull request, #26775: URL: https://github.com/apache/datafusion/pull/26775
## Which issue does this PR close? Addresses #25618. ## Rationale for this change When Parquet metadata lacks NDVs, the join estimator substitutes bounds derived from row counts and key ranges. Sparse keys can make these bounds much larger than the actual NDV: the orders–lineitem example underestimates the join output by about four times. Row counts and min/max alone cannot distinguish sparse repeated keys from a dense domain with unmatched values. This change provides an explicit way to collect NDVs instead of assuming that every value in the range occurs. ## What changes are included in this PR? - Add `ListingTable::with_collected_statistics` to scan selected columns and collect exact row counts, null counts, and NDVs using `COUNT(DISTINCT ...)` across all files. NULL is excluded from NDV. - Reuse collected statistics only when the scan's file metadata matches the collected snapshot. Pruned file subsets and changed listings fall back to file statistics; file discovery retains the existing listing-cache policy. - Add repeatable `dfbench statistics --collect-ndv TABLE.COLUMN` options and document their use. - Separate the NDV fallback from its upper-bound calculation and mark positive range/row-count fallbacks as inexact. Explicit NDVs and provable zero bounds retain their precision. The numeric fallback remains unchanged when collection is disabled. ## What is the testing strategy for this PR? Parquet integration tests cover sparse-join estimates and actual results, deduplication across files and row groups, projected/reordered columns, duplicate requests, empty/all-NULL columns, partition pruning, and invalidation when a new file appears. CLI tests cover collection before reporting and invalid requests; join utility tests cover NDV precedence, dense/disjoint ranges, symmetry, and fallback precision. Validation passed: - Parquet collector integration tests: 4 passed. - Statistics command tests: 10 passed. - Join utility tests: 72 passed. - `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, and documentation formatting checks. - Full local CI lint suite: `uv run ./dev/rust_lint.sh` (including Rust/HTML documentation builds, security audit, and dependency checks). On synthetic sparse-key Parquet data matching the issue's input row counts (1,500,000 orders and 6,001,215 lineitems), the join estimate improves from 1,500,303 to 6,001,215 rows with collection enabled. Actual output is 6,001,215 rows, reducing join q-error from 4.00x to 1.00x. This measures estimate accuracy, not an execution-time speedup. ## Are there any user-facing changes? Yes: an additive Rust API and an opt-in statistics command option. For the issue's query, add: ```sh --collect-ndv orders.o_orderkey --collect-ndv lineitem.l_orderkey ``` Collection runs before query reporting, scans the selected columns, and uses memory proportional to NDV. It is never triggered automatically during query planning. No breaking API or SQL-result changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
