alexanderbianchi opened a new pull request, #125: URL: https://github.com/apache/datafusion-iceberg/pull/125
Eager scans retained whole-snapshot row estimates even after file pruning, while individual output partitions reported unknown counts. A taxi scan selecting two of seven daily files therefore reported 175,000 rows instead of 50,000. Derive row estimates from the retained `FileScanTask`s for the whole assignment or the requested partition. The two-file scan now reports 50,000 rows total and 25,000 per partition. This also works for explicitly supplied file groups. Adapt the statistics portion of [DFD #787](https://github.com/datafusion-contrib/datafusion-distributed/pull/787): - Keep every nonempty assignment's count inexact, including unfiltered scans and zero-valued file counts. - Report exact zero rows and bytes for an empty assignment. - Leave row counts unknown when a task lacks a count or the total overflows. - Let the shared source apply fetch and estimate projected output bytes. Compressed file sizes and distributed task-count calculations remain scheduling concerns in DFD. Extend the existing per-partition fetch test and update the fully pruned scan's plan: DataFusion can now replace its `COUNT(*)` with zero. Add one harness-based pruning regression and focused metadata-boundary unit tests. Validation: - Confirmed the pruning regression fails against the original implementation at `a8e46f5` (175,000 rows and absent per-partition estimates). - `cargo test --workspace --locked` - `cargo clippy --workspace --locked --all-targets -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check` Developed with AI assistance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
