alexanderbianchi opened a new pull request, #125:
URL: https://github.com/apache/datafusion-iceberg/pull/125

   Eager scans retained whole-snapshot row estimates even after file pruning, 
while individual output partitions reported unknown counts. A taxi scan 
selecting two of seven daily files therefore reported 175,000 rows instead of 
50,000.
   
   Derive row estimates from the retained `FileScanTask`s for the whole 
assignment or the requested partition. The two-file scan now reports 50,000 
rows total and 25,000 per partition. This also works for explicitly supplied 
file groups.
   
   Adapt the statistics portion of [DFD 
#787](https://github.com/datafusion-contrib/datafusion-distributed/pull/787):
   - Keep every nonempty assignment's count inexact, including unfiltered scans 
and zero-valued file counts.
   - Report exact zero rows and bytes for an empty assignment.
   - Leave row counts unknown when a task lacks a count or the total overflows.
   - Let the shared source apply fetch and estimate projected output bytes. 
Compressed file sizes and distributed task-count calculations remain scheduling 
concerns in DFD.
   
   Extend the existing per-partition fetch test and update the fully pruned 
scan's plan: DataFusion can now replace its `COUNT(*)` with zero. Add one 
harness-based pruning regression and focused metadata-boundary unit tests.
   
   Validation:
   - Confirmed the pruning regression fails against the original implementation 
at `a8e46f5` (175,000 rows and absent per-partition estimates).
   - `cargo test --workspace --locked`
   - `cargo clippy --workspace --locked --all-targets -- -D warnings`
   - `cargo fmt --all -- --check`
   - `git diff --check`
   
   Developed with AI assistance.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to