alexanderbianchi opened a new pull request, #124: URL: https://github.com/apache/datafusion-iceberg/pull/124
Iceberg scans exposed baseline output metrics, but `EXPLAIN ANALYZE` could not show how many files the reader started or how many bytes it read. Port the instrumentation from [DFD #850](https://github.com/datafusion-contrib/datafusion-distributed/pull/850) into the shared `IcebergDataSource`, for eager and lazy task providers. Add two per-partition metrics: - `files_opened`: file tasks pulled by the reader, including read-ahead for limited scans. - `bytes_scanned`: reader-reported bytes for data and delete files, excluding manifest discovery. Update byte metrics when a stream poll completes, including when the reader returns no batches. Harness-driven `EXPLAIN ANALYZE` snapshots cover a lazy scan with zero matching rows and an eager scan across two partitions. The existing limit snapshot now checks that only one file was opened. Validation: - Both new tests fail without the instrumentation. - `cargo test --workspace --locked` - `cargo clippy --workspace --locked --all-targets -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check` Developed with AI assistance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
