alexanderbianchi opened a new pull request, #126: URL: https://github.com/apache/datafusion-iceberg/pull/126
Add an interim session-to-reader mapping using the current iceberg-rust pin, based on the reader controls in [DFD #595](https://github.com/datafusion-contrib/datafusion-distributed/pull/595) / [#720](https://github.com/datafusion-contrib/datafusion-distributed/pull/720). - Add `iceberg.execution.data_file_concurrency_limit` per output partition. Preserve the current default of one and normalize zero to one, matching iceberg-rust #3514. - Honor `datafusion.execution.parquet.pruning` for reader row-group filtering. - Centralize these settings with the existing batch-size and page-index mapping in a private reader-construction function. Explicit `with_reader(...)` overrides continue to take precedence. Once the dependency includes [iceberg-rust #3514](https://github.com/apache/iceberg-rust/pull/3514), replace the private setter mapping with `ArrowReaderOptions` and `with_options(...)`. The session configuration can remain the same across that update. Tests use the existing SQL harness: disabling row-group pruning increases data reads without changing results, and zero concurrency still completes a scan. Confirmed the tests fail when the pruning mapping and zero normalization are removed. Existing batch-size and page-index tests remain in place. Validation: - `cargo test --workspace --locked` - `cargo clippy --workspace --locked --all-targets -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check` Developed with AI assistance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
