alexanderbianchi opened a new pull request, #126:
URL: https://github.com/apache/datafusion-iceberg/pull/126

   Add an interim session-to-reader mapping using the current iceberg-rust pin, 
based on the reader controls in [DFD 
#595](https://github.com/datafusion-contrib/datafusion-distributed/pull/595) / 
[#720](https://github.com/datafusion-contrib/datafusion-distributed/pull/720).
   
   - Add `iceberg.execution.data_file_concurrency_limit` per output partition. 
Preserve the current default of one and normalize zero to one, matching 
iceberg-rust #3514.
   - Honor `datafusion.execution.parquet.pruning` for reader row-group 
filtering.
   - Centralize these settings with the existing batch-size and page-index 
mapping in a private reader-construction function. Explicit `with_reader(...)` 
overrides continue to take precedence.
   
   Once the dependency includes [iceberg-rust 
#3514](https://github.com/apache/iceberg-rust/pull/3514), replace the private 
setter mapping with `ArrowReaderOptions` and `with_options(...)`. The session 
configuration can remain the same across that update.
   
   Tests use the existing SQL harness: disabling row-group pruning increases 
data reads without changing results, and zero concurrency still completes a 
scan. Confirmed the tests fail when the pruning mapping and zero normalization 
are removed. Existing batch-size and page-index tests remain in place.
   
   Validation:
   - `cargo test --workspace --locked`
   - `cargo clippy --workspace --locked --all-targets -- -D warnings`
   - `cargo fmt --all -- --check`
   - `git diff --check`
   
   Developed with AI assistance.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to