nikkhiil-hadoop opened a new pull request, #102: URL: https://github.com/apache/datafusion-iceberg/pull/102
Closes #100. `IcebergDataSource::open` didn't pass `execution.batch_size` to `get_batch_stream`, which never called `TableScanBuilder::with_batch_size`. Scans therefore always read arrow-rs's default of 1,024-row batches, and `SET datafusion.execution.batch_size` had no effect on them. This passes the session's value through, as suggested in the issue. ### Test `crates/datafusion/tests/batch_size.rs` sets `batch_size = 2000` and checks that the largest batch from `SELECT pickup_date FROM taxi` has 2,000 rows. It fails on `main` with 1,024. The size is deliberately above 1,024: DataFusion splits batches that are larger than `batch_size` after the scan, so a smaller value passes without the fix. The test also snapshots the plan (`CooperativeExec` over `DataSourceExec`), so an operator that merges batches above the scan would show up as a snapshot change instead of silently weakening the assertion. To get at the batches, `IcebergTestHarness` gains `query_batches`, which returns them unformatted; `query` now delegates to it. ### Checks - `cargo fmt --all -- --check` - `cargo clippy --workspace --locked --all-targets -- -D warnings` - `cargo test --workspace --locked` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
