ZENOTME opened a new issue, #1199: URL: https://github.com/apache/paimon-rust/issues/1199
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. ### Motivation The PK sort-merge reader ignores `read.batch-size` for its output. Since #547 the setting only reaches the input decoders. `SortMergeReaderBuilder` always emits 1024-row batches: `with_batch_size` is `#[cfg(test)]`-only ([sort_merge.rs#L1245](https://github.com/apache/paimon-rust/blob/7baef42fbeb97353036036b1bb71a1f57510e00b/crates/paimon/src/table/sort_merge.rs#L1245)), and `KeyValueFileReader` never sets it ([kv_file_reader.rs#L820](https://github.com/apache/paimon-rust/blob/7baef42fbeb97353036036b1bb71a1f57510e00b/crates/paimon/src/table/kv_file_reader.rs#L820)). With `read.batch-size=8192`, every merged split (e.g. all partial-update tables without DV) yields batches 8x smaller, and downstream per-batch overhead multiplies. Measured on v0.4.0: 5 partial-update tables (25.4M rows each) read in parallel from Python. | | batches | wall time | |---|---|---| | v0.4.0 | 102739 | 141–149 s | | merge output honors `read.batch-size` | 16050 | 67.8 s | Output data is identical; only batch boundaries change. ### Solution Pass the configured size to the merge stream and make the setter available outside tests: ```diff --- a/crates/paimon/src/table/sort_merge.rs - #[cfg(test)] pub(crate) fn with_batch_size(mut self, batch_size: usize) -> Self { --- a/crates/paimon/src/table/kv_file_reader.rs .with_user_sequence_descending(...) + .with_batch_size(config.read_batch_size) .build()?; ``` ### Anything else? _No response_ ### Willingness to contribute - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
