Tpt opened a new pull request, #11798: URL: https://github.com/apache/arrow-rs/pull/11798
# Which issue does this PR close? - Closes #11792 # Rationale for this change `max_row_group_bytes` has no effect when a single batch is larger than the limit: the first batch written to a row group is accepted whole, as there are no buffered rows to estimate a row size from. Row groups then hold one batch each, however large it is. See the issue for a reproduction. # What changes are included in this PR? When a byte limit is set and the in-progress row group is empty, `ArrowWriter::write` now writes a single row first. That gives the existing logic a measured row size to work out how many of the remaining rows fit, so they get split the same way as for a non-empty row group. The row limit still applies first, and nothing changes when `max_row_group_bytes` is unset. # Are these changes tested? Yes, `test_row_group_limit_bytes_splits_first_batch` writes 100-row batches against a byte limit of about 35 rows. It checks that no row group holds a whole batch and that every row group stays within one row of the limit. It fails on `main` with row groups of `[100, 100, 100]`. The existing `test_row_group_limit_*` tests pass unchanged. # Are there any significant user-facing changes? Writers that set `max_row_group_bytes` with batches larger than it now get row groups that respect the limit, so they produce more and smaller row groups than before. The first row of each row group is encoded on its own. That adds a small per-row-group cost, but only when a byte limit is set. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
