This is an automated email from the ASF dual-hosted git repository.
etseidl pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new c6eb173d02 refactor(parquet): simplify guarded row-group byte-limit
arithmetic (#11269)
c6eb173d02 is described below
commit c6eb173d0215b5a33e4024c25ae1b9cddf00c276
Author: Hippolyte Barraud <[email protected]>
AuthorDate: Mon Sep 28 19:37:31 2026 -0400
refactor(parquet): simplify guarded row-group byte-limit arithmetic (#11269)
# Which issue does this PR close?
NA
# Rationale for this change
The row-group byte-limit calculation already runs under a guard that
ensures the buffered row count is nonzero. Using checked division for
that first calculation repeats the guard, while a separate zero-average
filter and fallback value make the remaining-budget calculation more
complicated than necessary.
Simplify the arithmetic while preserving the existing split and flush
decisions.
# What changes are included in this PR?
- Calculate average row bytes directly under the existing nonzero
buffered-row guard.
- Divide the remaining byte budget by that average using `checked_div`.
- Remove the separate zero-average filter and redundant fallback value.
# Are these changes tested?
All tests passing.
# Are there any user-facing changes?
No.
Signed-off-by: Hippolyte Barraud <[email protected]>
---
parquet/src/arrow/arrow_writer/mod.rs | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
diff --git a/parquet/src/arrow/arrow_writer/mod.rs
b/parquet/src/arrow/arrow_writer/mod.rs
index 6b5f29d0f1..a72584bbd8 100644
--- a/parquet/src/arrow/arrow_writer/mod.rs
+++ b/parquet/src/arrow/arrow_writer/mod.rs
@@ -401,14 +401,10 @@ impl<W: Write + Send> ArrowWriter<W> {
continue;
}
- if let Some(avg_row_bytes) = current_bytes
- .checked_div(buffered_rows)
- .filter(|avg_row_bytes| *avg_row_bytes > 0)
+ let avg_row_bytes = current_bytes / buffered_rows;
+ if let Some(rows_that_fit) = (max_bytes -
current_bytes).checked_div(avg_row_bytes)
{
// At this point, `current_bytes < max_bytes` (checked
above)
- let remaining_bytes = max_bytes - current_bytes;
- let rows_that_fit =
remaining_bytes.checked_div(avg_row_bytes).unwrap_or(0);
-
if candidate_rows > rows_that_fit {
if rows_that_fit > 0 {
split_at = Some(rows_that_fit);
