zhf999 opened a new pull request, #50743:
URL: https://github.com/apache/arrow/pull/50743
# Add max_row_group_size writer property to limit row groups by compressed
byte size
GH-#48467 Add configure to limit the row group size
# PR Description
### Rationale for this change
Currently the Parquet writer only supports limiting row groups by row count
(`WriterProperties::max_row_group_length()`). For tables with wide rows or
highly variable row sizes, a row-count limit alone can produce row groups
that
are much larger (or smaller) than desired. Many query engines and storage
systems tune around a target row group size in bytes (e.g. HDFS block size),
so it is useful to also support limiting row groups by their compressed byte
size.
### What changes are included in this PR?
- Add `WriterProperties::max_row_group_size()` and
`WriterProperties::Builder::max_row_group_size(int64_t)` to specify the max
row group size in compressed bytes. The default is unlimited
(`std::numeric_limits<int64_t>::max()`), so existing behavior is unchanged
unless users opt in.
- `parquet::arrow::FileWriter::WriteRecordBatch` now checks the size of the
current buffered row group before/after each write, counting:
- compressed pages already written to the sink
(`RowGroupWriter::total_compressed_bytes_written()`),
- compressed pages still buffered in column writers
(`RowGroupWriter::total_compressed_bytes()`),
- the buffered dictionary not yet written as dictionary pages
(`RowGroupWriter::estimated_buffered_stats().dict_bytes`, uncompressed).
When the accumulated size reaches the limit, a new buffered row group is
started. The check happens at write boundaries, so a row group may exceed
the limit by up to the size of a single written batch.
- The limit only applies to buffered row groups (`WriteRecordBatch`).
`WriteTable` and the manual `NewRowGroup()`/`WriteColumnChunk()` paths
control row group boundaries explicitly via `chunk_size`/API calls and are
intentionally not affected (row counts are fixed before writing in the
non-buffered mode, so the byte size limit cannot be enforced there). This
is
documented on the builder method.
### Are these changes tested?
Yes.
- `WriterPropertiesTest.RoundTripThroughBuilder` (properties_test.cc) now
covers `max_row_group_size`, with a non-default value added to the
`override_defaults` test case.
- New test `TestArrowReadWrite.WriteRecordBatchRespectsMaxRowGroupSize`
(arrow_reader_writer_test.cc) verifies that record batches exceeding the
byte size limit cause row group rollover on subsequent writes.
- Existing `WriteRecordBatch`/`WriteTable` tests pass unchanged, confirming
no
behavior change with the default (unlimited) setting.
### Are there any user-facing changes?
Yes, a new opt-in writer property `max_row_group_size` is added to the public
C++ API (`WriterProperties` / `WriterProperties::Builder`). The default is
unlimited, so there is no behavior change for existing users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
