zhf999 commented on code in PR #50743:
URL: https://github.com/apache/arrow/pull/50743#discussion_r3687569621
##########
cpp/src/parquet/properties.h:
##########
@@ -492,6 +496,18 @@ class PARQUET_EXPORT WriterProperties {
return this;
}
+ /// Specify the max row group size in compressed bytes.
+ /// Default unlimited.
+ ///
+ /// The limit is checked against the compressed pages accumulated in the
+ /// current row group, so the actual row group size may slightly exceed it.
+ /// Only effective for buffered row groups (
+ /// parquet::arrow::FileWriter::WriteRecordBatch).
Review Comment:
`WriteRecordBatch` writes into a buffered row group, which accumulates
across calls. That makes the byte size observable between writes, so we can
simply stop adding to the current row group once the accumulated compressed
size reaches the limit.
While `WriteTable` uses the non-buffered path, where each call maps to
exactly one row group whose row count is fixed up front by the user-supplied
chunk_size. Enforcing a byte limit there would require predicting the
compressed size before writing, e.g. by estimating an average row size from the
previous row group. That's inherently approximate, especially since chunk_size
is an explicit contract from the caller.
I wonder if estimation is acceptable in `WriteTable`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
