etseidl commented on code in PR #12322:
URL: https://github.com/apache/arrow-rs/pull/12322#discussion_r4620188570
##########
parquet/src/file/properties.rs:
##########
@@ -229,6 +229,30 @@ enum OffsetIndexSetting {
/// Some(Encoding::PLAIN)
/// );
/// ```
+///
+/// # Page limits and batch size
+///
+///
[`set_data_page_size_limit`](WriterPropertiesBuilder::set_data_page_size_limit),
+///
[`set_data_page_row_count_limit`](WriterPropertiesBuilder::set_data_page_row_count_limit)
+/// and
[`set_dictionary_page_size_limit`](WriterPropertiesBuilder::set_dictionary_page_size_limit)
+/// are best effort, so a page can end up larger than the limit set for it.
Limits
+/// are only checked between batches, and
[`set_write_batch_size`](WriterPropertiesBuilder::set_write_batch_size)
+/// sets how large those batches are. A page can therefore pass its limit by
up to
+/// one batch worth of data, and lowering the write batch size keeps pages
closer
+/// to their limits.
+///
+/// There is one exception. When the definition and repetition levels of a
batch
Review Comment:
Should point out that this depends on the write path. The Arrow writer can
represent levels compactly, but other paths, including caller-supplied level
slices passed to the column writer, continue to use the write batch size.
##########
parquet/src/file/properties.rs:
##########
@@ -229,6 +229,30 @@ enum OffsetIndexSetting {
/// Some(Encoding::PLAIN)
/// );
/// ```
+///
+/// # Page limits and batch size
+///
+///
[`set_data_page_size_limit`](WriterPropertiesBuilder::set_data_page_size_limit),
+///
[`set_data_page_row_count_limit`](WriterPropertiesBuilder::set_data_page_row_count_limit)
+/// and
[`set_dictionary_page_size_limit`](WriterPropertiesBuilder::set_dictionary_page_size_limit)
+/// are best effort, so a page can end up larger than the limit set for it.
Limits
+/// are only checked between batches, and
[`set_write_batch_size`](WriterPropertiesBuilder::set_write_batch_size)
+/// sets how large those batches are. A page can therefore pass its limit by
up to
+/// one batch worth of data, and lowering the write batch size keeps pages
closer
+/// to their limits.
Review Comment:
should mention here that chunks extend to record boundaries, so a large
list, for instance, could overshoot by quite a bit.
##########
parquet/src/file/properties.rs:
##########
@@ -229,6 +229,30 @@ enum OffsetIndexSetting {
/// Some(Encoding::PLAIN)
/// );
/// ```
+///
+/// # Page limits and batch size
+///
+///
[`set_data_page_size_limit`](WriterPropertiesBuilder::set_data_page_size_limit),
+///
[`set_data_page_row_count_limit`](WriterPropertiesBuilder::set_data_page_row_count_limit)
+/// and
[`set_dictionary_page_size_limit`](WriterPropertiesBuilder::set_dictionary_page_size_limit)
+/// are best effort, so a page can end up larger than the limit set for it.
Limits
+/// are only checked between batches, and
[`set_write_batch_size`](WriterPropertiesBuilder::set_write_batch_size)
+/// sets how large those batches are. A page can therefore pass its limit by
up to
+/// one batch worth of data, and lowering the write batch size keeps pages
closer
+/// to their limits.
+///
+/// There is one exception. When the definition and repetition levels of a
batch
+/// are all the same value, for example a nullable column whose batch happens
to
+/// contain no nulls, there is no level array to split and a larger batch costs
+/// no extra work. Those batches are written
+///
[`set_data_page_row_count_limit`](WriterPropertiesBuilder::set_data_page_row_count_limit)
+/// rows at a time instead, so for them that limit is the enforcement
granularity
+/// and the write batch size does not move page boundaries.
Review Comment:
Should mention that byte-budget sub-batching can split chunks further. E.g.
a INT64 column with a 1M row limit and a 10,000 byte page limit can produce
1,250 rows per page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]