adriangb opened a new issue, #10538: URL: https://github.com/apache/arrow-rs/issues/10538
**Is your feature request related to a problem or challenge?** Follow-up to #10505 (which fixes #10489). The mechanism described here is on `main` today and dates to #9972; #10505 raises its cost but does not introduce it. When `write_batch_internal` must sub-batch a chunk because values might overflow the data page byte budget, `byte_budget_sub_batch_size` computes how many values fit and converts that to a level count using the chunk-wide level:value ratio, rounded up: ```rust (values_per_subbatch * chunk_size).div_ceil(vals_in_chunk).max(1) ``` For a chunk with no nulls this is exact. With at least one null in the chunk, one value maps to two levels (17 levels : 16 values gives `ceil(17/16) = 2`), and `write_granular_chunk` slices the whole chunk into uniform two-level windows, most of which carry two values. Two observable effects for values larger than `data_page_size_limit`: - On `main` today: the per-page bound from #9972 is one value per page on chunks without nulls, but two values per page (~2x the value size) on chunks with a null, because the second value lands before the post-mini-batch size check runs. Minor. - With #10505: the first-value exemption that preserves `DELTA_BYTE_ARRAY` prefix dedup triggers when a page opens with a single-value mini-batch. Pages that open with a two-value window miss it and store their first value in full, so dedup on such columns is partial rather than full. Worked example (pinned by `test_column_writer_delta_byte_array_nullable_shared_prefix_partial_dedup` in #10505): 16 identical 64 KiB values with one null at index 8, against a 16 KiB page limit. Result is pages of `[2, 2, 2, 2, 9]` levels with ~5 values stored in full, vs ~1 ideally. Null position sets the severity: a null early in the chunk costs almost nothing, a null at the end means about half the chunk's values are stored in full. **Describe the solution you'd like** Cut mini-batch windows after an exact number of values rather than a ratio-scaled number of levels, in the granular path only. Shape-wise: either the chunker returns a value count and `write_granular_chunk` ends each window by walking def levels until it has covered that many values, or the chunker produces per-window boundaries. Cost: the walk only runs in the granular path (values already overflowing a page budget) and touches each of the chunk's levels once; that path already does full def-level (`value_count`) and rep-level (record alignment) passes. Values in this regime are larger than the page limit by definition, so the walk is noise next to the value writes, and the change strictly reduces bytes written and page count where it fires. Acceptance criterion: the worked example becomes one ~64 KiB page of 17 levels. The pinning test then fails with the improved layout and should be re-pinned; its comment says exactly that. Scope note: for repeated columns, records never span pages, so a single record containing several over-limit values can never yield a one-value opening window. That case is inherent; the improvement targets flat nullable columns. **Describe alternatives you've considered** - Keying #10505's exemption on the page's value count going 0 -> 1 instead of on the opening mini-batch holding exactly one value: only helps pages that open with null-only windows (adjacent nulls); the common two-value window is unaffected. - Exempting the whole opening mini-batch regardless of value count: that exempts a value that could legally move to the next page and erodes the #9972 page bound (toward four values per page in the no-shared-prefix case). **Additional context** #9972 introduced byte-budget sub-batching; #10489 reports the dedup regression; #10505 restores dedup (fully for chunks without nulls) and pins the nullable gap as a known limitation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
