adriangb opened a new pull request, #10836: URL: https://github.com/apache/arrow-rs/pull/10836
Adds writer benchmarks for three shapes that drive **mini-batch windowing** and that `arrow_writer` does not reach. When a chunk's values do not all fit in the page byte budget, the column writer splits the chunk into mini-batches. On a nullable column, how wide those windows are depends on the ratio between the value size and `data_page_size_limit` — and the existing suite only samples the two ends of that range. ## What is uncovered today | | `arrow_writer` | gap | | --- | --- | --- | | value size | ~1 KiB (`small_string_*`) or 2 MiB against the 1 MiB default (`large_string_*`) | nothing in between, where *several* values share a page budget | | `data_page_size_limit` | always the 1 MiB default | a smaller limit reaches a one-value window with far less encoding work per value | | how DBA is reached | always pinned via `set_dictionary_enabled(false)` | dictionary spilling into `DELTA_BYTE_ARRAY`, which is how byte-array columns are actually written | ## What is added * **`subpage_*`** — 128 KiB and 512 KiB values against the 1 MiB default. The existing case at this size, `medium_string_shared_prefix_nullable`, is shared-prefix only; the *distinct* case, where deduplication saves nothing and a narrower window is not paid back, is uncovered. A shared-prefix counterpart at 128 KiB varies only the prefix, so a movement present in one and absent in the other is attributable to that. * **`small_page_limit_*`** — 64 KiB values against a 64 KiB `data_page_size_limit`, a realistic setting for selective reads. * **`dictionary_fallback_*`** — a column that starts dictionary-encoded and becomes `DELTA_BYTE_ARRAY` only when the dictionary spills, so the windowing changes part-way through the column. `pinned_delta_byte_array` writes the same data with the dictionary disabled, isolating the dictionary phase and the transition. `plain` variants accompany the `delta_byte_array` ones throughout: `PLAIN` does not compress a value against its predecessor, so it is insensitive to where a page boundary falls and acts as the control for whether a movement is windowing or the machine. ## Notes New `writer_page_windows` bench target rather than adding to `arrow_writer` — that suite is already long enough that a full run has to be split across jobs, and these are heavy (64 MiB written per iteration, sized so times are comparable across value sizes). Benchmarks only; no library code is touched. Motivated by #10538 / #10554, where these shapes came up as unmeasured. Landing them separately means both sides of that comparison have them, so the change can be measured on the shapes it actually affects. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
