adriangb opened a new pull request, #10561: URL: https://github.com/apache/arrow-rs/pull/10561
## Why Every large-value benchmark in `arrow_writer` builds its array with `StringArray::from_iter_values`. With no nulls present, `RecordBatch::try_from_iter` marks the field non-nullable, the column has `max_def_level == 0`, and its definition levels are absent. That matters more than it looks. The writer's byte-budget sub-batching resolves an absent-level chunk's value count in O(1) and never inspects levels, so **no benchmark currently exercises a nullable column whose values exceed `data_page_size_limit`** — the path that decides how many values share a data page, and on `DELTA_BYTE_ARRAY` whether prefix deduplication survives at all. Sparse nulls put the level:value ratio just above 1:1, which is exactly where that path's decisions bind. The repeated case is a third level shape, also uncovered: records cannot span data pages, so mini-batches must step whole records. ## What Three batch builders and three benchmark groups, each run under `PLAIN` and `DELTA_BYTE_ARRAY` like the existing large-value groups: - `large_string_shared_prefix_nullable` — 2 MiB values sharing a long prefix, one null every 8 rows - `large_string_distinct_nullable` — same shape, values differing from byte 0, where prefix dedup cannot help and only the page bound is in play - `large_string_shared_prefix_list` — 4 × 2 MiB values per record, so a record is ~8 MiB against the 1 MiB default ## They resolve what they target Measured against #10554, which changes exactly this path, run base → branch → base so the two base passes give a per-benchmark noise floor: | benchmark | delta | noise floor | | --- | --- | --- | | `large_string_shared_prefix_nullable/delta_byte_array` | **−31%** | 1.6% | | `large_string_shared_prefix_nullable/plain` | **+25%** | 8.1% | | `large_string_distinct_nullable/plain` | **+24%** | 4.5% | | `large_string_distinct_nullable/delta_byte_array` | +4% | 2.0% | | `large_string_shared_prefix_list/plain` | −1% | 2.2% | | `large_string_shared_prefix_list/delta_byte_array` | ±0% | 0.6% | Both directions are informative. The `delta_byte_array` improvement is the deduplication that was previously invisible; the `plain` regression is the cost of a tighter page bound (one over-limit value per page instead of two, so twice the pages) — a trade-off that was being made on reasoning alone because nothing measured it. The list variants staying flat is the expected result rather than a null one: a record holding several over-limit values cannot be split across pages whatever the sub-batching does. ## Notes Benchmark-only; no library code touched. CI clippy is currently red on `main` for an unrelated reason — `arrow-arith/src/numeric.rs:756` trips `collapsible_if`, introduced by #10409 — so this PR's Clippy job will fail until that is fixed. `cargo clippy -p parquet --all-features --benches -- -D warnings` is clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
