adriangb opened a new pull request, #10512: URL: https://github.com/apache/arrow-rs/pull/10512
# Which issue does this PR close? - Part of #10511 # Rationale for this change No existing benchmark writes `DELTA_BYTE_ARRAY` through the writer: the `arrow_writer` property matrix varies writer version, compression, bloom filters, and CDC, all on the default encoding, and `encoding.rs` benches encoders in isolation. That gap matters for the large-value write path specifically: the byte-budget sub-batching in `write_batch_internal` (#9972) measures raw payload bytes, and page-boundary behavior interacts with the encoding's cross-value state (#10489, #10505). None of that is visible to any current benchmark. #10511 tracks whether making the byte budget encoded-size-aware is worthwhile; these benchmarks are the measurement for that question. # What changes are included in this PR? Two new batches and one new bench group in `parquet/benches/arrow_writer.rs`: - `large_string_shared_prefix`: 128 rows x 2 MiB, long common prefix with a short distinct suffix (the case `DELTA_BYTE_ARRAY` exists for) - `large_string_distinct`: same shape, values differing from byte 0 (the adversarial case, prefix length ~0) Values are sized so one value alone exceeds the default 1 MiB page limit, the regime of #10489. Each batch runs under `plain` and `delta_byte_array` properties (dictionary disabled), so the delta-vs-plain gap on identical data separates inherent encoding cost from writer overhead. Results on an Apple M-series laptop, current `main`: | group | `plain` | `delta_byte_array` | | --- | --- | --- | | `large_string_shared_prefix` | 59.0 ms (4.2 GiB/s) | 61.4 ms (4.1 GiB/s) | | `large_string_distinct` | 37.2 ms (6.7 GiB/s) | 44.7 ms (5.6 GiB/s) | Note these numbers shift when #10505 lands, in both directions: shared-prefix delta gets slower in CPU time because the encoder starts doing real prefix comparisons instead of degenerating to per-page `PLAIN` (while the output shrinks ~128x), and distinct delta gets faster from halving the page count. That sensitivity is the point of having the benchmark. # Are these changes tested? The change is itself a benchmark; it compiles and runs under `cargo bench -p parquet --bench arrow_writer`. # Are there any user-facing changes? No. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
