This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new f8a57f83f0 bench(parquet): cover DELTA_BYTE_ARRAY large values in the
arrow_writer benchmark (#10512)
f8a57f83f0 is described below
commit f8a57f83f00cc6e694826bd13351092d8c8a1fc0
Author: Adrian Garcia Badaracco <[email protected]>
AuthorDate: Mon Aug 3 10:48:40 2026 -0500
bench(parquet): cover DELTA_BYTE_ARRAY large values in the arrow_writer
benchmark (#10512)
# Which issue does this PR close?
- Part of #10511
# Rationale for this change
No existing benchmark writes `DELTA_BYTE_ARRAY` through the writer: the
`arrow_writer` property matrix varies writer version, compression, bloom
filters, and CDC, all on the default encoding, and `encoding.rs` benches
encoders in isolation.
That gap matters for the large-value write path specifically: the
byte-budget sub-batching in `write_batch_internal` (#9972) measures raw
payload bytes, and page-boundary behavior interacts with the encoding's
cross-value state (#10489, #10505). None of that is visible to any
current benchmark. #10511 tracks whether making the byte budget
encoded-size-aware is worthwhile; these benchmarks are the measurement
for that question.
# What changes are included in this PR?
Two new batches and one new bench group in
`parquet/benches/arrow_writer.rs`:
- `large_string_shared_prefix`: 128 rows x 2 MiB, long common prefix
with a short distinct suffix (the case `DELTA_BYTE_ARRAY` exists for)
- `large_string_distinct`: same shape, values differing from byte 0 (the
adversarial case, prefix length ~0)
Values are sized so one value alone exceeds the default 1 MiB page
limit, the regime of #10489. Each batch runs under `plain` and
`delta_byte_array` properties (dictionary disabled), so the
delta-vs-plain gap on identical data separates inherent encoding cost
from writer overhead.
Results on an Apple M-series laptop, current `main`:
| group | `plain` | `delta_byte_array` |
| --- | --- | --- |
| `large_string_shared_prefix` | 59.0 ms (4.2 GiB/s) | 61.4 ms (4.1
GiB/s) |
| `large_string_distinct` | 37.2 ms (6.7 GiB/s) | 44.7 ms (5.6 GiB/s) |
Note these numbers shift when #10505 lands, in both directions:
shared-prefix delta gets slower in CPU time because the encoder starts
doing real prefix comparisons instead of degenerating to per-page
`PLAIN` (while the output shrinks ~128x), and distinct delta gets faster
from halving the page count. That sensitivity is the point of having the
benchmark.
# Are these changes tested?
The change is itself a benchmark; it compiles and runs under `cargo
bench -p parquet --bench arrow_writer`.
# Are there any user-facing changes?
No.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <[email protected]>
---
parquet/benches/arrow_writer.rs | 76 +++++++++++++++++++++++++++++++++++++++--
1 file changed, 74 insertions(+), 2 deletions(-)
diff --git a/parquet/benches/arrow_writer.rs b/parquet/benches/arrow_writer.rs
index adc4dcc9ea..c8dd7b8479 100644
--- a/parquet/benches/arrow_writer.rs
+++ b/parquet/benches/arrow_writer.rs
@@ -20,7 +20,7 @@ extern crate criterion;
use criterion::{Bencher, Criterion, Throughput};
use parquet::arrow::ArrowWriter;
-use parquet::basic::{Compression, ZstdLevel};
+use parquet::basic::{Compression, Encoding, ZstdLevel};
extern crate arrow;
extern crate parquet;
@@ -123,6 +123,31 @@ fn create_large_string_bench_batch(size: usize,
value_size: usize) -> Result<Rec
Ok(RecordBatch::try_from_iter([("col", array)])?)
}
+/// `size` rows of `value_size`-byte strings sharing a long common prefix and
+/// ending in a short distinct suffix — the case `DELTA_BYTE_ARRAY` exists
+/// for: consecutive values dedup to a prefix length plus a few suffix bytes.
+fn create_large_string_shared_prefix_bench_batch(
+ size: usize,
+ value_size: usize,
+) -> Result<RecordBatch> {
+ let prefix = "x".repeat(value_size - 8);
+ let array = Arc::new(StringArray::from_iter_values(
+ (0..size).map(|i| format!("{prefix}{i:08}")),
+ )) as _;
+ Ok(RecordBatch::try_from_iter([("col", array)])?)
+}
+
+/// `size` rows of `value_size`-byte strings whose leading bytes differ — the
+/// adversarial case for `DELTA_BYTE_ARRAY`, where every prefix length is ~0
+/// and the encoding stores each value in full.
+fn create_large_string_distinct_bench_batch(size: usize, value_size: usize) ->
Result<RecordBatch> {
+ let filler = "x".repeat(value_size - 8);
+ let array = Arc::new(StringArray::from_iter_values(
+ (0..size).map(|i| format!("{i:08}{filler}")),
+ )) as _;
+ Ok(RecordBatch::try_from_iter([("col", array)])?)
+}
+
fn create_string_and_binary_view_bench_batch(
size: usize,
null_density: f32,
@@ -651,5 +676,52 @@ fn bench_all_writers(c: &mut Criterion) {
}
}
-criterion_group!(benches, bench_all_writers);
+/// Writes BYTE_ARRAY columns of large (multi-MiB) string values with
+/// `DELTA_BYTE_ARRAY`, with `PLAIN` on the same data as a baseline.
+///
+/// Two data shapes bracket the encoding's best and worst case:
+/// * `large_string_shared_prefix`: values like `xxx…x00000000`,
+/// `xxx…x00000001`, … share a long common prefix and differ only in a
+/// short suffix — the case `DELTA_BYTE_ARRAY` is designed to handle well,
+/// encoding each value as a prefix length plus a few suffix bytes.
+/// * `large_string_distinct`: values like `00000000x…xxx`, `00000001x…xxx`, …
+/// differ in their leading bytes, so prefix deduplication saves nothing and
+/// the encoding stores each value in full.
+fn bench_delta_byte_array_writers(c: &mut Criterion) {
+ // Each 2 MiB value alone exceeds the default 1 MiB data page size limit.
+ let shared = create_large_string_shared_prefix_bench_batch(128, 2 * 1024 *
1024).unwrap();
+ let distinct = create_large_string_distinct_bench_batch(128, 2 * 1024 *
1024).unwrap();
+
+ let plain = WriterProperties::builder()
+ .set_dictionary_enabled(false)
+ .set_encoding(Encoding::PLAIN)
+ .build();
+ let delta = WriterProperties::builder()
+ .set_dictionary_enabled(false)
+ .set_encoding(Encoding::DELTA_BYTE_ARRAY)
+ .build();
+
+ for (batch_name, batch) in [
+ ("large_string_shared_prefix", &shared),
+ ("large_string_distinct", &distinct),
+ ] {
+ let mut group = c.benchmark_group(batch_name);
+ group.throughput(Throughput::Bytes(
+ batch
+ .columns()
+ .iter()
+ .map(|f| f.get_array_memory_size() as u64)
+ .sum(),
+ ));
+
+ for (prop_name, prop) in [("plain", &plain), ("delta_byte_array",
&delta)] {
+ group.bench_function(prop_name, |b| {
+ write_batch_with_option(b, batch,
Some((*prop).clone())).unwrap()
+ });
+ }
+ group.finish();
+ }
+}
+
+criterion_group!(benches, bench_all_writers, bench_delta_byte_array_writers);
criterion_main!(benches);
