This is an automated email from the ASF dual-hosted git repository.
etseidl pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 87c7c3d653 bench(parquet): cover nullable and repeated large-value
writes (#10561)
87c7c3d653 is described below
commit 87c7c3d6539b347d83878c943b25688b6305200e
Author: Adrian Garcia Badaracco <[email protected]>
AuthorDate: Tue Aug 11 18:24:22 2026 -0400
bench(parquet): cover nullable and repeated large-value writes (#10561)
## Why
Every large-value benchmark in `arrow_writer` builds its array with
`StringArray::from_iter_values`. With no nulls present,
`RecordBatch::try_from_iter` marks the field non-nullable, the column
has `max_def_level == 0`, and its definition levels are absent.
The writer resolves an absent-level chunk's value count in O(1) and
never inspects levels, so **nothing here covers a nullable column whose
values exceed `data_page_size_limit`** — the path that decides how many
values share a data page, and on `DELTA_BYTE_ARRAY` whether prefix
deduplication survives at all. The repeated case is a third level shape,
also uncovered: records cannot span data pages, so mini-batches must
step whole records.
## What
One nullable case, then five that each vary a single property of it, so
a movement can be attributed rather than guessed at:
| group | varies | |
| --- | --- | --- |
| `large_string_shared_prefix_nullable` | — | 2 MiB values, one null in
16 |
| `..._nullable_dense` | null density → one in two | ratio is exactly
two levels per value, so window arithmetic is exact — the case that
should *not* move |
| `..._nullable_trailing` | null placement, count fixed | a run of nulls
at the end leaves every earlier window holding values only |
| `large_string_distinct_nullable` | prefix | removes what deduplication
has to work with |
| `medium_string_shared_prefix_nullable` | value size → 256 KiB |
several values share a page budget instead of one overrunning it |
| `..._shared_prefix_list` | level shape | repeated column, records
cannot span pages |
`PLAIN` is measured where page count alone drives the result and omitted
where it would restate a neighbour: it never reads the previous value,
so prefix, null placement and value size do not change its per-page
work.
## Validation
Run against #10554, which changes exactly this path, as base → branch →
base on an idle machine so the two base passes give a per-benchmark
noise floor. Base is #10505's head, so the effect is #10554 alone.
| benchmark | effect | noise floor |
| --- | --- | --- |
| `..._nullable_trailing/delta_byte_array` | **−27.8%** | 1.2% |
| `..._nullable/plain` | **+27.6%** | 2.1% |
| `..._nullable/delta_byte_array` | +7.6% | 1.0% |
| `..._nullable_dense/delta_byte_array` | **−0.2%** | 0.1% |
| `large_string_distinct_nullable/delta_byte_array` | +3.1% | 3.1% |
| `medium_string_shared_prefix_nullable/delta_byte_array` | +1.3% | 2.9%
|
| `..._shared_prefix_list/delta_byte_array` | +1.3% | 0.9% |
| `list_struct_with_list/*` *(untouched control)* | −0.1% to +0.8% |
0.1–2.4% |
The design holds up: the two cases built to move do, the case built as a
control lands at −0.2% against a 0.1% floor, and the cases whose axis
should not matter stay inside their own noise.
Time alone understates what is happening, so the same batches written to
a file:
| case | #10505 | #10505 + #10554 |
| --- | --- | --- |
| `uniform_1in16` / `DELTA_BYTE_ARRAY` | 16 MiB | **2 MiB** |
| `trailing_8` / `DELTA_BYTE_ARRAY` | 120 MiB | **2 MiB** |
| either / `PLAIN` | 240 MiB | 240 MiB |
Which is why the throughput numbers split the way they do. `_trailing`
gets faster because 118 MiB of writing disappears; the uniform case pays
a small amount because its output was already close to deduplicated and
value-exact windows roughly double the mini-batch count; `PLAIN` pays
the most because it doubles pages for no reduction in output at all.
## Notes
Benchmark-only; no library code touched.
CI clippy is currently red on `main` for an unrelated reason —
`arrow-arith/src/numeric.rs:756` trips `collapsible_if`, from #10409 —
so this PR's Clippy job will fail until that lands a fix. `cargo clippy
-p parquet --all-features --benches -- -D warnings` is clean.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
parquet/benches/arrow_writer.rs | 167 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 162 insertions(+), 5 deletions(-)
diff --git a/parquet/benches/arrow_writer.rs b/parquet/benches/arrow_writer.rs
index 93c9b8ebdc..73dc6fae79 100644
--- a/parquet/benches/arrow_writer.rs
+++ b/parquet/benches/arrow_writer.rs
@@ -31,6 +31,7 @@ use std::sync::Arc;
use arrow::datatypes::*;
use arrow::util::bench_util::{create_f16_array, create_f32_array,
create_f64_array};
use arrow::{record_batch::RecordBatch, util::data_gen::*};
+use arrow_array::builder::{ListBuilder, StringBuilder};
use arrow_array::{RecordBatchOptions, StringArray};
use parquet::errors::Result;
use parquet::file::properties::{CdcOptions, WriterProperties, WriterVersion};
@@ -179,6 +180,83 @@ fn create_large_string_distinct_bench_batch(size: usize,
value_size: usize) -> R
Ok(RecordBatch::try_from_iter([("col", array)])?)
}
+/// Where nulls fall in a generated batch, for
+/// [`create_large_string_nullable_bench_batch`].
+#[derive(Clone, Copy)]
+enum NullPattern {
+ /// One null every `n` rows, spread evenly.
+ Every(usize),
+ /// `n` nulls in a single run at the end of the batch.
+ Trailing(usize),
+}
+
+/// `size` rows of `value_size`-byte strings with nulls, sharing a long common
+/// prefix when `shared_prefix` is set and differing from their first byte
+/// otherwise.
+///
+/// Nullability is the point. The non-null large-value batches leave the
+/// column's definition levels absent, so the writer's byte-budget
+/// sub-batching resolves a chunk's value count in O(1) and never inspects
+/// levels. Nulls put it on the general path, where the number of values that
+/// share a data page is derived from the chunk's level-to-value ratio.
+///
+/// That ratio is why the density levels chosen at the call sites are not
+/// simply "few" and "many". Where a single value already fills the page
+/// budget, the derived window spans `ceil(levels / values)` levels, so it
+/// covers about `levels / values` values instead of one — an overshoot that
+/// is largest when nulls are *sparse* and disappears exactly when the ratio
+/// is a whole number, as at one-null-in-two.
+fn create_large_string_nullable_bench_batch(
+ size: usize,
+ value_size: usize,
+ shared_prefix: bool,
+ nulls: NullPattern,
+) -> Result<RecordBatch> {
+ let filler = "x".repeat(value_size - 8);
+ let is_null = |i: usize| match nulls {
+ NullPattern::Every(n) => i % n == n - 1,
+ NullPattern::Trailing(n) => i >= size - n,
+ };
+ let array = Arc::new(StringArray::from_iter((0..size).map(|i| {
+ (!is_null(i)).then(|| {
+ if shared_prefix {
+ format!("{filler}{i:08}")
+ } else {
+ format!("{i:08}{filler}")
+ }
+ })
+ }))) as _;
+ Ok(RecordBatch::try_from_iter([("col", array)])?)
+}
+
+/// `size` records of `values_per_record` strings of `value_size` bytes,
+/// sharing a long common prefix.
+///
+/// A repeated column is the third level shape the writer sub-batches against,
+/// after absent and flat-nullable levels. Records cannot span data pages, so
+/// mini-batches must step whole records; with values this large a single
+/// record overruns the page limit on its own.
+fn create_list_large_string_bench_batch(
+ size: usize,
+ values_per_record: usize,
+ value_size: usize,
+) -> Result<RecordBatch> {
+ let prefix = "x".repeat(value_size - 8);
+ let mut builder = ListBuilder::new(StringBuilder::new());
+ for i in 0..size {
+ for j in 0..values_per_record {
+ builder
+ .values()
+ .append_value(format!("{prefix}{:08}", i * values_per_record +
j));
+ }
+ builder.append(true);
+ }
+ Ok(RecordBatch::try_from_iter([(
+ "col",
+ Arc::new(builder.finish()) as _,
+ )])?)
+}
+
/// `size` rows of `value_size`-byte strings sharing their first
/// `shared_bytes` bytes and differing thereafter — the realistic sorted-column
/// case (paths, URLs, keys), where prefix deduplication saves part of each
@@ -813,9 +891,88 @@ fn bench_delta_byte_array_writers(c: &mut Criterion) {
.set_encoding(Encoding::DELTA_BYTE_ARRAY)
.build();
- for (batch_name, batch) in [
- ("large_string_shared_prefix", &shared),
- ("large_string_distinct", &distinct),
+ // Nullable and repeated counterparts. Each varies one property of the
+ // first nullable case, so a movement can be attributed to that property
+ // rather than to some combination:
+ //
+ // * `_dense` changes only the null density, to a ratio of exactly two
+ // levels per value. Deriving a window from that ratio is exact, so this
+ // case is the one where sub-batching arithmetic cannot go wrong — it
+ // should stay flat when the others move.
+ // * `_trailing` changes only where the nulls sit, keeping the count. A
+ // run of nulls at the end leaves every window before it holding values
+ // only, which is the worst placement for `DELTA_BYTE_ARRAY`.
+ // * `distinct_nullable` changes only the prefix, removing what
+ // deduplication has to work with.
+ // * `medium_string_*` changes only the value size, to a size where
+ // several values share a page budget rather than one overrunning it.
+ // * `_list` changes only the level shape, to a repeated column. Records
+ // cannot span pages, so this one is a control: the writer's output for
+ // it is byte for byte identical across the changes it is used to judge.
+ //
+ // `PLAIN` is measured wherever page count alone drives the result, and
+ // omitted where it would only restate a neighbouring case: it does not
+ // read the previous value, so prefix, null placement and value size do
+ // not change its per-page work.
+ let nullable = create_large_string_nullable_bench_batch(
+ 128,
+ 2 * 1024 * 1024,
+ true,
+ NullPattern::Every(16),
+ )
+ .unwrap();
+ let nullable_dense =
+ create_large_string_nullable_bench_batch(128, 2 * 1024 * 1024, true,
NullPattern::Every(2))
+ .unwrap();
+ let nullable_trailing = create_large_string_nullable_bench_batch(
+ 128,
+ 2 * 1024 * 1024,
+ true,
+ NullPattern::Trailing(8),
+ )
+ .unwrap();
+ let nullable_distinct = create_large_string_nullable_bench_batch(
+ 128,
+ 2 * 1024 * 1024,
+ false,
+ NullPattern::Every(16),
+ )
+ .unwrap();
+ // 256 KiB against the 1 MiB default limit: several values to a page.
+ let nullable_medium =
+ create_large_string_nullable_bench_batch(1024, 256 * 1024, true,
NullPattern::Every(16))
+ .unwrap();
+ // 4 values per record, so one record is ~8 MiB and cannot be split.
+ let list = create_list_large_string_bench_batch(32, 4, 2 * 1024 *
1024).unwrap();
+
+ let both: &[(&str, &WriterProperties)] = &[("plain", &plain),
("delta_byte_array", &delta)];
+ let delta_only: &[(&str, &WriterProperties)] = &[("delta_byte_array",
&delta)];
+
+ for (batch_name, batch, props) in [
+ ("large_string_shared_prefix", &shared, both),
+ ("large_string_distinct", &distinct, both),
+ ("large_string_shared_prefix_nullable", &nullable, both),
+ (
+ "large_string_shared_prefix_nullable_dense",
+ &nullable_dense,
+ delta_only,
+ ),
+ (
+ "large_string_shared_prefix_nullable_trailing",
+ &nullable_trailing,
+ delta_only,
+ ),
+ (
+ "large_string_distinct_nullable",
+ &nullable_distinct,
+ delta_only,
+ ),
+ (
+ "medium_string_shared_prefix_nullable",
+ &nullable_medium,
+ delta_only,
+ ),
+ ("large_string_shared_prefix_list", &list, delta_only),
] {
let mut group = c.benchmark_group(batch_name);
group.throughput(Throughput::Bytes(
@@ -826,8 +983,8 @@ fn bench_delta_byte_array_writers(c: &mut Criterion) {
.sum(),
));
- for (prop_name, prop) in [("plain", &plain), ("delta_byte_array",
&delta)] {
- group.bench_function(prop_name, |b| {
+ for (prop_name, prop) in props {
+ group.bench_function(*prop_name, |b| {
write_batch_with_option(b, batch,
Some((*prop).clone())).unwrap()
});
}
