adriangb opened a new pull request, #10878:
URL: https://github.com/apache/arrow-rs/pull/10878
# Which issue does this PR close?
- Contributes to #9722.
# Rationale for this change
`parquet/benches/writer_overhead.rs` writes a single one-row batch per file.
That
measures column writer construction, allocation and metadata assembly, and it
measures them well, but it never reaches the work a wide schema repeats on
every
batch: the per-column setup inside `ColumnWriter::write_batch`, and the
per-mini-batch and per-page checks below it.
With one `write` call per file, a change to that per-batch path is invisible
here.
# What changes are included in this PR?
Adds `writer_overhead/{1000,5000}_cols/repeated_batches`. It holds the
number of
column writers fixed and writes 32 batches of 32 rows into a single row
group,
reusing the same wide schema and the same per-column `WriterProperties` as
the
existing cases, so the only difference from `writer_overhead/{n}_cols` is the
number of `write` calls.
The 10,000 column case is deliberately left out of the new benchmark so that
one
iteration stays well under a second. On the machine used here an iteration is
roughly 13 ms at 1,000 columns and 90 ms at 5,000 columns.
# Are these changes tested?
No, this is a benchmark-only change. Run it with:
cargo bench -p parquet --bench writer_overhead
# Are there any user-facing changes?
No.
# AI usage
This PR was written with Claude Code and reviewed by a human.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
