dtenedor opened a new pull request, #12273:
URL: https://github.com/apache/arrow-rs/pull/12273
# Which issue does this PR close?
None.
# Rationale for this change
The column-level delta byte array encoders, which the non-Arrow column
writer API uses, do extra
work for every value:
- `DeltaLengthByteArrayEncoder` keeps a `ByteArray` clone of every value, an
atomic increment and
decrement each, and copies the bytes at flush.
- Both encoders allocate temporary `Vec`s on every `put`.
- `DeltaByteArrayEncoder` copies every value into its previous-value buffer
and slices a new
`ByteArray` for every suffix.
The Arrow writer has its own byte array encoders and is not affected.
# What changes are included in this PR?
- Values are copied once, as they are put, into a single byte buffer that is
reserved from the
input's total length.
- Lengths are encoded in stack batches of 128.
- `DeltaByteArrayEncoder` compares each value with its predecessor in place
and keeps only the last
value of each `put`.
Output is byte-identical.
The first commit adds the benchmark cases, so it can serve as the benchmark
baseline. #12272 starts
with the same commit.
# Are these changes tested?
Covered by the existing encoder tests. Output was also checked to be
byte-identical to main's for
DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY over several datasets, value
lengths and `put`
chunkings.
The first commit adds delta byte array cases to the `encoding` bench:
- 16K sorted keys with shared prefixes
- 16K random strings of 8-31 bytes
- 1K values of 1 KiB
The encoder is reused across iterations. Measured on an Intel Xeon Platinum
8559C, default x86-64
target, median of 5 alternating rounds:
| `encoding` bench | Speedup |
| :--- | ---: |
| DELTA_BYTE_ARRAY encode: random / sorted_keys / large_1k | 3.26x / 2.27x /
1.70x |
| DELTA_LENGTH_BYTE_ARRAY encode: random / sorted_keys | 2.90x / 1.90x |
| **DELTA_LENGTH_BYTE_ARRAY encode: large_1k** | **0.78x** |
| decode, all cases | 0.99-1.01x |
DELTA_LENGTH_BYTE_ARRAY gets slower on large values because their bytes are
now copied twice: once
at `put` and again at flush. Before, `put` only took a clone. The Arrow
writer's fallback encoder
already works this way.
# Are there any significant user-facing changes?
`estimated_memory_size` of `DeltaLengthByteArrayEncoder` now counts the byte
buffer's capacity.
Before, it counted the number of buffered values.
# AI usage disclosure
An AI coding agent (Claude Code) generated the code, the benchmark cases and
this description, and
ran the benchmarks and the output checks. This PR is a draft until the
submitter has reviewed it.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]