iemejia opened a new pull request, #3504:
URL: https://github.com/apache/parquet-java/pull/3504
### Rationale for this change
`ByteStreamSplitValuesWriter` is the primary writer for
`BYTE_STREAM_SPLIT`-encoded `FLOAT`, `DOUBLE`, `INT32`, and `INT64` columns.
Each value goes through a hot path that performs both an unnecessary allocation
and N single-byte virtual dispatches.
For `FloatByteStreamSplitValuesWriter.writeFloat(float v)`:
```java
super.scatterBytes(BytesUtils.intToBytes(Float.floatToIntBits(v)));
```
`BytesUtils.intToBytes` allocates a fresh `byte[4]` on every call.
`scatterBytes` then loops:
```java
for (int i = 0; i < bytes.length; ++i) {
this.byteStreams[i].write(bytes[i]); //
CapacityByteArrayOutputStream.write(int)
}
```
So **per value**: 1 `byte[4]` allocation + 4 single-byte virtual dispatches.
For a 100k-value `FLOAT` page that is 100k allocations and 400k single-byte
writes. `DOUBLE`/`LONG` are even worse (`byte[8]`, 800k single-byte writes).
### What changes are included in this PR?
Two stacked changes in `ByteStreamSplitValuesWriter`:
1. **Eliminate per-value allocation**: replace
`super.scatterBytes(BytesUtils.intToBytes(v))` with `bufferInt(v)` /
`bufferLong(v)` that perform the little-endian decomposition with bit shifts
directly, no temporary `byte[]`.
2. **Batch single-byte writes**: accumulate `BATCH_SIZE = 128` values in a
small per-instance scratch buffer and flush them as `N` bulk `write(byte[],
off, len)` calls (one per stream), replacing `BATCH_SIZE * elementSizeInBytes`
single-byte virtual dispatches with `elementSizeInBytes` bulk writes per flush.
The constant was chosen by sweeping 16/32/64/128/256/512/1024 — 128 is the
sweet spot for `FLOAT` throughput while still capturing most of the
`DOUBLE`/`LONG` gains.
Pending values are included in `getBufferedSize()` (so page-sizing decisions
remain correct) and flushed in `getBytes()`. `reset()` and `close()` clear
pending state. Only the four numeric subclasses (Float/Double/Integer/Long) use
the batching path; `FixedLenByteArrayByteStreamSplitValuesWriter` continues to
use `scatterBytes(byte[])` since its values arrive as already-laid-out byte
arrays.
### Benchmark
New `ByteStreamSplitEncodingBenchmark` (100k values per invocation, JDK 18,
JMH `-wi 5 -i 10 -f 3`, 30 samples per row):
| Type | Before | After | Δ | Alloc B/op |
|--------|--------:|--------:|---------------:|------------------|
| Float | 15.08M | 65.06M | **+331% (4.3x)** | 33.27 → 9.27 (**-72%**) |
| Double | 6.99M | 49.48M | **+608% (7.1x)** | 42.54 → 18.55 (**-56%**) |
| Int | 15.64M | 68.13M | **+335% (4.4x)** | 33.27 → 9.27 (**-72%**) |
| Long | 7.09M | 53.23M | **+651% (7.5x)** | 42.54 → 18.55 (**-56%**) |
The remaining per-op allocation (~9 B/op for Int/Float, ~19 B/op for
Long/Double) is the `BytesInput[]` returned by `getBytes()` and the streams'
internal slabs, which are amortised across the page rather than per value.
### Are these changes tested?
Yes. All 573 `parquet-column` tests pass; 51 BSS-specific tests pass (`mvn
test -pl parquet-column -Dtest='*ByteStreamSplit*'`). No new test was added
because behaviour is unchanged (covered by the existing round-trip and writer
tests).
### Are there any user-facing changes?
No. Only an internal writer optimization. No public API, file format, or
configuration change.
### Closes #3503
Part of a small series of focused performance PRs from work in
[parquet-perf](https://github.com/iemejia/parquet-perf). Previous: #3494,
#3496, #3500.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
