Rich-T-kid opened a new issue, #12197:
URL: https://github.com/apache/arrow-rs/issues/12197
Parent: #12195
### Problem
A requested batch can cross from one Parquet dictionary into another, or
from dictionary-encoded data into a plain-encoded page. `DictionaryBuffer` then
spills its preserved dictionary into flat byte values so the reader can return
one Arrow dictionary array.
```text
requested batch
dictionary A rows dictionary B rows
<------------------><---------------------->
|
| boundary inside batch
v
spill and rebuild dictionary
```
A simpler way to preserve the zero-copy dictionary path is to stop the batch
at the boundary:
```text
batch 1 batch 2
dictionary A only dictionary B only
<----------------> <---------------->
no dictionary merge or spill
```
### Proposed change
Evaluate allowing the Parquet array reader to return a short batch before it
changes dictionary state. The next batch would begin with the new dictionary or
plain encoding.
This should avoid materializing dictionary values, but it may produce
smaller or irregular batches. That can move work downstream into
`BatchCoalescer`, filtering, or repartitioning, so the change should be driven
by end-to-end results rather than only a Parquet microbenchmark.
### Requirements
- Handle dictionary-to-dictionary and dictionary-to-plain boundaries.
- Preserve row selections, filtering, nulls, and reader progress across a
short batch.
- Do not emit empty batches or loop without consuming input.
- Benchmark aligned and deliberately misaligned batch sizes.
- Report output batch-size distributions as well as Parquet decode time.
- Include an end-to-end benchmark that captures downstream coalescing and
repartitioning costs.
- Compare against both the current spill path and the dictionary-remapping
design in #12195.
If smaller batches erase the decoding gain, this should remain an explored
alternative rather than becoming the default reader behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
