Le 10/07/2026 à 00:13, Andrew McCormick via dev a écrit :
I built a prototype of hint-supported reads for option C on top of Rok's
work. Here's the results I see:
arm ns/row (mean +/- sd) note
A/FLBA 2730 +/- 13 no levels on disk (FLBA->float
reinterpret adds a bit)
B/VECTOR 2337 +/- 10 no levels, not forward compatible
C-hint 2356 +/- 5 skip-levels reader on a plain
LIST; fully backward-compatible
C-dremel 3830 +/- 22 annotation ignored, full Dremel
(aka what Rok measured)
So basically when you use the hint C is within noise of B (<1%). Full
details and code here: https://termbin.com/kj2x
(gist isn't availble on my db github).
One remaining question is what happens for definition levels (not
repetition) in option C. Empty lists need a specific definition level to
encode, therefore option C makes the max def level larger than option B.
An optional column with option C might therefore take more time decoding
than option B (especially if crossing the threshold from 1-bit to 2-bit
levels).
Regards
Antoine.
On Tue, Jul 7, 2026 at 8:59 AM Gunnar Morling <[email protected]>
wrote:
On Tue, 7 Jul 2026 at 17:18, Antoine Pitrou <[email protected]> wrote:
Le 06/07/2026 à 23:29, Gunnar Morling a écrit :
2. Without the logical type (and with a little bit extra complexity)
a
smart enough *reader* can walk the def/rep levels before decoding,
infer
At the cost of higher implementation complexity and maintenance cost.
Does any mainstream open source implementation of Parquet do this?
Triggered by the conversation on the call last week, I implemented
pretty much this in Hardwood [1].
Great, thank you. `FixedSizeListDetector.java` is highly non-trivial and
definitely has a maintenance cost. Though part of it seems about not
having a RLE parser abstraction available.
Yes, I think we all agree that a dedicated type will make maintainers'
lives much easier and is the right solution eventually. But until that
has landed, I think there's some juice worth the squeeze here.
Regards
Antoine.
