Hi Antoine, I wrote some more code to answer your question, details here: https://termbin.com/0zzc8.
I added a nullability knob to the benchmark that writes an optional outer array with zero nulls actually present (your "empty lists need \ a def level" case, but at 100% presence) and verified on disk that this really is maxDef=2 (asserted in a test). The results are: arm maxdef=1 (required) maxdef=2 (optional, 0 nulls) delta B/VECTOR 1401 +/- 6 1403 +/- 12 +0.2% (same required file; no nullable form) C-hint 1436 +/- 8 1428 +/- 8 -0.6% (annotation-aware skip; FLAT) C-naive 2606 +/- 11 2603 +/- 10 -0.1% (full Dremel decode) LIST 2613 +/- 11 2617 +/- 13 +0.1% Note that the option B number above isn't actually possible with the optional array, since Rok's impl only kicks in on required arrays, but I included it here for reference to show the optional array doesn't slow option C down. On Thu, Jul 9, 2026 at 11:10 PM Antoine Pitrou <[email protected]> wrote: > > Le 10/07/2026 à 00:13, Andrew McCormick via dev a écrit : > > I built a prototype of hint-supported reads for option C on top of Rok's > > work. Here's the results I see: > > > > arm ns/row (mean +/- sd) note > > A/FLBA 2730 +/- 13 no levels on disk (FLBA->float > > reinterpret adds a bit) > > B/VECTOR 2337 +/- 10 no levels, not forward compatible > > C-hint 2356 +/- 5 skip-levels reader on a plain > > LIST; fully backward-compatible > > C-dremel 3830 +/- 22 annotation ignored, full Dremel > > (aka what Rok measured) > > > > So basically when you use the hint C is within noise of B (<1%). Full > > details and code here: https://termbin.com/kj2x > > (gist isn't availble on my db github). > > One remaining question is what happens for definition levels (not > repetition) in option C. Empty lists need a specific definition level to > encode, therefore option C makes the max def level larger than option B. > > An optional column with option C might therefore take more time decoding > than option B (especially if crossing the threshold from 1-bit to 2-bit > levels). > > Regards > > Antoine. > > > > > > > > > > > On Tue, Jul 7, 2026 at 8:59 AM Gunnar Morling < > [email protected]> > > wrote: > > > >> On Tue, 7 Jul 2026 at 17:18, Antoine Pitrou <[email protected]> wrote: > >>> > >>> Le 06/07/2026 à 23:29, Gunnar Morling a écrit : > >>>>>> 2. Without the logical type (and with a little bit extra complexity) > >> a > >>>>>> smart enough *reader* can walk the def/rep levels before decoding, > >> infer > >>>> > >>>>> At the cost of higher implementation complexity and maintenance cost. > >>>>> Does any mainstream open source implementation of Parquet do this? > >>>> > >>>> Triggered by the conversation on the call last week, I implemented > >>>> pretty much this in Hardwood [1]. > >>> > >>> Great, thank you. `FixedSizeListDetector.java` is highly non-trivial > and > >>> definitely has a maintenance cost. Though part of it seems about not > >>> having a RLE parser abstraction available. > >> > >> Yes, I think we all agree that a dedicated type will make maintainers' > >> lives much easier and is the right solution eventually. But until that > >> has landed, I think there's some juice worth the squeeze here. > >> > >>> > >>> Regards > >>> > >>> Antoine. > >>> > >>> > >> > > > > >
