Hey all,
As briefly mentioned before, we've implemented a fixed-length list
fast path in Hardwood; it detects effectively fixed-length lists by
scanning the encoded def/rep level streams and bypasses the standard
Dremel reconstruction in this case. For larger lists, e.g. 768-element
vector embeddings, this brings read times to the level of a flat
column with the same data.
I just published a blog post about it, perhaps it's interesting to folks here:
https://www.morling.dev/blog/fast-path-for-fixed-length-lists-in-parquet/
I think this can be a nice interim read-side optimization until a
native FIXED_SIZE_LIST type lands in the spec. Would love to hear from
others too if they apply similar techniques.
All the best,
--Gunnar
On Sat, 11 Jul 2026 at 02:06, Rok Mihevc <[email protected]> wrote:
>
> Hi all,
>
> Thanks everyone for the discussion so far! Let me summarize where I believe
> we are and propose a variant of Option B.
>
> The three options represent vectors differently at the schema and physical
> levels:
>
> Option A
> - Proposal: represent vectors using FLBA
> - Pros: simple and efficient, close to representations used by existing
> engines
> - Cons: limited composability, no element nullability, and no natural
> support for element-wise encodings
> Option B
> - Proposal: add a VECTOR repetition type to allow the VECTOR dimension
> to be omitted from repetition levels
> - Pros: avoids storing and processing redundant repetition levels,
> performs best on writes in current prototypes
> - Cons: introduces a new repetition type, with unclear compatibility
> and implementation impact. (can we estimate this somehow??)
> Option C
> - Proposal: annotate the existing LIST representation with a logical
> type containing the vector length
> - Pros: allows optimizing readers to avoid reading redundant repetition
> levels, Andrew's prototype shows read performance can match Option B,
> strongest compatibility, and implementation impact, optimizations can
> follow at pace they choose
> - Cons: redundant repetition levels are written
>
>
> > On Mon, Jul 6, 2026 at 5:03 PM Alkis Evlogimenos <
> [email protected]> wrote:
> >
> > 1. The logical type makes the implementation that assumes all arrays are
> fixed length, on par in performance with Option B
> > 2. Without the logical type (and with a little bit extra complexity) a
> smart enough *reader* can walk the def/rep levels before decoding, infer
> (1) - the writer wrote fixed len arrays - and call the implementation in
> (1). Compared to (1) this is 1.5x slower.
>
> Then nothing is stopping implementations from doing this today? Gunnar
> already merged this. We should document it as a useful implementation
> optimization.
>
>
> I have been thinking about whether we could obtain Option B-like levels
> without introducing a new repetition type. One possibility would be to add
> an optional i32 length field to SchemaElement, allowing us to represent
> types such as:
>
> optional float embedding[3];
>
> Here, optional controls the nullability of the entire vector, while length
> = 3 means that every present vector contains exactly three floats. The
> fixed dimension would not contribute
> a repetition level. The presence of length would itself carry the
> fixed-vector semantics, so a separate VECTOR logical annotation may not be
> necessary.
>
> Given the following three vectors:
>
> [1.0, 2.0, 3.0]
> null
> [4.0, 5.0, 6.0]
>
> the physical representation would be:
>
> repetition levels: omitted
> definition levels: [1, 0, 1]
> physical values: [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
>
> Reader would use SchemaElement.length=3 to interpret this. VECTOR-unaware
> readers must therefore reject the vector columns. One way to provide that
> gate would be to require VECTOR-specific page encodings, such as
> VECTOR_PLAIN and VECTOR_BYTE_STREAM_SPLIT, on pages with such a column. A
> VECTOR-unaware reader would reject the unsupported encoding before
> attempting to decode the page.
>
> The other possible gating mechanism is, of course, versioning. :)
>
> The primary purpose of a VECTOR encoding here would be to gate out
> VECTOR-unaware readers cleanly. There may also be another motivation:
> vector similarity workloads may benefit from storing values by
> dimension rather than by vector.
>
> For M vectors of length N:
> - By vector: v0[0] ... v0[N], v1[0] ... v1[N], ...
> - By dimension: v0[0] ... vM[0], v0[1] ... vM[1], ...
>
> This layout is discussed in the PDX paper [1] and would naturally be
> represented as a page encoding in parquet.
>
> Would an augmented Option B along these lines be worth exploring?
>
> Rok
>
> [1] https://arxiv.org/pdf/2503.04422
>
> On Fri, Jul 10, 2026 at 7:02 PM Andrew McCormick via dev <
> [email protected]> wrote:
>
> > Hi Antoine,
> >
> > I wrote some more code to answer your question, details here:
> > https://termbin.com/0zzc8.
> >
> > I added a nullability knob to the benchmark that writes an optional outer
> > array with zero nulls actually present (your "empty lists need \
> > a def level" case, but at 100% presence) and verified on disk that this
> > really is maxDef=2 (asserted in a test). The results are:
> > arm maxdef=1 (required) maxdef=2 (optional, 0 nulls) delta
> > B/VECTOR 1401 +/- 6 1403 +/- 12 +0.2%
> > (same required file; no nullable form)
> > C-hint 1436 +/- 8 1428 +/- 8 -0.6%
> > (annotation-aware skip; FLAT)
> > C-naive 2606 +/- 11 2603 +/- 10 -0.1%
> > (full Dremel decode)
> > LIST 2613 +/- 11 2617 +/- 13 +0.1%
> >
> > Note that the option B number above isn't actually possible with the
> > optional array, since Rok's impl only kicks in on required arrays, but I
> > included it here for reference to show the optional array doesn't slow
> > option C down.
> >
> >
> > On Thu, Jul 9, 2026 at 11:10 PM Antoine Pitrou <[email protected]> wrote:
> >
> > >
> > > Le 10/07/2026 à 00:13, Andrew McCormick via dev a écrit :
> > > > I built a prototype of hint-supported reads for option C on top of
> > Rok's
> > > > work. Here's the results I see:
> > > >
> > > > arm ns/row (mean +/- sd) note
> > > > A/FLBA 2730 +/- 13 no levels on disk (FLBA->float
> > > > reinterpret adds a bit)
> > > > B/VECTOR 2337 +/- 10 no levels, not forward compatible
> > > > C-hint 2356 +/- 5 skip-levels reader on a plain
> > > > LIST; fully backward-compatible
> > > > C-dremel 3830 +/- 22 annotation ignored, full Dremel
> > > > (aka what Rok measured)
> > > >
> > > > So basically when you use the hint C is within noise of B (<1%). Full
> > > > details and code here: https://termbin.com/kj2x
> > > > (gist isn't availble on my db github).
> > >
> > > One remaining question is what happens for definition levels (not
> > > repetition) in option C. Empty lists need a specific definition level to
> > > encode, therefore option C makes the max def level larger than option B.
> > >
> > > An optional column with option C might therefore take more time decoding
> > > than option B (especially if crossing the threshold from 1-bit to 2-bit
> > > levels).
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > >
> > > >
> > > >
> > > >
> > > > On Tue, Jul 7, 2026 at 8:59 AM Gunnar Morling <
> > > [email protected]>
> > > > wrote:
> > > >
> > > >> On Tue, 7 Jul 2026 at 17:18, Antoine Pitrou <[email protected]>
> > wrote:
> > > >>>
> > > >>> Le 06/07/2026 à 23:29, Gunnar Morling a écrit :
> > > >>>>>> 2. Without the logical type (and with a little bit extra
> > complexity)
> > > >> a
> > > >>>>>> smart enough *reader* can walk the def/rep levels before decoding,
> > > >> infer
> > > >>>>
> > > >>>>> At the cost of higher implementation complexity and maintenance
> > cost.
> > > >>>>> Does any mainstream open source implementation of Parquet do this?
> > > >>>>
> > > >>>> Triggered by the conversation on the call last week, I implemented
> > > >>>> pretty much this in Hardwood [1].
> > > >>>
> > > >>> Great, thank you. `FixedSizeListDetector.java` is highly non-trivial
> > > and
> > > >>> definitely has a maintenance cost. Though part of it seems about not
> > > >>> having a RLE parser abstraction available.
> > > >>
> > > >> Yes, I think we all agree that a dedicated type will make maintainers'
> > > >> lives much easier and is the right solution eventually. But until that
> > > >> has landed, I think there's some juice worth the squeeze here.
> > > >>
> > > >>>
> > > >>> Regards
> > > >>>
> > > >>> Antoine.
> > > >>>
> > > >>>
> > > >>
> > > >
> > >
> > >
> > >
> >
