Thank you Gunnar for pushing this in hardwood and blogging about it. This verifies the same work Andrew did internally on Photon.
With the above in mind the fixed size list discussion shifts from physical type changes to parquet to parser quality of implementation. The logical type discussion is still open: do we need a vector type with more restrictive definition of what vector is? On Sat, Jul 11, 2026 at 3:06 AM Rok Mihevc <[email protected]> wrote: > Hi all, > > Thanks everyone for the discussion so far! Let me summarize where I believe > we are and propose a variant of Option B. > > The three options represent vectors differently at the schema and physical > levels: > > Option A > - Proposal: represent vectors using FLBA > - Pros: simple and efficient, close to representations used by existing > engines > - Cons: limited composability, no element nullability, and no natural > support for element-wise encodings > Option B > - Proposal: add a VECTOR repetition type to allow the VECTOR dimension > to be omitted from repetition levels > - Pros: avoids storing and processing redundant repetition levels, > performs best on writes in current prototypes > - Cons: introduces a new repetition type, with unclear compatibility > and implementation impact. (can we estimate this somehow??) > Option C > - Proposal: annotate the existing LIST representation with a logical > type containing the vector length > - Pros: allows optimizing readers to avoid reading redundant repetition > levels, Andrew's prototype shows read performance can match Option B, > strongest compatibility, and implementation impact, optimizations can > follow at pace they choose > - Cons: redundant repetition levels are written > > > > On Mon, Jul 6, 2026 at 5:03 PM Alkis Evlogimenos < > [email protected]> wrote: > > > > 1. The logical type makes the implementation that assumes all arrays are > fixed length, on par in performance with Option B > > 2. Without the logical type (and with a little bit extra complexity) a > smart enough *reader* can walk the def/rep levels before decoding, infer > (1) - the writer wrote fixed len arrays - and call the implementation in > (1). Compared to (1) this is 1.5x slower. > > Then nothing is stopping implementations from doing this today? Gunnar > already merged this. We should document it as a useful implementation > optimization. > > > I have been thinking about whether we could obtain Option B-like levels > without introducing a new repetition type. One possibility would be to add > an optional i32 length field to SchemaElement, allowing us to represent > types such as: > > optional float embedding[3]; > > Here, optional controls the nullability of the entire vector, while length > = 3 means that every present vector contains exactly three floats. The > fixed dimension would not contribute > a repetition level. The presence of length would itself carry the > fixed-vector semantics, so a separate VECTOR logical annotation may not be > necessary. > > Given the following three vectors: > > [1.0, 2.0, 3.0] > null > [4.0, 5.0, 6.0] > > the physical representation would be: > > repetition levels: omitted > definition levels: [1, 0, 1] > physical values: [1.0, 2.0, 3.0, 4.0, 5.0, 6.0] > > Reader would use SchemaElement.length=3 to interpret this. VECTOR-unaware > readers must therefore reject the vector columns. One way to provide that > gate would be to require VECTOR-specific page encodings, such as > VECTOR_PLAIN and VECTOR_BYTE_STREAM_SPLIT, on pages with such a column. A > VECTOR-unaware reader would reject the unsupported encoding before > attempting to decode the page. > > The other possible gating mechanism is, of course, versioning. :) > > The primary purpose of a VECTOR encoding here would be to gate out > VECTOR-unaware readers cleanly. There may also be another motivation: > vector similarity workloads may benefit from storing values by > dimension rather than by vector. > > For M vectors of length N: > - By vector: v0[0] ... v0[N], v1[0] ... v1[N], ... > - By dimension: v0[0] ... vM[0], v0[1] ... vM[1], ... > > This layout is discussed in the PDX paper [1] and would naturally be > represented as a page encoding in parquet. > > Would an augmented Option B along these lines be worth exploring? > > Rok > > [1] https://arxiv.org/pdf/2503.04422 > > On Fri, Jul 10, 2026 at 7:02 PM Andrew McCormick via dev < > [email protected]> wrote: > > > Hi Antoine, > > > > I wrote some more code to answer your question, details here: > > https://termbin.com/0zzc8. > > > > I added a nullability knob to the benchmark that writes an optional outer > > array with zero nulls actually present (your "empty lists need \ > > a def level" case, but at 100% presence) and verified on disk that this > > really is maxDef=2 (asserted in a test). The results are: > > arm maxdef=1 (required) maxdef=2 (optional, 0 nulls) delta > > B/VECTOR 1401 +/- 6 1403 +/- 12 +0.2% > > (same required file; no nullable form) > > C-hint 1436 +/- 8 1428 +/- 8 -0.6% > > (annotation-aware skip; FLAT) > > C-naive 2606 +/- 11 2603 +/- 10 -0.1% > > (full Dremel decode) > > LIST 2613 +/- 11 2617 +/- 13 +0.1% > > > > Note that the option B number above isn't actually possible with the > > optional array, since Rok's impl only kicks in on required arrays, but I > > included it here for reference to show the optional array doesn't slow > > option C down. > > > > > > On Thu, Jul 9, 2026 at 11:10 PM Antoine Pitrou <[email protected]> > wrote: > > > > > > > > Le 10/07/2026 à 00:13, Andrew McCormick via dev a écrit : > > > > I built a prototype of hint-supported reads for option C on top of > > Rok's > > > > work. Here's the results I see: > > > > > > > > arm ns/row (mean +/- sd) note > > > > A/FLBA 2730 +/- 13 no levels on disk (FLBA->float > > > > reinterpret adds a bit) > > > > B/VECTOR 2337 +/- 10 no levels, not forward > compatible > > > > C-hint 2356 +/- 5 skip-levels reader on a plain > > > > LIST; fully backward-compatible > > > > C-dremel 3830 +/- 22 annotation ignored, full Dremel > > > > (aka what Rok measured) > > > > > > > > So basically when you use the hint C is within noise of B (<1%). Full > > > > details and code here: https://termbin.com/kj2x > > > > (gist isn't availble on my db github). > > > > > > One remaining question is what happens for definition levels (not > > > repetition) in option C. Empty lists need a specific definition level > to > > > encode, therefore option C makes the max def level larger than option > B. > > > > > > An optional column with option C might therefore take more time > decoding > > > than option B (especially if crossing the threshold from 1-bit to 2-bit > > > levels). > > > > > > Regards > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 7, 2026 at 8:59 AM Gunnar Morling < > > > [email protected]> > > > > wrote: > > > > > > > >> On Tue, 7 Jul 2026 at 17:18, Antoine Pitrou <[email protected]> > > wrote: > > > >>> > > > >>> Le 06/07/2026 à 23:29, Gunnar Morling a écrit : > > > >>>>>> 2. Without the logical type (and with a little bit extra > > complexity) > > > >> a > > > >>>>>> smart enough *reader* can walk the def/rep levels before > decoding, > > > >> infer > > > >>>> > > > >>>>> At the cost of higher implementation complexity and maintenance > > cost. > > > >>>>> Does any mainstream open source implementation of Parquet do > this? > > > >>>> > > > >>>> Triggered by the conversation on the call last week, I implemented > > > >>>> pretty much this in Hardwood [1]. > > > >>> > > > >>> Great, thank you. `FixedSizeListDetector.java` is highly > non-trivial > > > and > > > >>> definitely has a maintenance cost. Though part of it seems about > not > > > >>> having a RLE parser abstraction available. > > > >> > > > >> Yes, I think we all agree that a dedicated type will make > maintainers' > > > >> lives much easier and is the right solution eventually. But until > that > > > >> has landed, I think there's some juice worth the squeeze here. > > > >> > > > >>> > > > >>> Regards > > > >>> > > > >>> Antoine. > > > >>> > > > >>> > > > >> > > > > > > > > > > > > > > > >
