Hi all,
Thanks everyone for the discussion so far! Let me summarize where I believe
we are and propose a variant of Option B.
The three options represent vectors differently at the schema and physical
levels:
Option A
- Proposal: represent vectors using FLBA
- Pros: simple and efficient, close to representations used by existing
engines
- Cons: limited composability, no element nullability, and no natural
support for element-wise encodings
Option B
- Proposal: add a VECTOR repetition type to allow the VECTOR dimension
to be omitted from repetition levels
- Pros: avoids storing and processing redundant repetition levels,
performs best on writes in current prototypes
- Cons: introduces a new repetition type, with unclear compatibility
and implementation impact. (can we estimate this somehow??)
Option C
- Proposal: annotate the existing LIST representation with a logical
type containing the vector length
- Pros: allows optimizing readers to avoid reading redundant repetition
levels, Andrew's prototype shows read performance can match Option B,
strongest compatibility, and implementation impact, optimizations can
follow at pace they choose
- Cons: redundant repetition levels are written
> On Mon, Jul 6, 2026 at 5:03 PM Alkis Evlogimenos <
[email protected]> wrote:
>
> 1. The logical type makes the implementation that assumes all arrays are
fixed length, on par in performance with Option B
> 2. Without the logical type (and with a little bit extra complexity) a
smart enough *reader* can walk the def/rep levels before decoding, infer
(1) - the writer wrote fixed len arrays - and call the implementation in
(1). Compared to (1) this is 1.5x slower.
Then nothing is stopping implementations from doing this today? Gunnar
already merged this. We should document it as a useful implementation
optimization.
I have been thinking about whether we could obtain Option B-like levels
without introducing a new repetition type. One possibility would be to add
an optional i32 length field to SchemaElement, allowing us to represent
types such as:
optional float embedding[3];
Here, optional controls the nullability of the entire vector, while length
= 3 means that every present vector contains exactly three floats. The
fixed dimension would not contribute
a repetition level. The presence of length would itself carry the
fixed-vector semantics, so a separate VECTOR logical annotation may not be
necessary.
Given the following three vectors:
[1.0, 2.0, 3.0]
null
[4.0, 5.0, 6.0]
the physical representation would be:
repetition levels: omitted
definition levels: [1, 0, 1]
physical values: [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
Reader would use SchemaElement.length=3 to interpret this. VECTOR-unaware
readers must therefore reject the vector columns. One way to provide that
gate would be to require VECTOR-specific page encodings, such as
VECTOR_PLAIN and VECTOR_BYTE_STREAM_SPLIT, on pages with such a column. A
VECTOR-unaware reader would reject the unsupported encoding before
attempting to decode the page.
The other possible gating mechanism is, of course, versioning. :)
The primary purpose of a VECTOR encoding here would be to gate out
VECTOR-unaware readers cleanly. There may also be another motivation:
vector similarity workloads may benefit from storing values by
dimension rather than by vector.
For M vectors of length N:
- By vector: v0[0] ... v0[N], v1[0] ... v1[N], ...
- By dimension: v0[0] ... vM[0], v0[1] ... vM[1], ...
This layout is discussed in the PDX paper [1] and would naturally be
represented as a page encoding in parquet.
Would an augmented Option B along these lines be worth exploring?
Rok
[1] https://arxiv.org/pdf/2503.04422
On Fri, Jul 10, 2026 at 7:02 PM Andrew McCormick via dev <
[email protected]> wrote:
> Hi Antoine,
>
> I wrote some more code to answer your question, details here:
> https://termbin.com/0zzc8.
>
> I added a nullability knob to the benchmark that writes an optional outer
> array with zero nulls actually present (your "empty lists need \
> a def level" case, but at 100% presence) and verified on disk that this
> really is maxDef=2 (asserted in a test). The results are:
> arm maxdef=1 (required) maxdef=2 (optional, 0 nulls) delta
> B/VECTOR 1401 +/- 6 1403 +/- 12 +0.2%
> (same required file; no nullable form)
> C-hint 1436 +/- 8 1428 +/- 8 -0.6%
> (annotation-aware skip; FLAT)
> C-naive 2606 +/- 11 2603 +/- 10 -0.1%
> (full Dremel decode)
> LIST 2613 +/- 11 2617 +/- 13 +0.1%
>
> Note that the option B number above isn't actually possible with the
> optional array, since Rok's impl only kicks in on required arrays, but I
> included it here for reference to show the optional array doesn't slow
> option C down.
>
>
> On Thu, Jul 9, 2026 at 11:10 PM Antoine Pitrou <[email protected]> wrote:
>
> >
> > Le 10/07/2026 à 00:13, Andrew McCormick via dev a écrit :
> > > I built a prototype of hint-supported reads for option C on top of
> Rok's
> > > work. Here's the results I see:
> > >
> > > arm ns/row (mean +/- sd) note
> > > A/FLBA 2730 +/- 13 no levels on disk (FLBA->float
> > > reinterpret adds a bit)
> > > B/VECTOR 2337 +/- 10 no levels, not forward compatible
> > > C-hint 2356 +/- 5 skip-levels reader on a plain
> > > LIST; fully backward-compatible
> > > C-dremel 3830 +/- 22 annotation ignored, full Dremel
> > > (aka what Rok measured)
> > >
> > > So basically when you use the hint C is within noise of B (<1%). Full
> > > details and code here: https://termbin.com/kj2x
> > > (gist isn't availble on my db github).
> >
> > One remaining question is what happens for definition levels (not
> > repetition) in option C. Empty lists need a specific definition level to
> > encode, therefore option C makes the max def level larger than option B.
> >
> > An optional column with option C might therefore take more time decoding
> > than option B (especially if crossing the threshold from 1-bit to 2-bit
> > levels).
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
> > >
> > >
> > >
> > > On Tue, Jul 7, 2026 at 8:59 AM Gunnar Morling <
> > [email protected]>
> > > wrote:
> > >
> > >> On Tue, 7 Jul 2026 at 17:18, Antoine Pitrou <[email protected]>
> wrote:
> > >>>
> > >>> Le 06/07/2026 à 23:29, Gunnar Morling a écrit :
> > >>>>>> 2. Without the logical type (and with a little bit extra
> complexity)
> > >> a
> > >>>>>> smart enough *reader* can walk the def/rep levels before decoding,
> > >> infer
> > >>>>
> > >>>>> At the cost of higher implementation complexity and maintenance
> cost.
> > >>>>> Does any mainstream open source implementation of Parquet do this?
> > >>>>
> > >>>> Triggered by the conversation on the call last week, I implemented
> > >>>> pretty much this in Hardwood [1].
> > >>>
> > >>> Great, thank you. `FixedSizeListDetector.java` is highly non-trivial
> > and
> > >>> definitely has a maintenance cost. Though part of it seems about not
> > >>> having a RLE parser abstraction available.
> > >>
> > >> Yes, I think we all agree that a dedicated type will make maintainers'
> > >> lives much easier and is the right solution eventually. But until that
> > >> has landed, I think there's some juice worth the squeeze here.
> > >>
> > >>>
> > >>> Regards
> > >>>
> > >>> Antoine.
> > >>>
> > >>>
> > >>
> > >
> >
> >
> >
>
