Hi all, Following the discussion here and on the community call, I've opened a PR for the LIST-based VECTOR logical type:
https://github.com/apache/parquet-format/pull/624 This proposal separates the logical contract from the physical-layout changes: - Every non-null vector has exactly num_elements. - Elements must be non-null and finite. - Elements are numeric or boolean Parquet primitive types. No nesetd elements. Writers must enforce these constraints and readers may rely on them. VECTOR-unaware readers read VECTOR as ordinary LIST. Physical layout changes, vector indexes, quantisation and normalisation guarantees are out of scope for this proposal. Can we agree to move this logical annotation forward independently of the physical-layout proposal? Feedback on the scope and details is welcome, here or on the PR. This proposal closely resembles Option C in the original design doc [1], which also contains the earlier discussion. [1] https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0 Best, Rok On Wed, Sep 9, 2026 at 1:08 PM Rok Mihevc <[email protected]> wrote: > Antoine: > > > * Vectors (as in vector DB) storage needs are different to what we > need for > > > fixed size list applications we've been disucssing so far [0]. > > > > This seems unsubstantiated. Can you please provide an explanation? > > Fair point, "different storage needs" is too broad. What we need from a > vector logical type is a set of constraints (contract) that we currently > can't express: > 1. Vector has a fixed number of numeric elements > 2. Elements cannot be null, are finite or rather make sense as vectors in > a space > 3. Potential future vector specific properties (normalization guarantee, > statistics) or vector specific encodings > > These constraints do not necessarily require a different physical > representation. For example Lance uses FixedSizeList with specialized > physical encodings, Hudi specifies vector as a single FLBA [2] in parquet > with vector semantics carried in metadata. > > [1] Lance vector - > https://github.com/lance-format/lance/blob/main/docs/src/format/table/schema.md#fixed-size-list-types > [2] Hudi vector - > https://github.com/apache/hudi/blob/master/rfc/rfc-99/vector-appendix.md#vector-schema-constraints > --- > > Will: > > - Vector indexes, along the lines of Lance: how does a reader discover > > that a column or auxiliary structure indexes another column, or is > derived > > from it? What lets an engine recognise when it can use that structure? > > > > - Quantisation, normalisation guarantees, and additional numeric > formats > > such as BF16/FP8. We already have FLOAT16 and small integer annotations, > > but a coordinate type alone doesn't describe all of these. Some > quantised > > representations are naturally composite values: packed coordinates plus > > scales, offsets, or codebooks. Block-scaled FP4 is a concrete example. > > These could be structs with separate children for codes and scales, and a > > logical annotation describing how they collectively represent a vector. > > Shared parameters might live elsewhere. Either way, the relationship > needs > > expressing. > > Lance stores the index as a separate structure and table metadata maps the > index to the vector [3], the index carries information about distance > metric and quantization [4]. > As per Option C1 we would add new types to parquet's type system. Under > Option A we would have a vector element-type enum that we could amend. > Expressing more information like codebooks and quantizations seems to > require another logical type or other rather than tacking on type > parameters here. > > [3] Lance index-to-column binding - > https://github.com/lance-format/lance/blob/main/docs/src/format/index/index.md#creating-and-updating-index-segments > [4] Vector index storage - > https://github.com/lance-format/lance/blob/main/docs/src/format/index/vector/index.md#storage-layout-v3 > --- > > Daniel: > > I feel like the intent behind having a logical type for vector is to > > differentiate its semantics from a simple fixed size list. Vectors > > shouldn't contain null, NaN, inf as elements (in practice they may > present, > > but those are typically due to erroneous handling). I would expect > vectors > > to prohibit special/missing values that render them invalid. Are the > > statistics intended to detect these erroneous scenarios or is the intent > to > > allow these values? > > The intent is to prohibit non-finite values and writers should enforce > this. However using stats would allow readers to act defensively and > validate data at read time. > > On Wed, Sep 9, 2026 at 1:09 AM Daniel Weeks <[email protected]> wrote: > >> Thanks for the summary Rok, >> >> I like the direction C1 is going, but I'm a little concerned about the >> following statement: >> >> Regarding NaN/Inf limitations, the consensus was that these should be >> > handled by statistics. This leads to a need for a new inf_count >> statustics. >> >> >> I feel like the intent behind having a logical type for vector is to >> differentiate its semantics from a simple fixed size list. Vectors >> shouldn't contain null, NaN, inf as elements (in practice they may >> present, >> but those are typically due to erroneous handling). I would expect >> vectors >> to prohibit special/missing values that render them invalid. Are the >> statistics intended to detect these erroneous scenarios or is the intent >> to >> allow these values? >> >> -Dan >> >> >> On Tue, Sep 8, 2026 at 1:41 PM Will Edwards via dev < >> [email protected]> >> wrote: >> >> > Hi all, >> > >> > Andrew's and Gunnar's results show that existing LIST storage can be >> read >> > efficiently. This matches my own experience too. I think efficiency is a >> > software problem not a Parquet format problem. E.g. the reader can >> expose >> > flat typed memory; nothing in the format requires allocating a >> List<Float> >> > per row. A nullable schema doesn't change that for pages without nulls, >> > which the reader can establish at page scope. >> > >> > So is the immediate question what SHOW CREATE TABLE should say when the >> > only source of schema is a Parquet file? ARRAY<FLOAT>, a fixed-length >> > array, or VECTOR<FLOAT, 768>? That's a useful discussion about the >> contract >> > we're expressing. >> > >> > SELECT COS_SIM(a, b) etc can already work on numeric lists, with lengths >> > checked efficiently through the encoded levels. Whether a and b belong >> to >> > the same embedding space remains the user's responsibility; declaring >> both >> > to have 768 dimensions doesn't establish that. >> > >> > There are other semantics I'd be interested in discussing: >> > >> > - Vector indexes, along the lines of Lance: how does a reader discover >> > that a column or auxiliary structure indexes another column, or is >> derived >> > from it? What lets an engine recognise when it can use that structure? >> > >> > - Quantisation, normalisation guarantees, and additional numeric >> formats >> > such as BF16/FP8. We already have FLOAT16 and small integer annotations, >> > but a coordinate type alone doesn't describe all of these. Some >> quantised >> > representations are naturally composite values: packed coordinates plus >> > scales, offsets, or codebooks. Block-scaled FP4 is a concrete example. >> > These could be structs with separate children for codes and scales, and >> a >> > logical annotation describing how they collectively represent a vector. >> > Shared parameters might live elsewhere. Either way, the relationship >> needs >> > expressing. >> > >> > Should we pin down those requirements before choosing a physical >> > representation? Adding an element-type enum doesn't answer all of >> these, >> > and packing whole vectors into FLBA limits existing element encoding >> > choices. >> > >> > What concrete vector workload still needs a physical change after >> applying >> > the reader optimisations we've discussed? >> > >> > Will >> > >> > On Tue, 8 Sept 2026 at 21:22, Antoine Pitrou <[email protected]> >> wrote: >> > >> > > >> > > Le 08/09/2026 à 20:42, Rok Mihevc a écrit : >> > > > Hi all, >> > > > >> > > > As proposed during the general community call we held a focused call >> > > today >> > > > to discuss the proposal for adding a logical vector type to parquet. >> > This >> > > > was called because it seems: >> > > > >> > > > * Vectors (as in vector DB) storage needs are different to what we >> need >> > > for >> > > > fixed size list applications we've been disucssing so far [0]. >> > > >> > > This seems unsubstantiated. Can you please provide an explanation? >> > > >> > > Ideally we'd use the *same* underlying solution for both problems. >> > > >> > > > We have converged on two options that could work, but have not yet >> > > decided >> > > > which one to propose: >> > > > >> > > > - Option A - FLBA as physical storage (FLBA(num_elements * >> > > element_width)). >> > > > Vector logical type would provide an enum specifying the element >> type, >> > > and >> > > > the element would be non-nullable. The element type enum could be >> > > > independent of the parquet type system, making evolution easier. >> > Downside >> > > > is loss of element-level encodings available today, though they can >> be >> > > > added later. Upside is simplicity and ease of evolution. >> > > > >> > > > - Option C1 - LIST with numeric parquet type elements as physical >> > > storage. >> > > > New float types (e.g. BF16) can be added in parquet proper as >> needed. >> > > > Downside is we need to add new types to parquet, DL/RL overhead is >> > likely >> > > > not a concern. Upside is we have encoding support now. >> > > >> > > Option A is a short-term solution with severe limitations (lack of >> > > encoding choices) that make me entirely negative about the idea. >> > > >> > > I'm lukewarm about Option C (especially the optimistic statement that >> > > "DL/RL overhead is likely not a concern"), but at least it doesn't >> paint >> > > us in a corner. >> > > >> > > Regards >> > > >> > > Antoine. >> > > >> > > >> > > >> > >> >
