Could we clarify the use cases for a VECTOR annotation on LIST<FLOAT>? Specifically, what does it make possible that's impossible today, and what does it make easier that's currently hard?
I'm struggling to find a strong answer. I'd like every engine to offer vector functions on lists of floats without needing a new logical type. To be clear, I'm still enthusiastic about a vector type that describes how to unpack and scale composite representations (quantised codes plus scales, codebooks, etc.), in the direction ML is going. That adds information engines don't have. A label on LIST<FLOAT> doesn't: engines already understand the physical type, and many already offer vector functions on it. I don't see a performance benefit, and NaN/null/Inf checks are cheap enough that engines should do them anyway. The label also doesn't say what you can do with the column. Knowing that a column holds embeddings doesn't say how they can be used, just as zipcode and age are both integers but multiplying them is meaningless. My concern is the opposite effect. We're establishing that engines should ignore logical types they don't understand. So a tool that doesn't know VECTOR will read the column as LIST<FLOAT>, let the user compute centroids (say), and write plain LIST<FLOAT> back out. If other engines restrict embedding functions to VECTOR-annotated columns, that file stops working with them, through no fault of the user. An annotation that doesn't change what the data is shouldn't be able to make it unusable. Am I missing a use case? Best, Will On Thu, 8 Oct 2026 at 19:53, Steven Wu <[email protected]> wrote: > I also think a FINITE_VECTOR type brings a lot of values to many use cases. > It is would be great to make progress on the FINITE_VECTOR, and not to fit > every use cases into the FIXED_LENGTH_LIST. > > On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <[email protected]> wrote: > > > Going through this thread again, it looks like there are a few things > that > > people generally agree on: > > > > - In the short term, we want to introduce a logical type or types; a > better > > physical representation can be done later. We can focus on logical types. > > - There is precedent in logical types for restricting values, like UTF8 > > strings to contain only UTF-8 codepoints. I think we also agree that we > > could restrict vector elements to non-nested or primitive types. > > - There are many examples of databases that restrict their vector types > to > > non-null, finite elements. > > - Some people want to have write-time guarantees that vectors have only > > non-null, finite elements > > - Some people want to extend this work to include tensors, shorter > floating > > point representations, variable lengths, NaN or Infinity elements, etc. > > > > It sounds like we are nearly there, but have two halves of the community > > that care about different things. That just means we should have multiple > > types. It's entirely reasonable to want a type that allows NaN values, > and > > it's also reasonable for those of us building databases, table standards, > > and vector indexes to care about guarantees that elements are finite. > > There's more than enough examples to show that this is a legitimate use > > case. > > > > Since I don't see the concerns of the different use cases changing, it > > looks like we need to find a compromise to move forward. I propose that > we > > introduce a FINITE_VECTOR type and separately a FIXED_LENGTH_LIST or > > another VECTOR type that allows infinite and NaN values. > > > > Can we agree on a compromise? Is there a strong reason to try to fit > these > > use cases into a single type? What if that means not making progress? > > > > Ryan > > > > On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev < > > [email protected]> wrote: > > > > > Thinking about this more, I am also fine with pulling out the “NaN/Inf” > > > requirement. Either into a type boolean (my preference), or putting no > > > restrictions on the Parquet level, allowing all IEEE 754 float numbers. > > If > > > some top-level application wants to put additional constraints on the > > data > > > type, like Iceberg not allowing NaN/infinity, it can still do so one > > layer > > > above. It seems like this was the biggest point of discussion > remaining. > > > All these options work for me, and we should move forward with this > > > proposal. > > > > > > Concerning additional element types like bf16, they can be introduced > as > > > Parquet types (at least in the current LIST-based spec proposal). > > > > > > Concerning Will’s suggestion around representing VECTOR as a variable > > > length list with a MIN/MAX_LENGTH metadata restriction: I think that a > > > dedicated logical VECTOR type still provides enough benefits by > > describing > > > the semantics of the data, and allowing to capture future changes > around > > > statistics. In that case, the length is better captured as an exact > > length > > > for this type specifically. I’m also not sure about other use cases for > > > MIN_LENGTH, so introducing a new type constraint system just for the > > > MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough value in my > > > view. > > > > > > Best, > > > Philipp > > > > > > > > > On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev < > > > [email protected]> > > > wrote: > > > > > > > +1 Sebastian, a vector type suggests usage beyond normalized > embeddings > > > and > > > > engines really aren't going to be able to elide all the checks > anyway. > > > > > > > > Antoine, you suggested we explore metadata solutions too :) > > > > > > > > Today, a database supporting CHAR(N) or VARCHAR(n) can't express this > > in > > > > Parquet. Engines can write opaque blobs they understand in the > > metadata > > > > (e.g. Arrow does) so the extra type info round-trips, but this isn't > > > > standardized, portable, nor interoperable. > > > > > > > > So perhaps standardizing constraint metadata or new Thrift fields (my > > > pref) > > > > can generically support VECTOR(N) whilst also being more broadly > useful > > > for > > > > other use cases and data types? > > > > > > > > Imagine metadata or Thrift fields to tag the MIN_LENGTH and > MAX_LENGTH > > > for > > > > Parquet's variable-length physcial types. These would count logical > > > units > > > > (e.g. UTF8 code points for STRING) rather than bytes. > > > > > > > > Then one engine's VECTOR(N) can be written to Parquet as LIST(N) with > > > > MIN_LENGTH = MAX_LENGTH = N and then readers - even different > engines - > > > can > > > > pick up these constraints if they support doing so. We don't > actually > > > need > > > > a logical type to express a VECTOR then. > > > > > > > > This also allows standard SQL's CHAR(N) and VARCHAR(N) to be > expressed > > on > > > > UTF8 columns etc. > > > > > > > > This also seems generally useful in the future as we approach > > expressing > > > > other tensor data shapes and types. > > > > > > > > This won't change the mechanics: the storage type doesn't enforce > these > > > > constraints and engines must check them. It's previously established > > that > > > > performance is just a software problem and there is no fundamental > > > > performance advantage to the fixed size types in Dremel-like formats > > like > > > > Parquet. > > > > > > > > Best, > > > > Will > > > > > > > > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard < > > > [email protected] > > > > > > > > > wrote: > > > > > > > > > After listening in on the last meeting Wednesday, I have been > > thinking > > > > > about this quite a bit, and thought I would share my 2 cents. > > > > > > > > > > I suggest not requiring VECTOR elements to be finite, as proposed > in > > > the > > > > PR > > > > > <https://github.com/apache/parquet-format/pull/624>. > > > > > > > > > > While Divjot and others note the specialization for UTF-8, Parquet > > > > already > > > > > defines DOUBLE and FLOAT as IEEE encodings > > > > > < > > https://github.com/apache/parquet-format/blob/master/README.md#types > > > >. > > > > > > > > > > Making NaN or +/-Inf illegal inside VECTOR would be a new > restriction > > > on > > > > > those same underlying physical types. Scalar FLOAT/DOUBLE columns > > have > > > > > never had such a rule. > > > > > > > > > > It would also not match how the systems that actually produce these > > > > vectors > > > > > treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB, > > NumPy, > > > > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS. > > > > > > > > > > pgvector's "all elements must be finite" rule is an insert policy. > > > While > > > > I > > > > > think Parquet as a general-purpose format should allow them out of > > the > > > > box. > > > > > It is a user system decision to disallow writing these values or to > > > throw > > > > > errors when encountering them. > > > > > > > > > > A type-level ban also does not keep Inf/NaN out of vector math. > > Finite > > > > > coordinates still overflow. For instance: magnitude = ||[FLOAT_MAX, > > > > > FLOAT_MAX]|| where every coordinate is finite overflows to +Inf. > > > PyTorch > > > > > documents the same overflow: tensor([1e20, 1e20]).norm() is inf in > > > fp32. > > > > > > > > > > A generated MAGNITUDE(vec) column would then be Inf while the > VECTOR > > > > itself > > > > > was required to be finite. The ban only forbids storing specials in > > the > > > > > coordinates, and does not prevent even simple downstream > computations > > > > from > > > > > producing them. > > > > > > > > > > I suggest keeping VECTOR as the type and add an optional bool > > > > > VectorType.finite (default false). Engines that want pgvector-like > > > > > rejection can require the flag = true. Engines that use NumPy / > > > PyTorch / > > > > > Spark arrays can keep representable IEEE values. For the same > > reason, I > > > > do > > > > > not favor adding yet another type called FINITE_VECTOR. > > > > > > > > > > Best, > > > > > Sebastian > > > > > > > > > > > > > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev < > > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > I agree with moving forward with the proposal and renaming VECTOR > > to > > > > > > FINITE_VECTOR or something > > > > > > similar if others feel that better communicates its intent. I see > > > value > > > > > in > > > > > > having a vector type that guarantees > > > > > > its values are mathematically-correct vectors for a few reasons: > > > > > > > > > > > > 1. As others have mentioned, this is similar to the STRING > logical > > > type > > > > > > mandating UTF-8. > > > > > > 2. It is not possible to rely on statistics for detecting NaN or > > > +-inf > > > > > > values as all statistics are optional. > > > > > > To my knowledge, there is no precedent for requiring that writers > > > emit > > > > > > stats for specific logical types. > > > > > > 3. Without statistics, NaN/inf values could flow through the > engine > > > and > > > > > > block them for specific functionality. > > > > > > This seems to be Lance's behavior. However, I don't think this is > > as > > > > easy > > > > > > as it sounds and Lance has run > > > > > > into several bugs due to this ([1] [2] for examples). Their > Python > > > [3] > > > > > and > > > > > > Rust [4] APIs default to erroring on > > > > > > vectors that contain NaN values. > > > > > > > > > > > > I agree that Parquet is a general-purpose format, but I don't > think > > > > this > > > > > > implies the lack of domain-specific types. > > > > > > Instead, this broadens the use cases where Parquet would be > > > considered > > > > a > > > > > > good fit. I also fully agree with > > > > > > continuing work on a general fixed-size list type if we believe > > there > > > > are > > > > > > non-vector use cases that would benefit > > > > > > from such guarantees at the storage layer. > > > > > > > > > > > > [1] https://github.com/lance-format/lance/pull/3648 > > > > > > [2] https://github.com/lance-format/lance/issues/8607 (see > point 2 > > > in > > > > > > "root > > > > > > cause") > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 > > > > > > [4] > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 > > > > > > > > > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > I also favor moving forward with the proposal as is. I see > strong > > > > value > > > > > > and > > > > > > > use cases for the numeric vector type, and I agree with the > > > > > restrictions > > > > > > > around NaN/Inf values and element types in general. Precedent > > > exists > > > > in > > > > > > > other types that restrict which physical type values are > > logically > > > > > valid > > > > > > > (e.g., the geospatial types needing to be WKB-encoded). A > > general > > > > > > > fixed-size list also has value, but I think it makes sense to > > > > separate > > > > > > the > > > > > > > two types. The semantics are relevant, and I can see how the > two > > > > types > > > > > > > might diverge further in the future regarding statistics or > > > > encodings. > > > > > > > > > > > > > > I would be okay with renaming the vector type, but it seems > like > > > the > > > > > > VECTOR > > > > > > > name is already used across other engines mentioned in this > > thread. > > > > > > > > > > > > > > Best, > > > > > > > Philipp > > > > > > > > > > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <[email protected]> > > wrote: > > > > > > > > > > > > > > > To separate orthogonal concerns, I feel we should treat the > > > general > > > > > > > > FixedSizeList as the underlying physical layout, and separate > > it > > > > from > > > > > > the > > > > > > > > discussion of a specialized VECTOR (or broader ML tensor) > > logical > > > > > type. > > > > > > > > > > > > > > > > Currently, the immediate need is to define a specialized > VECTOR > > > > type > > > > > so > > > > > > > > that different engines and formats have a common semantic > > > contract > > > > > for > > > > > > > > interoperability. > > > > > > > > > > > > > > > > If the main point of contention is just the NaN/Inf/null > > > > limitations, > > > > > > > maybe > > > > > > > > we can resolve this by adding a qualification to the name > (like > > > > > Andrew > > > > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid > > > > > confusion > > > > > > > with > > > > > > > > broader ML use cases. Alternatively, we could introduce a > > boolean > > > > > flag > > > > > > > > (e.g., optional bool VectorType.finite) as proposed earlier. > > > > > > > > > > > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I am a little confused about the points of disagreement > here. > > > > > > > > > > > > > > > > > > Is it that the name `VECTOR` is too general for the > specific > > > use > > > > > case > > > > > > > of > > > > > > > > > fixed length arrays with non-finite values used for vector > > > > > > embeddings? > > > > > > > > If > > > > > > > > > so, maybe we could add a qualification to the name > > > > (FINITE_VECTOR? > > > > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?) > > > > > > > > > > > > > > > > > > It seems like there is precedent for defining logical types > > > that > > > > > > > specify > > > > > > > > > certain physical types and restrict their contents to help > > > > readers > > > > > > > > > interpret meaning, for example string[1] or JSON; Defining > > one > > > > for > > > > > > > > vector > > > > > > > > > embeddings seems reasonable enough to me. > > > > > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > > > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Well, Parquet is not a single-purpose format. > > > > > > > > > > > > > > > > > > > > > > Vector stores can choose whatever policy fits their > > usage, > > > > but > > > > > > > we're > > > > > > > > > > > talking about a general-purpose file format that aims > to > > be > > > > > > broadly > > > > > > > > > > > applicable. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree that the format is intended to be general > purpose, > > > but > > > > > > types > > > > > > > > > within > > > > > > > > > > the format define specific representation and semantics > to > > > > > achieve > > > > > > > that > > > > > > > > > > goal. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and it's > quite > > > > > trivial > > > > > > to > > > > > > > > > > > produce vectors with non-finite values using e.g. NumPy > > or > > > > > Pandas > > > > > > > (or > > > > > > > > > > > any other piece of dataframe or scientific computing > > > > software). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The justification for restricting non-finite values is > > > > directly > > > > > > > > related > > > > > > > > > to > > > > > > > > > > the semantic meaning of a vector. Vectors have specific > > > > > > > > characteristics > > > > > > > > > > (dimension, magnitude, direction, etc). Non-finite > values > > > are > > > > > > > invalid > > > > > > > > > and > > > > > > > > > > break those characteristics. I'd liken it to storing > > > non-utf8 > > > > > > values > > > > > > > > in > > > > > > > > > a > > > > > > > > > > string and still calling it a string. > > > > > > > > > > > > > > > > > > > > That also differentiates a vector from a fixed length > list, > > > > which > > > > > > is > > > > > > > > more > > > > > > > > > > general and does not carry the same semantic meaning. > > > > > > > > > > > > > > > > > > > > I also don't think we need to bundle all similarly shaped > > use > > > > > cases > > > > > > > > into > > > > > > > > > > the same solution (e.g. flxed size list). Vector is a > well > > > > > > > established > > > > > > > > > > data type at this point and aligns with the general > purpose > > > > > nature > > > > > > of > > > > > > > > the > > > > > > > > > > format (many other formats and databases natively support > > > > this). > > > > > > > > > > > > > > > > > > > > -Dan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou < > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit : > > > > > > > > > > > > I think something I haven't seen yet is the clear > > > use-case > > > > > for > > > > > > > > > > non-finite > > > > > > > > > > > > values within a vector type. There has been a lot of > > > > > > theoretical > > > > > > > > > > > discussion > > > > > > > > > > > > here about use cases but what toolings currently > > produce > > > > > these > > > > > > > > > vectors > > > > > > > > > > > and > > > > > > > > > > > > how do they function in downstream engines? I'd like > to > > > > limit > > > > > > > > > > introducing > > > > > > > > > > > > types that have a wide range of values which are > simply > > > are > > > > > > > failure > > > > > > > > > > modes > > > > > > > > > > > > of different flavors > > > > > > > > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and it's > quite > > > > > trivial > > > > > > to > > > > > > > > > > > produce vectors with non-finite values using e.g. NumPy > > or > > > > > Pandas > > > > > > > (or > > > > > > > > > > > any other piece of dataframe or scientific computing > > > > software). > > > > > > > > > > > > > > > > > > > > > > It's not like the general concept of a vector is a IETF > > > > > standard > > > > > > or > > > > > > > > > > > anything, so analogies with JSON or UTF8 (where the > > values > > > > are > > > > > > > > expected > > > > > > > > > > > to be valid according to the respective specs) do not > > > apply. > > > > > > > > > > > > > > > > > > > > > > This discussion would IMHO have a better chance of > > > converging > > > > > if > > > > > > we > > > > > > > > > > > separated the two orthogonal concerns of: > > > > > > > > > > > > > > > > > > > > > > 1. representing fixed-length list/vector data in > Parquet > > > > (with > > > > > a > > > > > > > > > > > potential limitation to non-nested types) > > > > > > > > > > > > > > > > > > > > > > 2. representing the information that a given column (or > > > > column > > > > > > > chunk, > > > > > > > > > > > etc.) does not contain non-finite values > > > > > > > > > > > > > > > > > > > > > > ... both of which are useful on their own right, but do > > not > > > > > have > > > > > > to > > > > > > > > be > > > > > > > > > > > conflated and bundled together. > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
