Going through this thread again, it looks like there are a few things that people generally agree on:
- In the short term, we want to introduce a logical type or types; a better physical representation can be done later. We can focus on logical types. - There is precedent in logical types for restricting values, like UTF8 strings to contain only UTF-8 codepoints. I think we also agree that we could restrict vector elements to non-nested or primitive types. - There are many examples of databases that restrict their vector types to non-null, finite elements. - Some people want to have write-time guarantees that vectors have only non-null, finite elements - Some people want to extend this work to include tensors, shorter floating point representations, variable lengths, NaN or Infinity elements, etc. It sounds like we are nearly there, but have two halves of the community that care about different things. That just means we should have multiple types. It's entirely reasonable to want a type that allows NaN values, and it's also reasonable for those of us building databases, table standards, and vector indexes to care about guarantees that elements are finite. There's more than enough examples to show that this is a legitimate use case. Since I don't see the concerns of the different use cases changing, it looks like we need to find a compromise to move forward. I propose that we introduce a FINITE_VECTOR type and separately a FIXED_LENGTH_LIST or another VECTOR type that allows infinite and NaN values. Can we agree on a compromise? Is there a strong reason to try to fit these use cases into a single type? What if that means not making progress? Ryan On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev < [email protected]> wrote: > Thinking about this more, I am also fine with pulling out the “NaN/Inf” > requirement. Either into a type boolean (my preference), or putting no > restrictions on the Parquet level, allowing all IEEE 754 float numbers. If > some top-level application wants to put additional constraints on the data > type, like Iceberg not allowing NaN/infinity, it can still do so one layer > above. It seems like this was the biggest point of discussion remaining. > All these options work for me, and we should move forward with this > proposal. > > Concerning additional element types like bf16, they can be introduced as > Parquet types (at least in the current LIST-based spec proposal). > > Concerning Will’s suggestion around representing VECTOR as a variable > length list with a MIN/MAX_LENGTH metadata restriction: I think that a > dedicated logical VECTOR type still provides enough benefits by describing > the semantics of the data, and allowing to capture future changes around > statistics. In that case, the length is better captured as an exact length > for this type specifically. I’m also not sure about other use cases for > MIN_LENGTH, so introducing a new type constraint system just for the > MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough value in my > view. > > Best, > Philipp > > > On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev < > [email protected]> > wrote: > > > +1 Sebastian, a vector type suggests usage beyond normalized embeddings > and > > engines really aren't going to be able to elide all the checks anyway. > > > > Antoine, you suggested we explore metadata solutions too :) > > > > Today, a database supporting CHAR(N) or VARCHAR(n) can't express this in > > Parquet. Engines can write opaque blobs they understand in the metadata > > (e.g. Arrow does) so the extra type info round-trips, but this isn't > > standardized, portable, nor interoperable. > > > > So perhaps standardizing constraint metadata or new Thrift fields (my > pref) > > can generically support VECTOR(N) whilst also being more broadly useful > for > > other use cases and data types? > > > > Imagine metadata or Thrift fields to tag the MIN_LENGTH and MAX_LENGTH > for > > Parquet's variable-length physcial types. These would count logical > units > > (e.g. UTF8 code points for STRING) rather than bytes. > > > > Then one engine's VECTOR(N) can be written to Parquet as LIST(N) with > > MIN_LENGTH = MAX_LENGTH = N and then readers - even different engines - > can > > pick up these constraints if they support doing so. We don't actually > need > > a logical type to express a VECTOR then. > > > > This also allows standard SQL's CHAR(N) and VARCHAR(N) to be expressed on > > UTF8 columns etc. > > > > This also seems generally useful in the future as we approach expressing > > other tensor data shapes and types. > > > > This won't change the mechanics: the storage type doesn't enforce these > > constraints and engines must check them. It's previously established that > > performance is just a software problem and there is no fundamental > > performance advantage to the fixed size types in Dremel-like formats like > > Parquet. > > > > Best, > > Will > > > > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard < > [email protected] > > > > > wrote: > > > > > After listening in on the last meeting Wednesday, I have been thinking > > > about this quite a bit, and thought I would share my 2 cents. > > > > > > I suggest not requiring VECTOR elements to be finite, as proposed in > the > > PR > > > <https://github.com/apache/parquet-format/pull/624>. > > > > > > While Divjot and others note the specialization for UTF-8, Parquet > > already > > > defines DOUBLE and FLOAT as IEEE encodings > > > <https://github.com/apache/parquet-format/blob/master/README.md#types > >. > > > > > > Making NaN or +/-Inf illegal inside VECTOR would be a new restriction > on > > > those same underlying physical types. Scalar FLOAT/DOUBLE columns have > > > never had such a rule. > > > > > > It would also not match how the systems that actually produce these > > vectors > > > treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB, NumPy, > > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS. > > > > > > pgvector's "all elements must be finite" rule is an insert policy. > While > > I > > > think Parquet as a general-purpose format should allow them out of the > > box. > > > It is a user system decision to disallow writing these values or to > throw > > > errors when encountering them. > > > > > > A type-level ban also does not keep Inf/NaN out of vector math. Finite > > > coordinates still overflow. For instance: magnitude = ||[FLOAT_MAX, > > > FLOAT_MAX]|| where every coordinate is finite overflows to +Inf. > PyTorch > > > documents the same overflow: tensor([1e20, 1e20]).norm() is inf in > fp32. > > > > > > A generated MAGNITUDE(vec) column would then be Inf while the VECTOR > > itself > > > was required to be finite. The ban only forbids storing specials in the > > > coordinates, and does not prevent even simple downstream computations > > from > > > producing them. > > > > > > I suggest keeping VECTOR as the type and add an optional bool > > > VectorType.finite (default false). Engines that want pgvector-like > > > rejection can require the flag = true. Engines that use NumPy / > PyTorch / > > > Spark arrays can keep representable IEEE values. For the same reason, I > > do > > > not favor adding yet another type called FINITE_VECTOR. > > > > > > Best, > > > Sebastian > > > > > > > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev < > > > [email protected]> > > > wrote: > > > > > > > I agree with moving forward with the proposal and renaming VECTOR to > > > > FINITE_VECTOR or something > > > > similar if others feel that better communicates its intent. I see > value > > > in > > > > having a vector type that guarantees > > > > its values are mathematically-correct vectors for a few reasons: > > > > > > > > 1. As others have mentioned, this is similar to the STRING logical > type > > > > mandating UTF-8. > > > > 2. It is not possible to rely on statistics for detecting NaN or > +-inf > > > > values as all statistics are optional. > > > > To my knowledge, there is no precedent for requiring that writers > emit > > > > stats for specific logical types. > > > > 3. Without statistics, NaN/inf values could flow through the engine > and > > > > block them for specific functionality. > > > > This seems to be Lance's behavior. However, I don't think this is as > > easy > > > > as it sounds and Lance has run > > > > into several bugs due to this ([1] [2] for examples). Their Python > [3] > > > and > > > > Rust [4] APIs default to erroring on > > > > vectors that contain NaN values. > > > > > > > > I agree that Parquet is a general-purpose format, but I don't think > > this > > > > implies the lack of domain-specific types. > > > > Instead, this broadens the use cases where Parquet would be > considered > > a > > > > good fit. I also fully agree with > > > > continuing work on a general fixed-size list type if we believe there > > are > > > > non-vector use cases that would benefit > > > > from such guarantees at the storage layer. > > > > > > > > [1] https://github.com/lance-format/lance/pull/3648 > > > > [2] https://github.com/lance-format/lance/issues/8607 (see point 2 > in > > > > "root > > > > cause") > > > > [3] > > > > > > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 > > > > [4] > > > > > > > > > > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 > > > > > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev < > > > > [email protected]> wrote: > > > > > > > > > I also favor moving forward with the proposal as is. I see strong > > value > > > > and > > > > > use cases for the numeric vector type, and I agree with the > > > restrictions > > > > > around NaN/Inf values and element types in general. Precedent > exists > > in > > > > > other types that restrict which physical type values are logically > > > valid > > > > > (e.g., the geospatial types needing to be WKB-encoded). A general > > > > > fixed-size list also has value, but I think it makes sense to > > separate > > > > the > > > > > two types. The semantics are relevant, and I can see how the two > > types > > > > > might diverge further in the future regarding statistics or > > encodings. > > > > > > > > > > I would be okay with renaming the vector type, but it seems like > the > > > > VECTOR > > > > > name is already used across other engines mentioned in this thread. > > > > > > > > > > Best, > > > > > Philipp > > > > > > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <[email protected]> wrote: > > > > > > > > > > > To separate orthogonal concerns, I feel we should treat the > general > > > > > > FixedSizeList as the underlying physical layout, and separate it > > from > > > > the > > > > > > discussion of a specialized VECTOR (or broader ML tensor) logical > > > type. > > > > > > > > > > > > Currently, the immediate need is to define a specialized VECTOR > > type > > > so > > > > > > that different engines and formats have a common semantic > contract > > > for > > > > > > interoperability. > > > > > > > > > > > > If the main point of contention is just the NaN/Inf/null > > limitations, > > > > > maybe > > > > > > we can resolve this by adding a qualification to the name (like > > > Andrew > > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid > > > confusion > > > > > with > > > > > > broader ML use cases. Alternatively, we could introduce a boolean > > > flag > > > > > > (e.g., optional bool VectorType.finite) as proposed earlier. > > > > > > > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > I am a little confused about the points of disagreement here. > > > > > > > > > > > > > > Is it that the name `VECTOR` is too general for the specific > use > > > case > > > > > of > > > > > > > fixed length arrays with non-finite values used for vector > > > > embeddings? > > > > > > If > > > > > > > so, maybe we could add a qualification to the name > > (FINITE_VECTOR? > > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?) > > > > > > > > > > > > > > It seems like there is precedent for defining logical types > that > > > > > specify > > > > > > > certain physical types and restrict their contents to help > > readers > > > > > > > interpret meaning, for example string[1] or JSON; Defining one > > for > > > > > > vector > > > > > > > embeddings seems reasonable enough to me. > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Well, Parquet is not a single-purpose format. > > > > > > > > > > > > > > > > > > Vector stores can choose whatever policy fits their usage, > > but > > > > > we're > > > > > > > > > talking about a general-purpose file format that aims to be > > > > broadly > > > > > > > > > applicable. > > > > > > > > > > > > > > > > > > > > > > > > I agree that the format is intended to be general purpose, > but > > > > types > > > > > > > within > > > > > > > > the format define specific representation and semantics to > > > achieve > > > > > that > > > > > > > > goal. > > > > > > > > > > > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and it's quite > > > trivial > > > > to > > > > > > > > > produce vectors with non-finite values using e.g. NumPy or > > > Pandas > > > > > (or > > > > > > > > > any other piece of dataframe or scientific computing > > software). > > > > > > > > > > > > > > > > > > > > > > > > The justification for restricting non-finite values is > > directly > > > > > > related > > > > > > > to > > > > > > > > the semantic meaning of a vector. Vectors have specific > > > > > > characteristics > > > > > > > > (dimension, magnitude, direction, etc). Non-finite values > are > > > > > invalid > > > > > > > and > > > > > > > > break those characteristics. I'd liken it to storing > non-utf8 > > > > values > > > > > > in > > > > > > > a > > > > > > > > string and still calling it a string. > > > > > > > > > > > > > > > > That also differentiates a vector from a fixed length list, > > which > > > > is > > > > > > more > > > > > > > > general and does not carry the same semantic meaning. > > > > > > > > > > > > > > > > I also don't think we need to bundle all similarly shaped use > > > cases > > > > > > into > > > > > > > > the same solution (e.g. flxed size list). Vector is a well > > > > > established > > > > > > > > data type at this point and aligns with the general purpose > > > nature > > > > of > > > > > > the > > > > > > > > format (many other formats and databases natively support > > this). > > > > > > > > > > > > > > > > -Dan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit : > > > > > > > > > > I think something I haven't seen yet is the clear > use-case > > > for > > > > > > > > non-finite > > > > > > > > > > values within a vector type. There has been a lot of > > > > theoretical > > > > > > > > > discussion > > > > > > > > > > here about use cases but what toolings currently produce > > > these > > > > > > > vectors > > > > > > > > > and > > > > > > > > > > how do they function in downstream engines? I'd like to > > limit > > > > > > > > introducing > > > > > > > > > > types that have a wide range of values which are simply > are > > > > > failure > > > > > > > > modes > > > > > > > > > > of different flavors > > > > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and it's quite > > > trivial > > > > to > > > > > > > > > produce vectors with non-finite values using e.g. NumPy or > > > Pandas > > > > > (or > > > > > > > > > any other piece of dataframe or scientific computing > > software). > > > > > > > > > > > > > > > > > > It's not like the general concept of a vector is a IETF > > > standard > > > > or > > > > > > > > > anything, so analogies with JSON or UTF8 (where the values > > are > > > > > > expected > > > > > > > > > to be valid according to the respective specs) do not > apply. > > > > > > > > > > > > > > > > > > This discussion would IMHO have a better chance of > converging > > > if > > > > we > > > > > > > > > separated the two orthogonal concerns of: > > > > > > > > > > > > > > > > > > 1. representing fixed-length list/vector data in Parquet > > (with > > > a > > > > > > > > > potential limitation to non-nested types) > > > > > > > > > > > > > > > > > > 2. representing the information that a given column (or > > column > > > > > chunk, > > > > > > > > > etc.) does not contain non-finite values > > > > > > > > > > > > > > > > > > ... both of which are useful on their own right, but do not > > > have > > > > to > > > > > > be > > > > > > > > > conflated and bundled together. > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
