+1 Sebastian, a vector type suggests usage beyond normalized embeddings and engines really aren't going to be able to elide all the checks anyway.
Antoine, you suggested we explore metadata solutions too :) Today, a database supporting CHAR(N) or VARCHAR(n) can't express this in Parquet. Engines can write opaque blobs they understand in the metadata (e.g. Arrow does) so the extra type info round-trips, but this isn't standardized, portable, nor interoperable. So perhaps standardizing constraint metadata or new Thrift fields (my pref) can generically support VECTOR(N) whilst also being more broadly useful for other use cases and data types? Imagine metadata or Thrift fields to tag the MIN_LENGTH and MAX_LENGTH for Parquet's variable-length physcial types. These would count logical units (e.g. UTF8 code points for STRING) rather than bytes. Then one engine's VECTOR(N) can be written to Parquet as LIST(N) with MIN_LENGTH = MAX_LENGTH = N and then readers - even different engines - can pick up these constraints if they support doing so. We don't actually need a logical type to express a VECTOR then. This also allows standard SQL's CHAR(N) and VARCHAR(N) to be expressed on UTF8 columns etc. This also seems generally useful in the future as we approach expressing other tensor data shapes and types. This won't change the mechanics: the storage type doesn't enforce these constraints and engines must check them. It's previously established that performance is just a software problem and there is no fundamental performance advantage to the fixed size types in Dremel-like formats like Parquet. Best, Will On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <[email protected]> wrote: > After listening in on the last meeting Wednesday, I have been thinking > about this quite a bit, and thought I would share my 2 cents. > > I suggest not requiring VECTOR elements to be finite, as proposed in the PR > <https://github.com/apache/parquet-format/pull/624>. > > While Divjot and others note the specialization for UTF-8, Parquet already > defines DOUBLE and FLOAT as IEEE encodings > <https://github.com/apache/parquet-format/blob/master/README.md#types>. > > Making NaN or +/-Inf illegal inside VECTOR would be a new restriction on > those same underlying physical types. Scalar FLOAT/DOUBLE columns have > never had such a rule. > > It would also not match how the systems that actually produce these vectors > treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB, NumPy, > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS. > > pgvector's "all elements must be finite" rule is an insert policy. While I > think Parquet as a general-purpose format should allow them out of the box. > It is a user system decision to disallow writing these values or to throw > errors when encountering them. > > A type-level ban also does not keep Inf/NaN out of vector math. Finite > coordinates still overflow. For instance: magnitude = ||[FLOAT_MAX, > FLOAT_MAX]|| where every coordinate is finite overflows to +Inf. PyTorch > documents the same overflow: tensor([1e20, 1e20]).norm() is inf in fp32. > > A generated MAGNITUDE(vec) column would then be Inf while the VECTOR itself > was required to be finite. The ban only forbids storing specials in the > coordinates, and does not prevent even simple downstream computations from > producing them. > > I suggest keeping VECTOR as the type and add an optional bool > VectorType.finite (default false). Engines that want pgvector-like > rejection can require the flag = true. Engines that use NumPy / PyTorch / > Spark arrays can keep representable IEEE values. For the same reason, I do > not favor adding yet another type called FINITE_VECTOR. > > Best, > Sebastian > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev < > [email protected]> > wrote: > > > I agree with moving forward with the proposal and renaming VECTOR to > > FINITE_VECTOR or something > > similar if others feel that better communicates its intent. I see value > in > > having a vector type that guarantees > > its values are mathematically-correct vectors for a few reasons: > > > > 1. As others have mentioned, this is similar to the STRING logical type > > mandating UTF-8. > > 2. It is not possible to rely on statistics for detecting NaN or +-inf > > values as all statistics are optional. > > To my knowledge, there is no precedent for requiring that writers emit > > stats for specific logical types. > > 3. Without statistics, NaN/inf values could flow through the engine and > > block them for specific functionality. > > This seems to be Lance's behavior. However, I don't think this is as easy > > as it sounds and Lance has run > > into several bugs due to this ([1] [2] for examples). Their Python [3] > and > > Rust [4] APIs default to erroring on > > vectors that contain NaN values. > > > > I agree that Parquet is a general-purpose format, but I don't think this > > implies the lack of domain-specific types. > > Instead, this broadens the use cases where Parquet would be considered a > > good fit. I also fully agree with > > continuing work on a general fixed-size list type if we believe there are > > non-vector use cases that would benefit > > from such guarantees at the storage layer. > > > > [1] https://github.com/lance-format/lance/pull/3648 > > [2] https://github.com/lance-format/lance/issues/8607 (see point 2 in > > "root > > cause") > > [3] > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 > > [4] > > > > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev < > > [email protected]> wrote: > > > > > I also favor moving forward with the proposal as is. I see strong value > > and > > > use cases for the numeric vector type, and I agree with the > restrictions > > > around NaN/Inf values and element types in general. Precedent exists in > > > other types that restrict which physical type values are logically > valid > > > (e.g., the geospatial types needing to be WKB-encoded). A general > > > fixed-size list also has value, but I think it makes sense to separate > > the > > > two types. The semantics are relevant, and I can see how the two types > > > might diverge further in the future regarding statistics or encodings. > > > > > > I would be okay with renaming the vector type, but it seems like the > > VECTOR > > > name is already used across other engines mentioned in this thread. > > > > > > Best, > > > Philipp > > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <[email protected]> wrote: > > > > > > > To separate orthogonal concerns, I feel we should treat the general > > > > FixedSizeList as the underlying physical layout, and separate it from > > the > > > > discussion of a specialized VECTOR (or broader ML tensor) logical > type. > > > > > > > > Currently, the immediate need is to define a specialized VECTOR type > so > > > > that different engines and formats have a common semantic contract > for > > > > interoperability. > > > > > > > > If the main point of contention is just the NaN/Inf/null limitations, > > > maybe > > > > we can resolve this by adding a qualification to the name (like > Andrew > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid > confusion > > > with > > > > broader ML use cases. Alternatively, we could introduce a boolean > flag > > > > (e.g., optional bool VectorType.finite) as proposed earlier. > > > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <[email protected]> > > > > wrote: > > > > > > > > > I am a little confused about the points of disagreement here. > > > > > > > > > > Is it that the name `VECTOR` is too general for the specific use > case > > > of > > > > > fixed length arrays with non-finite values used for vector > > embeddings? > > > > If > > > > > so, maybe we could add a qualification to the name (FINITE_VECTOR? > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?) > > > > > > > > > > It seems like there is precedent for defining logical types that > > > specify > > > > > certain physical types and restrict their contents to help readers > > > > > interpret meaning, for example string[1] or JSON; Defining one for > > > > vector > > > > > embeddings seems reasonable enough to me. > > > > > > > > > > Andrew > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > > > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <[email protected]> > > > wrote: > > > > > > > > > > > > > > > > > > > Well, Parquet is not a single-purpose format. > > > > > > > > > > > > > > Vector stores can choose whatever policy fits their usage, but > > > we're > > > > > > > talking about a general-purpose file format that aims to be > > broadly > > > > > > > applicable. > > > > > > > > > > > > > > > > > > I agree that the format is intended to be general purpose, but > > types > > > > > within > > > > > > the format define specific representation and semantics to > achieve > > > that > > > > > > goal. > > > > > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and it's quite > trivial > > to > > > > > > > produce vectors with non-finite values using e.g. NumPy or > Pandas > > > (or > > > > > > > any other piece of dataframe or scientific computing software). > > > > > > > > > > > > > > > > > > The justification for restricting non-finite values is directly > > > > related > > > > > to > > > > > > the semantic meaning of a vector. Vectors have specific > > > > characteristics > > > > > > (dimension, magnitude, direction, etc). Non-finite values are > > > invalid > > > > > and > > > > > > break those characteristics. I'd liken it to storing non-utf8 > > values > > > > in > > > > > a > > > > > > string and still calling it a string. > > > > > > > > > > > > That also differentiates a vector from a fixed length list, which > > is > > > > more > > > > > > general and does not carry the same semantic meaning. > > > > > > > > > > > > I also don't think we need to bundle all similarly shaped use > cases > > > > into > > > > > > the same solution (e.g. flxed size list). Vector is a well > > > established > > > > > > data type at this point and aligns with the general purpose > nature > > of > > > > the > > > > > > format (many other formats and databases natively support this). > > > > > > > > > > > > -Dan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit : > > > > > > > > I think something I haven't seen yet is the clear use-case > for > > > > > > non-finite > > > > > > > > values within a vector type. There has been a lot of > > theoretical > > > > > > > discussion > > > > > > > > here about use cases but what toolings currently produce > these > > > > > vectors > > > > > > > and > > > > > > > > how do they function in downstream engines? I'd like to limit > > > > > > introducing > > > > > > > > types that have a wide range of values which are simply are > > > failure > > > > > > modes > > > > > > > > of different flavors > > > > > > > > > > > > > > "Vector" has a more general sense than ML and it's quite > trivial > > to > > > > > > > produce vectors with non-finite values using e.g. NumPy or > Pandas > > > (or > > > > > > > any other piece of dataframe or scientific computing software). > > > > > > > > > > > > > > It's not like the general concept of a vector is a IETF > standard > > or > > > > > > > anything, so analogies with JSON or UTF8 (where the values are > > > > expected > > > > > > > to be valid according to the respective specs) do not apply. > > > > > > > > > > > > > > This discussion would IMHO have a better chance of converging > if > > we > > > > > > > separated the two orthogonal concerns of: > > > > > > > > > > > > > > 1. representing fixed-length list/vector data in Parquet (with > a > > > > > > > potential limitation to non-nested types) > > > > > > > > > > > > > > 2. representing the information that a given column (or column > > > chunk, > > > > > > > etc.) does not contain non-finite values > > > > > > > > > > > > > > ... both of which are useful on their own right, but do not > have > > to > > > > be > > > > > > > conflated and bundled together. > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
