After listening in on the last meeting Wednesday, I have been thinking about this quite a bit, and thought I would share my 2 cents.
I suggest not requiring VECTOR elements to be finite, as proposed in the PR <https://github.com/apache/parquet-format/pull/624>. While Divjot and others note the specialization for UTF-8, Parquet already defines DOUBLE and FLOAT as IEEE encodings <https://github.com/apache/parquet-format/blob/master/README.md#types>. Making NaN or +/-Inf illegal inside VECTOR would be a new restriction on those same underlying physical types. Scalar FLOAT/DOUBLE columns have never had such a rule. It would also not match how the systems that actually produce these vectors treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB, NumPy, PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS. pgvector's "all elements must be finite" rule is an insert policy. While I think Parquet as a general-purpose format should allow them out of the box. It is a user system decision to disallow writing these values or to throw errors when encountering them. A type-level ban also does not keep Inf/NaN out of vector math. Finite coordinates still overflow. For instance: magnitude = ||[FLOAT_MAX, FLOAT_MAX]|| where every coordinate is finite overflows to +Inf. PyTorch documents the same overflow: tensor([1e20, 1e20]).norm() is inf in fp32. A generated MAGNITUDE(vec) column would then be Inf while the VECTOR itself was required to be finite. The ban only forbids storing specials in the coordinates, and does not prevent even simple downstream computations from producing them. I suggest keeping VECTOR as the type and add an optional bool VectorType.finite (default false). Engines that want pgvector-like rejection can require the flag = true. Engines that use NumPy / PyTorch / Spark arrays can keep representable IEEE values. For the same reason, I do not favor adding yet another type called FINITE_VECTOR. Best, Sebastian On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <[email protected]> wrote: > I agree with moving forward with the proposal and renaming VECTOR to > FINITE_VECTOR or something > similar if others feel that better communicates its intent. I see value in > having a vector type that guarantees > its values are mathematically-correct vectors for a few reasons: > > 1. As others have mentioned, this is similar to the STRING logical type > mandating UTF-8. > 2. It is not possible to rely on statistics for detecting NaN or +-inf > values as all statistics are optional. > To my knowledge, there is no precedent for requiring that writers emit > stats for specific logical types. > 3. Without statistics, NaN/inf values could flow through the engine and > block them for specific functionality. > This seems to be Lance's behavior. However, I don't think this is as easy > as it sounds and Lance has run > into several bugs due to this ([1] [2] for examples). Their Python [3] and > Rust [4] APIs default to erroring on > vectors that contain NaN values. > > I agree that Parquet is a general-purpose format, but I don't think this > implies the lack of domain-specific types. > Instead, this broadens the use cases where Parquet would be considered a > good fit. I also fully agree with > continuing work on a general fixed-size list type if we believe there are > non-vector use cases that would benefit > from such guarantees at the storage layer. > > [1] https://github.com/lance-format/lance/pull/3648 > [2] https://github.com/lance-format/lance/issues/8607 (see point 2 in > "root > cause") > [3] > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 > [4] > > https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev < > [email protected]> wrote: > > > I also favor moving forward with the proposal as is. I see strong value > and > > use cases for the numeric vector type, and I agree with the restrictions > > around NaN/Inf values and element types in general. Precedent exists in > > other types that restrict which physical type values are logically valid > > (e.g., the geospatial types needing to be WKB-encoded). A general > > fixed-size list also has value, but I think it makes sense to separate > the > > two types. The semantics are relevant, and I can see how the two types > > might diverge further in the future regarding statistics or encodings. > > > > I would be okay with renaming the vector type, but it seems like the > VECTOR > > name is already used across other engines mentioned in this thread. > > > > Best, > > Philipp > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <[email protected]> wrote: > > > > > To separate orthogonal concerns, I feel we should treat the general > > > FixedSizeList as the underlying physical layout, and separate it from > the > > > discussion of a specialized VECTOR (or broader ML tensor) logical type. > > > > > > Currently, the immediate need is to define a specialized VECTOR type so > > > that different engines and formats have a common semantic contract for > > > interoperability. > > > > > > If the main point of contention is just the NaN/Inf/null limitations, > > maybe > > > we can resolve this by adding a qualification to the name (like Andrew > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid confusion > > with > > > broader ML use cases. Alternatively, we could introduce a boolean flag > > > (e.g., optional bool VectorType.finite) as proposed earlier. > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <[email protected]> > > > wrote: > > > > > > > I am a little confused about the points of disagreement here. > > > > > > > > Is it that the name `VECTOR` is too general for the specific use case > > of > > > > fixed length arrays with non-finite values used for vector > embeddings? > > > If > > > > so, maybe we could add a qualification to the name (FINITE_VECTOR? > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?) > > > > > > > > It seems like there is precedent for defining logical types that > > specify > > > > certain physical types and restrict their contents to help readers > > > > interpret meaning, for example string[1] or JSON; Defining one for > > > vector > > > > embeddings seems reasonable enough to me. > > > > > > > > Andrew > > > > > > > > > > > > [1]: > > > > > > > > > > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <[email protected]> > > wrote: > > > > > > > > > > > > > > > > Well, Parquet is not a single-purpose format. > > > > > > > > > > > > Vector stores can choose whatever policy fits their usage, but > > we're > > > > > > talking about a general-purpose file format that aims to be > broadly > > > > > > applicable. > > > > > > > > > > > > > > > I agree that the format is intended to be general purpose, but > types > > > > within > > > > > the format define specific representation and semantics to achieve > > that > > > > > goal. > > > > > > > > > > > > > > > > "Vector" has a more general sense than ML and it's quite trivial > to > > > > > > produce vectors with non-finite values using e.g. NumPy or Pandas > > (or > > > > > > any other piece of dataframe or scientific computing software). > > > > > > > > > > > > > > > The justification for restricting non-finite values is directly > > > related > > > > to > > > > > the semantic meaning of a vector. Vectors have specific > > > characteristics > > > > > (dimension, magnitude, direction, etc). Non-finite values are > > invalid > > > > and > > > > > break those characteristics. I'd liken it to storing non-utf8 > values > > > in > > > > a > > > > > string and still calling it a string. > > > > > > > > > > That also differentiates a vector from a fixed length list, which > is > > > more > > > > > general and does not carry the same semantic meaning. > > > > > > > > > > I also don't think we need to bundle all similarly shaped use cases > > > into > > > > > the same solution (e.g. flxed size list). Vector is a well > > established > > > > > data type at this point and aligns with the general purpose nature > of > > > the > > > > > format (many other formats and databases natively support this). > > > > > > > > > > -Dan > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit : > > > > > > > I think something I haven't seen yet is the clear use-case for > > > > > non-finite > > > > > > > values within a vector type. There has been a lot of > theoretical > > > > > > discussion > > > > > > > here about use cases but what toolings currently produce these > > > > vectors > > > > > > and > > > > > > > how do they function in downstream engines? I'd like to limit > > > > > introducing > > > > > > > types that have a wide range of values which are simply are > > failure > > > > > modes > > > > > > > of different flavors > > > > > > > > > > > > "Vector" has a more general sense than ML and it's quite trivial > to > > > > > > produce vectors with non-finite values using e.g. NumPy or Pandas > > (or > > > > > > any other piece of dataframe or scientific computing software). > > > > > > > > > > > > It's not like the general concept of a vector is a IETF standard > or > > > > > > anything, so analogies with JSON or UTF8 (where the values are > > > expected > > > > > > to be valid according to the respective specs) do not apply. > > > > > > > > > > > > This discussion would IMHO have a better chance of converging if > we > > > > > > separated the two orthogonal concerns of: > > > > > > > > > > > > 1. representing fixed-length list/vector data in Parquet (with a > > > > > > potential limitation to non-nested types) > > > > > > > > > > > > 2. representing the information that a given column (or column > > chunk, > > > > > > etc.) does not contain non-finite values > > > > > > > > > > > > ... both of which are useful on their own right, but do not have > to > > > be > > > > > > conflated and bundled together. > > > > > > > > > > > > Regards > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
