I agree with moving forward with the proposal and renaming VECTOR to FINITE_VECTOR or something similar if others feel that better communicates its intent. I see value in having a vector type that guarantees its values are mathematically-correct vectors for a few reasons:
1. As others have mentioned, this is similar to the STRING logical type mandating UTF-8. 2. It is not possible to rely on statistics for detecting NaN or +-inf values as all statistics are optional. To my knowledge, there is no precedent for requiring that writers emit stats for specific logical types. 3. Without statistics, NaN/inf values could flow through the engine and block them for specific functionality. This seems to be Lance's behavior. However, I don't think this is as easy as it sounds and Lance has run into several bugs due to this ([1] [2] for examples). Their Python [3] and Rust [4] APIs default to erroring on vectors that contain NaN values. I agree that Parquet is a general-purpose format, but I don't think this implies the lack of domain-specific types. Instead, this broadens the use cases where Parquet would be considered a good fit. I also fully agree with continuing work on a general fixed-size list type if we believe there are non-vector use cases that would benefit from such guarantees at the storage layer. [1] https://github.com/lance-format/lance/pull/3648 [2] https://github.com/lance-format/lance/issues/8607 (see point 2 in "root cause") [3] https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911 [4] https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110 On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev < [email protected]> wrote: > I also favor moving forward with the proposal as is. I see strong value and > use cases for the numeric vector type, and I agree with the restrictions > around NaN/Inf values and element types in general. Precedent exists in > other types that restrict which physical type values are logically valid > (e.g., the geospatial types needing to be WKB-encoded). A general > fixed-size list also has value, but I think it makes sense to separate the > two types. The semantics are relevant, and I can see how the two types > might diverge further in the future regarding statistics or encodings. > > I would be okay with renaming the vector type, but it seems like the VECTOR > name is already used across other engines mentioned in this thread. > > Best, > Philipp > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <[email protected]> wrote: > > > To separate orthogonal concerns, I feel we should treat the general > > FixedSizeList as the underlying physical layout, and separate it from the > > discussion of a specialized VECTOR (or broader ML tensor) logical type. > > > > Currently, the immediate need is to define a specialized VECTOR type so > > that different engines and formats have a common semantic contract for > > interoperability. > > > > If the main point of contention is just the NaN/Inf/null limitations, > maybe > > we can resolve this by adding a qualification to the name (like Andrew > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to avoid confusion > with > > broader ML use cases. Alternatively, we could introduce a boolean flag > > (e.g., optional bool VectorType.finite) as proposed earlier. > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <[email protected]> > > wrote: > > > > > I am a little confused about the points of disagreement here. > > > > > > Is it that the name `VECTOR` is too general for the specific use case > of > > > fixed length arrays with non-finite values used for vector embeddings? > > If > > > so, maybe we could add a qualification to the name (FINITE_VECTOR? > > > FEATURE_VECTOR? VECTOR_EMBEDDING?) > > > > > > It seems like there is precedent for defining logical types that > specify > > > certain physical types and restrict their contents to help readers > > > interpret meaning, for example string[1] or JSON; Defining one for > > vector > > > embeddings seems reasonable enough to me. > > > > > > Andrew > > > > > > > > > [1]: > > > > > > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <[email protected]> > wrote: > > > > > > > > > > > > > Well, Parquet is not a single-purpose format. > > > > > > > > > > Vector stores can choose whatever policy fits their usage, but > we're > > > > > talking about a general-purpose file format that aims to be broadly > > > > > applicable. > > > > > > > > > > > > I agree that the format is intended to be general purpose, but types > > > within > > > > the format define specific representation and semantics to achieve > that > > > > goal. > > > > > > > > > > > > > "Vector" has a more general sense than ML and it's quite trivial to > > > > > produce vectors with non-finite values using e.g. NumPy or Pandas > (or > > > > > any other piece of dataframe or scientific computing software). > > > > > > > > > > > > The justification for restricting non-finite values is directly > > related > > > to > > > > the semantic meaning of a vector. Vectors have specific > > characteristics > > > > (dimension, magnitude, direction, etc). Non-finite values are > invalid > > > and > > > > break those characteristics. I'd liken it to storing non-utf8 values > > in > > > a > > > > string and still calling it a string. > > > > > > > > That also differentiates a vector from a fixed length list, which is > > more > > > > general and does not carry the same semantic meaning. > > > > > > > > I also don't think we need to bundle all similarly shaped use cases > > into > > > > the same solution (e.g. flxed size list). Vector is a well > established > > > > data type at this point and aligns with the general purpose nature of > > the > > > > format (many other formats and databases natively support this). > > > > > > > > -Dan > > > > > > > > > > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <[email protected]> > > > > wrote: > > > > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit : > > > > > > I think something I haven't seen yet is the clear use-case for > > > > non-finite > > > > > > values within a vector type. There has been a lot of theoretical > > > > > discussion > > > > > > here about use cases but what toolings currently produce these > > > vectors > > > > > and > > > > > > how do they function in downstream engines? I'd like to limit > > > > introducing > > > > > > types that have a wide range of values which are simply are > failure > > > > modes > > > > > > of different flavors > > > > > > > > > > "Vector" has a more general sense than ML and it's quite trivial to > > > > > produce vectors with non-finite values using e.g. NumPy or Pandas > (or > > > > > any other piece of dataframe or scientific computing software). > > > > > > > > > > It's not like the general concept of a vector is a IETF standard or > > > > > anything, so analogies with JSON or UTF8 (where the values are > > expected > > > > > to be valid according to the respective specs) do not apply. > > > > > > > > > > This discussion would IMHO have a better chance of converging if we > > > > > separated the two orthogonal concerns of: > > > > > > > > > > 1. representing fixed-length list/vector data in Parquet (with a > > > > > potential limitation to non-nested types) > > > > > > > > > > 2. representing the information that a given column (or column > chunk, > > > > > etc.) does not contain non-finite values > > > > > > > > > > ... both of which are useful on their own right, but do not have to > > be > > > > > conflated and bundled together. > > > > > > > > > > Regards > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > >
