> > Well, Parquet is not a single-purpose format. > > Vector stores can choose whatever policy fits their usage, but we're > talking about a general-purpose file format that aims to be broadly > applicable.
I agree that the format is intended to be general purpose, but types within the format define specific representation and semantics to achieve that goal. > "Vector" has a more general sense than ML and it's quite trivial to > produce vectors with non-finite values using e.g. NumPy or Pandas (or > any other piece of dataframe or scientific computing software). The justification for restricting non-finite values is directly related to the semantic meaning of a vector. Vectors have specific characteristics (dimension, magnitude, direction, etc). Non-finite values are invalid and break those characteristics. I'd liken it to storing non-utf8 values in a string and still calling it a string. That also differentiates a vector from a fixed length list, which is more general and does not carry the same semantic meaning. I also don't think we need to bundle all similarly shaped use cases into the same solution (e.g. flxed size list). Vector is a well established data type at this point and aligns with the general purpose nature of the format (many other formats and databases natively support this). -Dan On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <[email protected]> wrote: > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit : > > I think something I haven't seen yet is the clear use-case for non-finite > > values within a vector type. There has been a lot of theoretical > discussion > > here about use cases but what toolings currently produce these vectors > and > > how do they function in downstream engines? I'd like to limit introducing > > types that have a wide range of values which are simply are failure modes > > of different flavors > > "Vector" has a more general sense than ML and it's quite trivial to > produce vectors with non-finite values using e.g. NumPy or Pandas (or > any other piece of dataframe or scientific computing software). > > It's not like the general concept of a vector is a IETF standard or > anything, so analogies with JSON or UTF8 (where the values are expected > to be valid according to the respective specs) do not apply. > > This discussion would IMHO have a better chance of converging if we > separated the two orthogonal concerns of: > > 1. representing fixed-length list/vector data in Parquet (with a > potential limitation to non-nested types) > > 2. representing the information that a given column (or column chunk, > etc.) does not contain non-finite values > > ... both of which are useful on their own right, but do not have to be > conflated and bundled together. > > Regards > > Antoine. > > >
