I think something I haven't seen yet is the clear use-case for non-finite values within a vector type. There has been a lot of theoretical discussion here about use cases but what toolings currently produce these vectors and how do they function in downstream engines? I'd like to limit introducing types that have a wide range of values which are simply are failure modes of different flavors
I also don't find any issue with the attitude of folks in this discussion, it seems reasonable that people are trying to work together to get things done in the format for the practical use cases they know about. On Thu, Sep 24, 2026 at 8:49 AM Antoine Pitrou <[email protected]> wrote: > Le 24/09/2026 à 08:48, Will Edwards via dev a écrit : > > > > So is Parquet just growing a vector column type for the fixed-size > > embeddings used in nearest-neighbour search to keep up with mainstream > > databases, or is it jumping to support the broader set of ML tensor data > to > > accommodate all the random things an ML engineer might want to jam into a > > Parquet file today? I guess that is the question? > > Yes, that is a question and that was the heart of my position as well: > """ > Well, Parquet is not a single-purpose format. > > Vector stores can choose whatever policy fits their usage, but we're > talking about a general-purpose file format that aims to be broadly > applicable. > """ > > I didn't see any objection to this point. > > I also didn't see any explanation why allowing non-finite values would > actually hamper database usage. There are precedents where we don't > impose limitations to data types which may be further limited in > databases (for example timestamps). > > I also didn't see any consideration for the more logical solution of > adding metadata to tell whether a numeric column may contain non-finite > values (regardless of whether it's in a vector or not). > > And in the end I find that some people here are having a very cavalier > attitude towards discussion and reaching consensus. > > Regards > > Antoine. > > >
