Le 24/09/2026 à 08:48, Will Edwards via dev a écrit :
So is Parquet just growing a vector column type for the fixed-size
embeddings used in nearest-neighbour search to keep up with mainstream
databases, or is it jumping to support the broader set of ML tensor data to
accommodate all the random things an ML engineer might want to jam into a
Parquet file today? I guess that is the question?
Yes, that is a question and that was the heart of my position as well:
"""
Well, Parquet is not a single-purpose format.
Vector stores can choose whatever policy fits their usage, but we're
talking about a general-purpose file format that aims to be broadly
applicable.
"""
I didn't see any objection to this point.
I also didn't see any explanation why allowing non-finite values would
actually hamper database usage. There are precedents where we don't
impose limitations to data types which may be further limited in
databases (for example timestamps).
I also didn't see any consideration for the more logical solution of
adding metadata to tell whether a numeric column may contain non-finite
values (regardless of whether it's in a vector or not).
And in the end I find that some people here are having a very cavalier
attitude towards discussion and reaching consensus.
Regards
Antoine.
