Agreed with finite vectors. It’s a clearly supported type in the greater ecosystem. Having nonfinite vectors seems to be the exception and as Alkis mentioned, usually just a failure mode for engines which allow them.
On Wed, Sep 23, 2026 at 11:22 AM Daniel Weeks <[email protected]> wrote: > I agree with moving forward with the proposal as-is. > > I think what we see from other databases with native vector support is that > they treat them as formal vectors, which distinguishes them from just a > fixed-size array. > > It also shows that this is a well-supported type and not some niche type > that is not generally useful/accepted. > > -Dan > > On Wed, Sep 23, 2026 at 11:04 AM Alkis Evlogimenos via dev < > [email protected]> wrote: > > > Engines that disallow nulls, nans and infs: > > - postgress > > - oracle > > - mysql > > - sql server > > - also: milvus, qdrant, pinecone > > > > Notable dbs that allow nulls, nans and infs: > > - duckdb > > - clickhouse > > These seem to have this behavior because they do not have a specific > vector > > type so they use array(float) or fixed-size-array(float) and disable > > functionality when nulls or non-finite values are present. > > > > > > So VECTOR across most database engines disallow nulls, nans and infs. > This > > means better interoperability between parquet and such dbs so I think we > > should move ahead with the proposal as is. > > > > > > On Wed, Sep 23, 2026 at 5:00 PM Antoine Pitrou <[email protected]> > wrote: > > > > > > > > Le 23/09/2026 à 16:26, Rok Mihevc a écrit : > > > >> Well, Parquet is not a single-purpose format. > > > >> > > > >> Vector stores can choose whatever policy fits their usage, but we're > > > >> talking about a general-purpose file format that aims to be broadly > > > >> applicable. > > > > > > > > What is the gate for Parquet to define something as a logical type? > > > > > > There is no "gate". But we should act in the broader community's > > > interest. A limitation that is only motivated by the needs of a single > > > segment of the user base goes against that goal. > > > > > > >> Why not? Why shouldn't it be used for storing, for example, NumPy > > tensor > > > >> data (which can contain NaNs and infinites)? > > > > > > > > A multidimensional arrays logical type might come with other > > > > requirements (shapes, strides, dimension names), that vector > > > > applications might not have. > > > > > > Well, first, you can use NumPy just for 1D data, and I'm sure some > > > people do that. Second, strides wouldn't be saved in Parquet where they > > > are meaningless. Third, all of this can be conveyed as additional > > > key-value metadata. > > > > > > >> It does not make sense to have *both* a "specialist" Vector type > and a > > > >> "general-purpose" FixedSizeList type. Parquet types are a > > > >> general-purpose vocabulary from which you can build up more > > specialized > > > >> applications. Specialist types can be left to domain-specific SW > users > > > >> of Parquet. > > > >> > > > >> (for example, Arrow implementations serialize the Arrow schema in > > > >> Parquet metadata so that the Arrow schema can be rebuilt at read > time, > > > >> without mandating that every Arrow datatype has a Parquet > equivalent) > > > > > > > > That does seem to work nicely for Arrow [0]. Should we revisit the > > > Parquet > > > > extensions discussion [1][2]? > > > > > > I think we should, and then specialized communities such as ML can > > > define their own specialized extension types (with arbitrary semantics) > > > on top of regular Parquet logical/physical types. > > > > > > Regards > > > > > > Antoine. > > > > > > > > > > > >
