Hi all, Initially, when I heard about formalizing the logical type for vectors, I thought of general ML tensor data. But after a few chats during the sync calls, I began to realise that Databricks narrowly wants a vector type to store fixed-length embeddings for nearest neighbour vector search, not a general ML tensor representation? In that light, the constraints regarding NaN, null, and infinity make sense, as does comparing the proposal to vector column support in other databases.
Vector search is cool. In the early ML explosion, specialist databases handled this, but mainstream databases later added general support, offering the advantage of not needing multiple servers etc. It's a bit like how mainstream databases eventually made JSON a first class column. When you look at Arrow and Lance, etc., you see a much broader set of features for tensor data. They also feature error handling for vector searches that consume vectors not meeting the constraints - a common issue with messy, real-world ML data. So is Parquet just growing a vector column type for the fixed-size embeddings used in nearest-neighbour search to keep up with mainstream databases, or is it jumping to support the broader set of ML tensor data to accommodate all the random things an ML engineer might want to jam into a Parquet file today? I guess that is the question? best, Will On Wed, 23 Sept 2026 at 21:47, Russell Spitzer <[email protected]> wrote: > Agreed with finite vectors. It’s a clearly supported type in the greater > ecosystem. Having nonfinite vectors seems to be the exception and as Alkis > mentioned, usually just a failure mode for engines which allow them. > > On Wed, Sep 23, 2026 at 11:22 AM Daniel Weeks <[email protected]> wrote: > > > I agree with moving forward with the proposal as-is. > > > > I think what we see from other databases with native vector support is > that > > they treat them as formal vectors, which distinguishes them from just a > > fixed-size array. > > > > It also shows that this is a well-supported type and not some niche type > > that is not generally useful/accepted. > > > > -Dan > > > > On Wed, Sep 23, 2026 at 11:04 AM Alkis Evlogimenos via dev < > > [email protected]> wrote: > > > > > Engines that disallow nulls, nans and infs: > > > - postgress > > > - oracle > > > - mysql > > > - sql server > > > - also: milvus, qdrant, pinecone > > > > > > Notable dbs that allow nulls, nans and infs: > > > - duckdb > > > - clickhouse > > > These seem to have this behavior because they do not have a specific > > vector > > > type so they use array(float) or fixed-size-array(float) and disable > > > functionality when nulls or non-finite values are present. > > > > > > > > > So VECTOR across most database engines disallow nulls, nans and infs. > > This > > > means better interoperability between parquet and such dbs so I think > we > > > should move ahead with the proposal as is. > > > > > > > > > On Wed, Sep 23, 2026 at 5:00 PM Antoine Pitrou <[email protected]> > > wrote: > > > > > > > > > > > Le 23/09/2026 à 16:26, Rok Mihevc a écrit : > > > > >> Well, Parquet is not a single-purpose format. > > > > >> > > > > >> Vector stores can choose whatever policy fits their usage, but > we're > > > > >> talking about a general-purpose file format that aims to be > broadly > > > > >> applicable. > > > > > > > > > > What is the gate for Parquet to define something as a logical type? > > > > > > > > There is no "gate". But we should act in the broader community's > > > > interest. A limitation that is only motivated by the needs of a > single > > > > segment of the user base goes against that goal. > > > > > > > > >> Why not? Why shouldn't it be used for storing, for example, NumPy > > > tensor > > > > >> data (which can contain NaNs and infinites)? > > > > > > > > > > A multidimensional arrays logical type might come with other > > > > > requirements (shapes, strides, dimension names), that vector > > > > > applications might not have. > > > > > > > > Well, first, you can use NumPy just for 1D data, and I'm sure some > > > > people do that. Second, strides wouldn't be saved in Parquet where > they > > > > are meaningless. Third, all of this can be conveyed as additional > > > > key-value metadata. > > > > > > > > >> It does not make sense to have *both* a "specialist" Vector type > > and a > > > > >> "general-purpose" FixedSizeList type. Parquet types are a > > > > >> general-purpose vocabulary from which you can build up more > > > specialized > > > > >> applications. Specialist types can be left to domain-specific SW > > users > > > > >> of Parquet. > > > > >> > > > > >> (for example, Arrow implementations serialize the Arrow schema in > > > > >> Parquet metadata so that the Arrow schema can be rebuilt at read > > time, > > > > >> without mandating that every Arrow datatype has a Parquet > > equivalent) > > > > > > > > > > That does seem to work nicely for Arrow [0]. Should we revisit the > > > > Parquet > > > > > extensions discussion [1][2]? > > > > > > > > I think we should, and then specialized communities such as ML can > > > > define their own specialized extension types (with arbitrary > semantics) > > > > on top of regular Parquet logical/physical types. > > > > > > > > Regards > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > >
