First, regarding NaNs in vectors, that's a good question Russell :D: it's common enough in ML dev to get them from your models on overflow, etc., and if you're not thinking about handling them, it's easy enough for them to end up going unguarded into output? Production models ought to be clamping and filtering but I just searched the internet and found a few threads complaining that OpenAI's text-embedding intermittently returns NaNs instead of valid embeddings. Weird but seems true.
More generally, beyond fixed-size embeddings, there are various use-cases for NaNs in feature vectors and Infs in probability outputs? Probably more? There are also sparse vectors, FP16/8/4/1, scaling etc (pgvector recently added support for most of them too). Secondly, stepping back, some things have changed as this conversation has gone on and on :) For example, early on there were performance concerns, but it's been established that Parquet's LIST has no significant performance penalty for float embeddings. To recap that, a column might be declared with an arbitrary number of dimensions and a writer could insert a jagged shape in there filled with nulls. However, if all rows actually have the same shape, this is trivially detectable at the data page level. This allows for readers that care to have fast paths or very cheap error checks. Likewise, NaNs and Infs can be trivially spotted at decode time, or else a reader could just let them propagate through the math and do one-cycle bitfield isnan check on the result. So we've moved the conversation from physical storage to logical types and have been talking about a 1D fixed-size vector logical type to represent embeddings, with constraints on nulls and infs and nans. Howeevr, I fear that a VECTOR(FLOAT, 768) logical type doesn't deliver what it appears to promise. It states "fixed-length, non-null, finite" - all things the reader can already detect per-page for free. And it omits the one factor that truly matters for correctness: whether two columns share the same embedding space. That's application-level knowledge (which model, which training run) that belongs in metadata, not the type system. So strong opinion weakly held: * No performance problem. Regular data compresses the levels to near-nothing, and the regularity is trivially detectable from the encoded streams. Hardwood demonstrated this concretely. * No correctness problem. Nulls are visible at decode time. NaNs/Infs are either filtered early or propagate harmlessly through math and caught by a bitmask check on the result. And if we had a logical type defining fixed sizes etc, a defensive reader would check the values anyway to ensure they conform- it would be a brave engine that forwent the check that is so cheap anyway. * No expressiveness problem. Arbitrary-dimensional tensors already work as nested lists. Dense vs ragged is a property of the data, not the schema, and readers adapt per-page. So perhaps the type annotation solves the wrong problem` Structural compatibility (same length, same element type) doesn't imply semantic compatibility (same embedding space). The type gives false confidence while the real constraint remains the caller's responsibility. So its only a software problem not a Parquet problem if an engine that supports reading Parquet doesn't offer good-performance distance functions for LIST<FLOAT> embeddings. I don't think a type annotation enables functionality in a way that the user couldn't ask the engine to do the same thing without the type annotation. Where Parquet is behind is in the various schemes for smaller floats, smaller ints, scaling, sparsity etc? Currently, all that is likely done in different ways by different writers using FLBA in custom ways. It's likely that only who wrote the file can basically understand and use those fields? If the logical type was unified and could describe a VECTOR<BF16>[256] (or, well, pretty much everything the superset of nummy and pytorch and arrow etc supports :D) and engines would understand how to unpack and use that, then that would IMO be what moves this into enabling new usecases and engine interoperability. Sorry for the wall of text :) On Thu, 24 Sept 2026 at 20:12, Russell Spitzer <[email protected]> wrote: > I think something I haven't seen yet is the clear use-case for non-finite > values within a vector type. There has been a lot of theoretical discussion > here about use cases but what toolings currently produce these vectors and > how do they function in downstream engines? I'd like to limit introducing > types that have a wide range of values which are simply are failure modes > of different flavors > > I also don't find any issue with the attitude of folks in this discussion, > it seems reasonable that people are trying to work together to get things > done in the format for the practical use cases they know about. > > On Thu, Sep 24, 2026 at 8:49 AM Antoine Pitrou <[email protected]> wrote: > > > Le 24/09/2026 à 08:48, Will Edwards via dev a écrit : > > > > > > So is Parquet just growing a vector column type for the fixed-size > > > embeddings used in nearest-neighbour search to keep up with mainstream > > > databases, or is it jumping to support the broader set of ML tensor > data > > to > > > accommodate all the random things an ML engineer might want to jam > into a > > > Parquet file today? I guess that is the question? > > > > Yes, that is a question and that was the heart of my position as well: > > """ > > Well, Parquet is not a single-purpose format. > > > > Vector stores can choose whatever policy fits their usage, but we're > > talking about a general-purpose file format that aims to be broadly > > applicable. > > """ > > > > I didn't see any objection to this point. > > > > I also didn't see any explanation why allowing non-finite values would > > actually hamper database usage. There are precedents where we don't > > impose limitations to data types which may be further limited in > > databases (for example timestamps). > > > > I also didn't see any consideration for the more logical solution of > > adding metadata to tell whether a numeric column may contain non-finite > > values (regardless of whether it's in a vector or not). > > > > And in the end I find that some people here are having a very cavalier > > attitude towards discussion and reaching consensus. > > > > Regards > > > > Antoine. > > > > > > >
