Thank you for writing up the alternative proposal Jan! > I still believe that FINITE_VECTOR, as proposed in PR #624 > <https://github.com/apache/parquet-format/pull/624>, is not the right > abstraction for a general-purpose format because it lacks composability.
FINITE_VECTOR is proposed as a narrow semantic type, aimed at a specific use case. We clearly also want a general-purpose fixed size list type (idealy a physical storage optimisation too) and FINITE_VECTOR does not preclude that, but we should pursue it as a separate effort. > *Finite floats:* Give floating-point types an optional isFinite property. While this would work - is anyone asking for finite elements other than this FINITE_VECTOR proposal? > Higher order tensors. You can just have list<list<finite float, 64>, 64>> for example. I don't think Parquet should model tensors (multidimensional arrays) this way. A common way to model tensors is a contiguous fixed size list of values plus metadata for shape and strides (dimension order) [1][2][3]. This matches how numeric libraries lay out tensors in memory, and strides allow for optimizations on data layout. If we model tensors as nested lists we get redundant repetition levels and likely still want to introduce strides to represent non-trivial memory layouts. [1] https://arrow.apache.org/docs/format/CanonicalExtensions.html#fixed-shape-tensor [2] https://pytorch.org/blog/tensor-memory-format-matters/ [3] https://blog.ezyang.com/2019/05/pytorch-internals/ Best, Rok On Mon, Nov 2, 2026 at 8:43 PM Jan Finis <[email protected]> wrote: > I still believe that FINITE_VECTOR, as proposed in PR #624 > <https://github.com/apache/parquet-format/pull/624>, is not the right > abstraction for a general-purpose format because it lacks composability. > > I've drafted an alternative proposal (PR #633) > <https://github.com/apache/parquet-format/pull/633> to demonstrate how we > could achieve the same functionality in a more composable way. This is by > no means ready to merge; it's primarily intended to illustrate the > approach. > > The idea is simple: introduce two orthogonal building blocks: > > - > > *Fixed-size lists:* Give lists an optional elementCount. > - > > *Finite floats:* Give floating-point types an optional isFinite > property. > > These two additions would allow us to express FINITE_VECTOR, but also > enable a much broader range of useful types: > > - > > Fixed-size lists of *any* element type, not just those supported by > FINITE_VECTOR. > - > > Finite floats in any context, whether standalone, in fixed- or > variable-size lists, in maps, or elsewhere. > - > > Higher order tensors. You can just have list<list<finite float, 64>, > 64>> for example. > > As a bonus, fixed-size lists would remain readable by legacy readers, which > would simply interpret them as variable-size lists. > As finiteness is applied only to floats, we also don't get the semantic > weirdness where FINITE_VECTOR can also annotate a vector of bools or ints, > which are trivially always finite. > > My main concern with FINITE_VECTOR is that it's highly specific to > embedding vectors. While embeddings are certainly a hot topic today, I > think baking such feature-specific abstractions into a general-purpose > format intended to remain relevant for decades is the wrong trade-off. > Different systems have different requirements, and those requirements will > evolve. > > I'd much rather see us build a small set of *orthogonal, composable types* > from which specialized types like FINITE_VECTOR can be constructed, rather > than introducing those specialized types directly into the format. > > > > Am Mo., 2. Nov. 2026 um 20:23 Uhr schrieb Jan Finis <[email protected]>: > > > > Yes, I expect different functions in engines. I think pgvector uses > more > > > specific functions for vectors, like `vector_dims` and `vector_norms`. > As > > > far as I'm aware, these aren't available for arrays without an explicit > > > cast. > > > > pg_vector is IMHO just one example that cannot be generalized. The reason > > (I suspect) that it uses its own type is that PostgreSQL's array type is > > somewhat > > overly bloated. It is not fixed length and it even supports multiple > > dimensions, leading > > to an overly verbose binary format. Had PostgreSQL a fixed-length array > > type that > > was just represented as the values in binary, I think that pgvector would > > not have > > chosen an own type. > > > > In contrast, I would expect most engines that already have a fixed-length > > array type > > to just use that for vectors. > > > > Am Mo., 2. Nov. 2026 um 15:03 Uhr schrieb Rok Mihevc < > [email protected] > > >: > > > >> > Could the proposal include a compatibility section addressing > >> unannotated > >> list data before we vote? > >> > >> Will, would a paragraph like this [1] be what you have in mind? > >> > >> On the topic of the vote - I would like to call it later today based on > >> parquet-format [2]. > >> > >> [1] > >> > >> > https://github.com/apache/parquet-format/pull/624/changes/8ad956d543e3074e1d9c8d4faa194199e06a8be4 > >> [2] https://github.com/apache/parquet-format/pull/624 > >> > >> Best, > >> Rok > >> > >> On Mon, Nov 2, 2026 at 12:42 PM Will Edwards via dev < > >> [email protected]> > >> wrote: > >> > >> > Thanks Ryan, Dan. > >> > > >> > My questions about what the type enables, and how you think Iceberg > >> tooling > >> > would use it, stem from the backward-compatibility concerns I've > raised > >> > earlier in the thread. I was trying to weigh the cost against the > >> benefit. > >> > I worry that this annotation will fragment and orphan a lot of data: > >> > perfectly good vectors that become second-class just because they > >> weren't > >> > written with the annotation. > >> > > >> > There is an enormous amount of embedding data in Parquet today. Many > >> many > >> > lottabytes of it :) Some of it is packed in bespoke formats, but a > very > >> > large share is plain, unannotated lists of floats or doubles. In > >> addition > >> > to all this existing data, many pipelines will keep writing new data > >> this > >> > way too, because they are unaware of the new annotation or can't be > >> > changed. > >> > > >> > Ryan, you said compatibility isn't a concern because this is a new > >> type. I > >> > think that assumes the data is new data written by an Iceberg writer. > >> But > >> > many Iceberg tables are built from files written by other engines. The > >> > standard implementations support this directly (add_files, migrate, > name > >> > mapping), and the popular public embedding datasets are a typical > >> source. > >> > If Iceberg requires vector columns to carry the annotation, none of > >> those > >> > files can back a vector column without being rewritten, even though > the > >> > bytes would be identical. > >> > > >> > It makes sense for Iceberg writers to check for inf/nan/null when they > >> > write a vector column. But since many files come from other writers, > >> robust > >> > readers should check correctness anyway, as they do with UTF-8 for > >> strings. > >> > > >> > So my position remains that this would be better done entirely in > >> Iceberg > >> > rather than pushed down to Parquet. Iceberg ought to accept > unannotated > >> > LIST<FLOAT>/LIST<DOUBLE> files for vector columns, including via > >> add_files > >> > and a metadata-only list → vector promotion. > >> > > >> > Could the proposal include a compatibility section addressing > >> unannotated > >> > list data before we vote? > >> > > >> > Will > >> > > >> > On Sat, 31 Oct 2026 at 00:00, Ryan Blue via dev < > [email protected] > >> > > >> > wrote: > >> > > >> > > > 1. do you expect that the backing files that are parquet MUST use > >> the > >> > new > >> > > logical vector type? > >> > > > >> > > I expect that the Iceberg spec will require writers to produce the > >> > > annotation. > >> > > > >> > > > 2. do you expect the software accessing an iceberg table . . . to > >> > offer > >> > > different functions on vector columns compared to naked lists? > >> > > > >> > > Yes, I expect different functions in engines. I think pgvector uses > >> more > >> > > specific functions for vectors, like `vector_dims` and > >> `vector_norms`. As > >> > > far as I'm aware, these aren't available for arrays without an > >> explicit > >> > > cast. > >> > > > >> > > > Will a nearest neighbour search on a naked list of doubles be > >> > supported, > >> > > for example? > >> > > > >> > > This is an engine decision, but a list like this would not be > exposed > >> as > >> > a > >> > > vector. You would most likely cast it to a vector in order to call > >> vector > >> > > functions. > >> > > > >> > > > 3. And will non-parquet backing files like avro and orc be > supported > >> > for > >> > > the > >> > > vector columns? > >> > > > >> > > All Iceberg types can be stored in supported file formats. > >> > > > >> > > It seems like the direction for this line of questioning is > basically > >> the > >> > > same as before: Can we just treat list<float> as a synonym of > vector? > >> > > > >> > > And again, I think this is the wrong question. We have users right > now > >> > that > >> > > struggle with errors that are found at read time, like the > dimensions > >> of > >> > > their vectors not matching. Finding this out at read time is a > problem > >> > and > >> > > we want to shift validation to write time so we can catch errors > >> earlier > >> > > and have guarantees that the data is usable. As I mentioned, one of > >> the > >> > > ways we have customers using data is sharing it with systems like > >> > pgvector > >> > > that also have these requirements. Just because you can think of a > >> vector > >> > > as a very special list of floats doesn't mean that write time > >> guarantees > >> > > are unreasonable or unnecessary. > >> > > > >> > > Ryan > >> > > > >> > > > >> > > On Thu, Oct 29, 2026 at 12:57 PM Will Edwards via dev < > >> > > [email protected]> wrote: > >> > > > >> > > > Thx for replying. > >> > > > > >> > > > I keep trying to explain that there are no performance advantages. > >> But > >> > > > anyway :) > >> > > > > >> > > > I understand you are envisaging a vector column type (with the > fixed > >> > > length > >> > > > nan inf constraints you’ve been detailing) being added to iceberg? > >> > Then > >> > > > the following choices are interesting: > >> > > > > >> > > > 1. do you expect that the backing files that are parquet MUST use > >> the > >> > new > >> > > > logical vector type? > >> > > > > >> > > > 2. do you expect the software accessing an iceberg table - > >> sparksql, > >> > for > >> > > > example - to offer different functions on vector columns compared > to > >> > > naked > >> > > > lists? Will a nearest neighbour search on a naked list of doubles > >> be > >> > > > supported, for example? > >> > > > > >> > > > 3. And will non-parquet backing files like avro and orc be > supported > >> > for > >> > > > the vector columns? > >> > > > > >> > > > thx > >> > > > > >> > > > Will > >> > > > > >> > > > On Sat, 24 Oct 2026 at 02:08, Ryan Blue <[email protected]> wrote: > >> > > > > >> > > > > > I'm asking for a workload that can't be done on LIST<FLOAT> > now > >> and > >> > > > would > >> > > > work with VECTOR. Jan asked the same question, and I don't think > >> either > >> > > of > >> > > > us has seen one yet. > >> > > > > >> > > > I think this is the wrong question to ask, but I'm glad that > you're > >> > > asking > >> > > > directly. Clearly, a variable-length list is a superset of a > >> > fixed-length > >> > > > list. And a list of floats is a superset of a list of finite > >> floats. A > >> > > list > >> > > > of optional elements is a superset of a list of required > elements. I > >> > > don't > >> > > > think there's a use case that _can't_ be done. > >> > > > > >> > > > Is there a use case that can't be done with binary in place of > >> string? > >> > > And > >> > > > why don't we use JSON or Variant for everything? Just because we > can > >> > use > >> > > a > >> > > > wider type doesn't mean that we don't benefit from communicating > >> > > guarantees > >> > > > like "this must be UTF-8". > >> > > > > >> > > > Types introduce guarantees that we can use to get better > performance > >> > and > >> > > to > >> > > > either prevent errors or raise them closer to where data is > >> produced. > >> > > > > >> > > > For vectors, the guarantee that elements are finite gives us > better > >> > > > performance because we don't need to check each element. This also > >> > moves > >> > > > the error closer to where data is produced because writers are > >> required > >> > > to > >> > > > verify the guarantees about each element and will detect bad > values > >> at > >> > > > write time rather than at read time. > >> > > > > >> > > > Ryan > >> > > > > >> > > > >> > > >> > > >
